Model reasoning acceleration method and computer equipment

Abstract

The invention discloses a model reasoning acceleration method and computer equipment, and relates to the field of artificial intelligence, and the method comprises the steps: obtaining the available calculation power of a current calculation unit for calculating and generating a key value cache, and the available transmission capability of a data transmission path connected with a plurality of storage units; on the basis of the available computing power and the available transmission capacity, the loading amount of the key value cache in each storage unit is determined, and the computing unit computes and generates the mapping relation between the computing amount of the key value cache and the to-be-optimized parameters; the to-be-optimized parameter represents the recovery efficiency of the key value cache; determining a target allocation strategy based on the mapping relation and the optimization target of the to-be-optimized parameter; the target allocation strategy comprises respective target values of the loading amount and the calculation amount, and executing a loading task of loading the key value cache from each storage unit and a calculation task of calculating and generating the key value cache by the calculation unit based on the target allocation strategy so as to obtain a target key value cache set.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and in particular to a method for accelerating model inference and a computer device. Background Technology

[0002] Currently, in the reasoning process of intelligent models such as Large Language Model (LLM), the attention mechanism needs to maintain a key-value cache (KV Cache, i.e., key-value vector) of historical tokens, which causes it to consume a lot of GPU memory resources in scenarios such as processing long texts or multi-turn dialogues, seriously affecting reasoning performance, limiting the computational throughput in the corresponding scenarios, and making it difficult to meet the real-time response requirements. Summary of the Invention

[0003] In view of the above problems, this application provides the following solution:

[0004] The first aspect of this application provides a method for accelerating model inference, the method comprising:

[0005] Obtain the available computing power of the current computing unit for calculating and generating the key-value cache, as well as the available transmission capacity of the data transmission path connecting multiple storage units;

[0006] Based on the available computing power and the available transmission capacity, the mapping relationship between the loading amount of the key-value cache in each of the storage units, the computational amount of the computing unit to generate the key-value cache, and the parameters to be optimized is determined; the parameters to be optimized characterize the recovery efficiency of the key-value cache.

[0007] Based on the mapping relationship and the optimization objective of the parameters to be optimized, a target allocation strategy is determined; the target allocation strategy includes the target values ​​of the loading amount and the computation amount, respectively.

[0008] Based on the target allocation strategy, a loading task is executed to load key-value cache from each of the storage units, and a calculation task is executed with the computing unit to generate key-value cache, so as to obtain the target key-value cache set.

[0009] A second aspect of this application provides a computer device, comprising:

[0010] At least one data transmission port is configured to connect to a storage unit to enable the transmission of key-value caches between the storage unit and the storage unit;

[0011] The processing apparatus is configured to implement the various steps of the model inference acceleration method provided in the first aspect of this application. Attached Figure Description

[0012] The above and other features, advantages, and aspects of the embodiments of this disclosure will become more apparent from the accompanying drawings and the following detailed description. Throughout the drawings, the same or similar reference numerals denote the same or similar elements. It should be understood that the drawings are schematic, and the originals and elements are not necessarily drawn to scale.

[0013] Figure 1 A schematic flowchart of the model inference acceleration method provided in Embodiment 1 of this application;

[0014] Figure 2 A schematic flowchart of the model inference acceleration method provided in Embodiment 2 of this application;

[0015] Figure 3 A schematic flowchart of the model inference acceleration method provided in Embodiment 3 of this application;

[0016] Figure 4 A schematic flowchart of the model inference acceleration method provided in Embodiment 4 of this application;

[0017] Figure 5 A schematic flowchart of the model inference acceleration method provided in Embodiment 5 of this application;

[0018] Figure 6 A schematic flowchart of the model inference acceleration method provided in Embodiment Six of this application;

[0019] Figure 7 This is a schematic diagram of the structure of a model inference acceleration device provided in an embodiment of this application;

[0020] Figure 8 This application provides a schematic diagram of the hardware structure of a computer device according to an embodiment of the present application.

[0021] Figure 9 This is a schematic diagram of the system architecture and workflow of the model inference acceleration method provided in the embodiments of this application. Detailed Implementation

[0022] The embodiments of this application are described below with reference to the accompanying drawings. The terminology used in the implementation section of this application is only for explaining specific embodiments and is not intended to limit the application. The embodiments of this application are described below with reference to the accompanying drawings. It will be understood by those skilled in the art that, with the development of technology and the emergence of new scenarios, the technical solutions provided in the embodiments of this application are also applicable to similar technical problems.

[0023] The terms “first,” “second,” etc., used throughout this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such terms are interchangeable where appropriate; this is merely a way of distinguishing objects with the same attributes in the embodiments of this application. Furthermore, the terms “comprising” and “having,” and any variations thereof, are intended to cover non-exclusive inclusion, so that a process, method, system, product, or apparatus that comprises a series of units is not necessarily limited to those units, but may include other units not explicitly listed or inherent to those processes, methods, products, or apparatuses.

[0024] Regarding the content described in the background technology section, for the key-value cache (KV cache, i.e., key cache and value cache) of historical tokens generated during the attention calculation process, in order to alleviate the rapid increase in GPU (Graphics Processing Unit, which can be called a computing unit) memory resource occupation caused by the length of the inference sequence, which seriously limits the GPU's concurrency capability and system throughput, and to further reduce the system's resource utilization and energy efficiency by occupying memory resources without participating in the calculation for a long time. Therefore, a proposal is made to migrate a large number of historical token KV caches from GPU memory to external storage media (which can be called storage units) for storage, so that they can be loaded back into GPU memory when needed, thereby effectively reducing memory occupation and avoiding memory overflow problems.

[0025] In the process of loading the KV Cache, if all the required KV Cache is directly loaded from the external storage unit to the video memory, i.e., the required KV Cache is restored, the single KV Cache restoration path is easily limited by the bandwidth of a single storage unit or high load, forming a transmission bottleneck. This leads to a significant increase in latency in the entire loading process, causing a degradation in attention computation performance and reducing model inference speed. To address this, this application provides a model inference acceleration method. By dynamically sensing the computational and transmission resources of the system, it utilizes multiple KV Cache restoration paths, such as directly loading the KV Cache from the storage unit and directly generating the KV Cache from the computation unit. Through collaborative balancing of computational and storage resources, and adaptive optimization of the allocation strategy for different KV Cache restoration paths, the method significantly improves KV Cache restoration efficiency. This enhances concurrency, response speed, and system throughput in scenarios such as long text inference, multi-turn dialogues, session pauses, or user input intervals, reliably meeting real-time response requirements. The model inference acceleration method of this application embodiment will be described in detail below with reference to the accompanying drawings.

[0026] It should be noted that the model involved in this application can be a general AI (Artificial Intelligence) model, which can adopt, but is not limited to, the Transformer or its architectural variants (such as using only encoder-only / decoder-only, encoder-decoder, or MoE (Mixture of Experts, a neural network architecture) or other basic architectures, to learn the features and rules of natural language by training on a large amount of diverse data, thereby being able to understand and generate natural language. It typically has hundreds of millions to hundreds of billions of model parameters (model parameters are variables that control the behavior of the target model) and is able to capture complex relationships and patterns in natural language.

[0027] The AI ​​models may include, but are not limited to, generative models and generative language models (GLMs). Examples include one or more of the following: large language models (LLMs), GPT (Generative Pre-trained Transformer) series models, T5 (Text to Text Transfer Transformer) models, large visual models, and multimodal large models. Depending on actual needs, the models involved in this application's embodiments may also be expert large models fine-tuned from general AI models based on application requirements, or lightweight models compressed through lightweight methods such as quantization, knowledge distillation, or pruning to meet the needs of edge deployments with limited computing resources. This application does not limit the model types mentioned in the context and can determine them based on actual application requirements.

[0028] Based on the above analysis, and referring to Figure 1 This is a flowchart illustrating the model inference acceleration method provided in Embodiment 1 of this application. This method can be applied to computer devices, which can be device nodes in systems such as servers or cloud platforms, or terminal devices such as smartphones, personal computers (PCs), augmented reality (AR) / virtual reality (VR) devices, robots, or smart medical / transportation equipment. This application does not limit the product form of the computer device. Figure 1 As shown, the model inference acceleration method proposed in this embodiment may include:

[0029] Step S11: Obtain the available computing power of the current computing unit for calculating and generating key-value cache, as well as the available transmission capacity of the data transmission path connecting multiple storage units;

[0030] Based on the above analysis, in scenarios requiring rapid recovery of KV Cache corresponding to historical tokens, such as long text reasoning, multi-turn dialogue continuation, and resource-constrained environments (e.g., edge devices), this application proposes two KV Cache recovery methods: recalculating and generating the KV Cache through the computing unit, and directly loading the KV Cache from the storage unit. This eliminates reliance on a single storage loading or computation-generated KV Cache recovery method, thereby improving KV Cache recovery efficiency. To reliably achieve optimal allocation for all KV Cache recovery, considering the amount of KV Cache data recovered by different recovery methods to significantly improve KV Cache recovery efficiency, this application will consider the impact of dynamic resource changes on optimal allocation. Therefore, when responding to a KV Cache recovery / loading request, this application can first obtain the current system resource status related to KV Cache recovery, i.e., the available computing power of the computing unit for calculating and generating the key-value cache, and the available transmission capacity of the data transmission paths for transmitting KV Cache data from various storage units.

[0031] In this context, a computing unit refers to the hardware used to calculate and generate the KV Cache for tokens. This typically includes a GPU (such as an NVIDIA GPU with a discrete or integrated graphics system) or other dedicated AI accelerators like a TPU (Tensor Processing Unit). This application uses a GPU as an example. Available operators refer to the remaining effective computing power allocated to recalculating the KV Cache after deducting the current load (already used computing power) of the computing unit. Under the same KV Cache computation load, the higher the available computing power of the computing unit, the faster it can recalculate and generate the KV Cache, and the more large-scale KV Cache computation tasks it can support. To avoid blindly calculating the KV Cache and causing a spike in inference latency, this application can dynamically sense the actual load status of the computing unit and obtain the available computing power currently used by the computing unit to calculate and generate the KV Cache. The implementation process is not limited in this application.

[0032] The multiple storage units connected to the computing unit refer to the KV Cache devices that store historical tokens. These may include, but are not limited to, CPU local memory (such as DRAM, Dynamic Random Access Memory), CXL (Compute Express Link, an open high-speed interconnect standard) memory, local hard disks (such as disks, solid state disks (SSDs), external hard disks, magnetic disks, or other types of hard disks), remote storage devices (such as high-performance all-flash storage platforms, distributed KV storage systems, or other remote storage), or other external / dynamically expandable storage devices. The number and type of storage units in the KV Cache storage system can be dynamically adjusted according to actual conditions.

[0033] In this application, multiple storage units can correspond to multiple independent data transmission paths, which can be determined based on the corresponding data transmission port type and its supported transmission protocol, such as one or more combinations of DDR (Double Data Rate), PCIe 4.0 (Peripheral Component Interconnect Express Generation 4.0), NVLink (NVIDIA Link), or other paths supporting high-bandwidth transmission protocols. To avoid situations where a storage unit can output KV Cache but the data transmission link capacity is insufficient to transmit the KV Cache, or where the data transmission link has sufficient capacity to transmit the KV Cache but cannot successfully read the KV Cache from the storage unit, this application can dynamically sense the effective bandwidth currently available for transmitting KV Cache on each data transmission path. For example, the remaining bandwidth after deducting the bandwidth occupied by other tasks is determined as the available transmission capacity of the data transmission path to measure the amount of KV Cache transmission it can currently support. This application does not restrict the method for obtaining the available transmission capacity.

[0034] It should be noted that for the KV Cache stored in each storage unit, loading the KV Cache from the storage unit and recalculating and generating the KV Cache by the computing unit can both be implemented according to the token dimension. In other words, all the processing of KV Cache in this application is based on the size of the KV Cache corresponding to a single token as the smallest processing unit (i.e., KV Cache processing granularity), so that the KV Caches corresponding to the same token are stored in the same storage unit, and the computing unit calculates and generates at least one KV Cache corresponding to a token.

[0035] Based on the above analysis, this application dynamically obtains the actual resource status of the computation and transmission KV Cache, providing accurate data support for the subsequent establishment of mapping relationships, avoiding resource mismatch caused by fixed allocation ratios or static rule allocation, thereby reducing the recovery efficiency of KV Cache.

[0036] Step S12: Based on available computing power and available transmission capacity, determine the mapping relationship between the loading amount of key-value cache in each storage unit, the computation amount of computing unit to generate key-value cache, and the parameters to be optimized.

[0037] Among them, the parameter to be optimized represents the recovery efficiency of the key-value cache. It can be a quantifiable indicator that represents the recovery efficiency, such as recovery latency (the total time from startup recovery to the availability of the KV Cache), resource utilization (the average of computing power utilization of computing units and bandwidth utilization of data transmission paths, etc.), and comprehensive utilization (the weighted sum of the inverse of recovery latency and resource utilization, the weight can be dynamically adjusted according to the scenario, such as a weight of 0.7 for latency-sensitive scenarios and a weight of 0.3 for resource-sensitive scenarios, etc., without any restrictions).

[0038] Based on the above analysis, this application can quantify the correlation between resource allocation and KV Cache recovery efficiency by establishing a mapping relationship, providing a basis for subsequent decision-making. The mapping relationship refers to a functional relationship where the candidate values ​​(number of KV Caches) for the loading amount of KV Cache in each storage unit and the computational amount for generating KV Cache in each computing unit are used as inputs (decision variables), and the parameters to be optimized are used as outputs. This application can construct the mapping relationship through data modeling, but is not limited to this method.

[0039] In this application, for each storage unit and computation unit, multiple mapping relationships can be determined according to, but not limited to, the methods described above. The mapping relationship between the load amount of the KV Cache in each storage unit (the number of readable KV Caches) and the parameters to be optimized is recorded as the first mapping relationship. The mapping relationship between the computation amount of each computation unit in generating the KV Cache (the number of KV Caches) and the parameters to be optimized is recorded as the second mapping relationship. Thus, multiple first mapping relationships and at least one second mapping relationship can be determined. Of course, this application can also comprehensively determine the mapping relationship between the load amount of the KV Cache in each storage unit, the computation amount of the load amount generated by each computation unit, and the parameters to be optimized. This application does not limit the representation of the mapping relationship.

[0040] Understandably, as the available computing power of a computing unit dynamically changes, the corresponding second mapping relationship can also change accordingly. For example, with the same KV Cache computational load, the computation generation speed / time of a computing unit may differ, meaning the value of the parameter to be optimized mapped to that KV Cache computational load will be different. Similarly, as the available bandwidth of each storage unit and / or data transmission path dynamically changes, the corresponding first mapping relationship can also change accordingly. For example, with the same KV Cache loading load, the transfer rate / time of transferring it from that storage unit to the video memory may differ, meaning the value of the parameter to be optimized mapped to that KV Cache loading load will be different.

[0041] Based on this, this application can determine the mapping relationship according to the method described above when the KV Cache needs to be restored in each round. Alternatively, it can determine the corresponding mapping relationship according to the method described above when the available computing power and / or available transmission capacity are detected to have changed (or the amount of change is greater than the threshold). If the available computing power and / or available transmission capacity have not changed (e.g., the amount of change is less than the threshold, which is considered to be basically unchanged), it is not necessary to rebuild the corresponding mapping relationship and the corresponding mapping relationship determined in the previous round can be read directly, but it is not limited to this.

[0042] Step S13: Based on the mapping relationship and the optimization objective of the parameters to be optimized, determine the target allocation strategy; the target allocation strategy includes the target values ​​of loading amount and computation amount respectively;

[0043] Based on the above analysis of the technical solution of this application, this application considers the dynamic changes in resources and dynamically decides the optimal resource allocation scheme (denoted as the target allocation strategy). To this end, based on a mapping relationship adapted to the current available computing power and available transmission capacity, and the optimization objectives of pre-configured parameters to be optimized, the target values ​​(KV Cache target quantity) for the loading volume corresponding to each storage unit (or each data transmission path) and the computational volume corresponding to each computing unit can be solved, ensuring optimal KV Cache recovery efficiency.

[0044] The optimization objective is the expected requirement of the parameter to be optimized in the current scenario. Based on the description of the parameter to be optimized above, the expected requirement for the same parameter to be optimized can differ in different scenarios. For example, in scenarios where inference speed needs to be prioritized, the optimization objective could be to minimize recovery latency (recovery time); in scenarios where hardware resource conservation needs to be prioritized, the optimization objective could be to maximize resource utilization; in scenarios where recovery latency ≤ 50ms and resource utilization ≥ 80% are required, the optimization objective could be multi-objective optimization: recovery latency ≤ 50ms and resource utilization ≥ 80%, but it is not limited to these.

[0045] The target allocation strategy can include specific values ​​(target values) for the amount of KV Cache loaded into each storage unit and specific values ​​(target values) for the amount of computation required to generate KV Cache in the computing unit, when the above optimization objectives are met. For example, loading 500,000 KV Caches from CPU memory, loading 200,000 KV Caches from SSD, and generating 300,000 KV Caches using GPU computation, but not limited to these. Based on the above analysis, it can be seen that the determined target allocation strategy will dynamically change after changes in available computing power and / or available transmission capacity, reliably meeting the optimization objectives.

[0046] In determining the target allocation strategy, to ensure its feasibility, it is necessary to follow corresponding optimization constraints, such as data integrity constraints (ensuring the sum of all loading and computational loads equals the total amount of data in the KVCache to be recovered), storage capacity constraints (the loading amount of any storage unit's KVCache does not exceed the actual number of KVCaches it stores), non-negative value constraints (each target value is a non-negative integer), and resource upper limit constraints (e.g., under the maximum allowable latency, the bandwidth consumed by loading each loading amount of KVCache does not exceed the corresponding available transmission capacity, and the computing resources consumed by calculating and generating the computational amount of KVCache do not exceed its available computing power). Therefore, when the optimization objective is as shown in the example above, this application can transform the mapping relationship into a constraint optimization problem, and determine the optimal solution for each decision variable using appropriate solution tools, i.e., determine the optimal allocation scheme (target allocation strategy) under the current system resource state. This application does not restrict the implementation process of step S13.

[0047] Step S14: Based on the target allocation strategy, execute the loading task of loading key-value cache from each storage unit and the calculation task of generating key-value cache with the computing unit to obtain the target key-value cache set.

[0048] In this embodiment, a multi-task collaborative startup approach can be used to ensure the rapid recovery of the required KV Cache to support model inference. Preferably, this application can use a multi-threaded / multi-stream mechanism to synchronously start the loading tasks of the KV Cache in each storage unit and the computation tasks (or generation tasks) of the computation unit to generate the KV Cache, in order to avoid the cumulative latency caused by serial execution and reduce the recovery efficiency of the KV Cache. For example, while the GPU starts the KV Cache computation task, various storage units such as CPU memory, SSD, CXL memory, and remote storage devices simultaneously start KV Cache loading tasks (reading KV Cache tasks). Data is transferred to GPU video memory through DMA (Direct Memory Access) or other methods, but it is not limited to this. The loading method of the KV Cache in the corresponding storage unit can be determined by combining the relevant description of the data transmission path above.

[0049] Optionally, for the loading tasks of KV Cache in each storage unit and the computation tasks of generating KV Cache in the computing unit, in addition to the parallel execution described above, some tasks can also be executed in parallel according to actual needs. Compared with the serial execution of each task, this can also improve the recovery efficiency of the required KV Cache and adapt to scenarios with dynamic resource changes. For example, in the long text inference scenario expansion, when the total amount of data in the KV Cache to be recovered reaches a certain threshold (such as 10 million or 50 million, etc., this application does not limit the value, and it can be determined based on experience or model inference performance, deployment equipment hardware resources, etc.), the KV Cache can be recovered in batches, and the remaining part can be recovered in a pipelined parallel manner. Taking the recovery of 10 million KV Cache as an example, 5 million KV Cache can be recovered first in the manner described above to support the first half of text inference, while the remaining 5 million KV Cache are recovered in parallel to reduce end-to-end latency. This application does not limit the implementation method of how to coordinate the execution of the above multiple loading tasks and computation tasks.

[0050] Each KV Cache obtained through the above tasks can be concatenated according to the corresponding token order and attention head dimension to form a complete target KV Cache set, which can be directly used to implement the model's attention calculation task, such as directly inputting the attention calculation module of the Transformer layer to support model inference, such as contextual understanding of long texts and reuse of historical information in multi-turn dialogues. This application does not restrict the subsequent processing tasks of the target KV Cache set, which can be determined as appropriate.

[0051] In summary, the embodiments of this application propose a diversified KV Cache recovery path. By quantifying resource capabilities, determining suitable parameter mapping relationships, and optimizing target decisions to achieve optimal allocation of KV Cache recovery, a closed-loop scheme for coordinating the execution of multiple loading and computation tasks is proposed. As resources change dynamically, the target allocation strategy for KV Cache recovery is adjusted in a timely manner, reliably solving the problem of low KV Cache recovery efficiency in traditional model inference.

[0052] In one possible implementation, the model inference acceleration method proposed in this application can be integrated into a model inference framework through a pluggable architecture, such as a general framework determined according to the model type and application scenario, an LLM-specific framework, or a vertical domain framework for specific needs. Without modifying the model structure, the deployment cost is low by adding functional modules such as resource acquisition modules, mapping relationship model construction modules, and allocation strategy decision modules.

[0053] In some embodiments, a fault-tolerance mechanism can be incorporated into the implementation of step S14 to improve the reliability of KV Cache recovery. Thus, when a data transmission path failure is detected, such as a failure to read the KV Cache in a storage unit, the load amount of the KV Cache in that storage unit can be promptly allocated to other data transmission paths (i.e., adjusting the load amount of the KV Cache in at least one other storage unit) or computation unit to ensure uninterrupted KV Cache recovery. During this reallocation process, a random allocation strategy can be used to reallocate the load amount of the KV Cache corresponding to the failed data transmission path, thereby increasing at least one other load amount and / or computation amount. Optionally, this application can also redetermine the target allocation strategy according to the method described above, and execute step S14, etc., on the new target allocation strategy. This application does not limit the reallocation implementation method.

[0054] Reference Figure 2 This is a flowchart illustrating the model inference acceleration method proposed in Embodiment 2 of this application. This embodiment describes one possible implementation of the model inference acceleration method proposed above, such as... Figure 2 As shown, the implementation method may include:

[0055] Step S21: Collect the computing power utilization rate of the current computing unit, the available output bandwidth of each of the multiple storage units, and the available input bandwidth of the data transmission path of each of the multiple storage units connected to the computing unit.

[0056] In this embodiment, KV Cache management can be achieved by dynamically sensing the system load status. For example, by monitoring the load status of the computing unit and each externally connected storage unit in real time, under the constraints of computing and transmission resources, the optimal allocation of the KV Cache to be restored (i.e., determining the target allocation strategy) can be determined, thereby realizing adaptive scheduling and efficient utilization of transmission (direct loading) and computing resources, and improving KV Cache recovery efficiency.

[0057] During the real-time perception of system load status, the real-time resource status of the computing and transmission links can be comprehensively acquired, providing accurate data support for subsequent quantitative analysis and avoiding deviations in allocation strategies due to missing or expired parameters. Therefore, this application can collect (e.g., implemented by the CPU) the current operating status indicators of each computing unit and each storage unit in the system in real time, such as the bandwidth occupied by the computing unit link (in this application, it refers to the available input bandwidth of each data transmission path, that is, the transmission bandwidth for writing KV Cache from the connected storage unit to the video memory), computing power utilization, and the available output bandwidth (available read bandwidth, which can be directly collected or determined based on the difference between the configured total read bandwidth and the currently occupied read bandwidth) of each storage unit connected to the computing unit. This application does not restrict the dynamic acquisition method of the current system resource status indicators.

[0058] Among them, computing power utilization refers to the ratio of the computing power currently occupied by a computing unit to the total configured computing power, such as the utilization rate of the GPU's SM (Streaming Multiprocessor, the core computing unit of the GPU architecture). It reflects the busyness of the GPU computing core and can be used to measure the efficiency of GPU computing resource utilization, such as the average percentage of time during which an SM has active thread execution within the sampling period. This application can use hardware monitoring tools to collect data in real time according to a preset collection period (such as 10-50ms) to ensure the capture of dynamic load changes of the computing unit.

[0059] The available output bandwidth of each storage unit can be the maximum real-time rate at which the storage unit outputs the KV Cache to the data transmission path. It is the remaining bandwidth after deducting the bandwidth occupied by other I / O tasks of the storage unit. It can be collected by the storage unit's built-in monitoring tools (such as memory bandwidth testing tools, NVMe tools, etc.), but is not limited to these. The available input bandwidth of each data transmission path refers to the maximum real-time rate at which the data transmission path inputs the KV Cache to the compute unit's video memory. It is the remaining bandwidth after deducting the bandwidth occupied by other transmission tasks. It can be collected by link controller monitoring tools, etc. This application does not limit the indicator collection tools and their working principles in step S21.

[0060] Step S22: Compare the available input bandwidth of each data transmission path with the available output bandwidth of the corresponding storage unit to determine the available transmission capacity of the corresponding data transmission path.

[0061] To identify KV Cache transmission bottlenecks in data transmission paths and ensure the actual supportable KV Cache transmission rate (i.e., to ensure the accuracy of transmission capacity assessment), this application provides a reliable basis for subsequent KV Cache loading allocation in corresponding storage units, avoiding overestimation of the transmission capacity of data transmission paths that could lead to loading task timeouts. This application uses a bidirectional bandwidth minimum method to accurately locate transmission bottlenecks. Specifically, for each data transmission path, the minimum of its available input bandwidth and the available output bandwidth of the corresponding storage unit is determined as the effective bandwidth available for loading the KV Cache, i.e., the available transmission capacity of the data transmission path, thus avoiding resource waste during subsequent KV Cache loading.

[0062] For example, if the available output bandwidth of the CPU DRAM is 128GB / s and the available input bandwidth of its DDR5 channel connected to the GPU is 512GB / s, then the available transmission capacity of this data transmission path is 128GB / s; if the available output bandwidth of the SSD is 8GB / s and the available input bandwidth of its PCIe 4.0 channel connected to the GPU is 10GB / s, then the available transmission capacity of this data transmission path is 8GB / s; if the available output bandwidth of the remote storage device is 25GB / s and the available input bandwidth of its network channel connected to the GPU is 128GB / s, then the available transmission capacity of this data transmission path is 25GB / s, and so on, but not limited to these examples.

[0063] Step S23: Determine the available computing power of the computing unit based on the computing power utilization rate and the total computing power allocated to the computing unit;

[0064] This application reliably monitors the dynamic changes in computing resources of the computing unit. By utilizing the real-time collected computing power utilization rate, it quantifies the actual computing power that the computing unit can use to generate KV Cache, avoiding the problems of computing overload or idle computing resources caused by static computing power allocation. Therefore, this application can achieve step S23 by calculating "available computing power = total computing power allocated to the computing unit × (1 - computing power utilization rate)".

[0065] The total computing power allocated to the computing unit refers to the dedicated computing power reserved for KV Cache recomputation, which can be determined by combining model structure parameters (such as the number of attention heads, hidden layer dimensions, and other parameters related to the computation and generation of KV Cache). Computing power utilization is the load data collected in real time from the computing unit, ensuring that the available computing power is dynamically adjusted according to the load. For example, if the total computing power allocated to the GPU is 100 TFLOPS (Tera Floating-point Operations Per Second, a core indicator for measuring the floating-point operation capability of a computing unit), and the current computing power utilization is 30%, the available computing power is 100 × (1 - 30%) = 70 TFLOPS. The amount of KV Cache recomputation that can be supported changes dynamically with the available computing power.

[0066] Step S24: Based on the storage amount of the key-value cache in each storage unit and the corresponding available transmission capacity, determine the first mapping relationship between the loading amount of the key-value cache in the corresponding storage unit and the parameters to be optimized.

[0067] Based on the above description of the parameters to be optimized, for each data transmission path (i.e., each storage unit), based on its current KV Cache storage volume and current available transmission capacity, a functional relationship is established between the loading volume of the KV Cache in the corresponding storage unit and the parameters to be optimized, denoted as the first mapping relationship, which can be expressed by the following formula (1):

[0068] (1);

[0069] Where i represents the i-th storage unit among multiple storage units, such as i∈{CPU,CXL,DISK}, etc. This can represent the storage amount of the KV Cache in the current i-th storage unit. It can be the current volume of the KV Cache stored in this storage unit, representing the current storage space occupied by this storage unit. Alternatively, it can be the current number of KV pairs stored, i.e., the total number of KV vectors. In subsequent optimization of the allocation strategy, this storage amount serves as the decision variable for the corresponding storage unit, and can take a value between the current storage amount (maximum value) and 0 (minimum value). This can represent the available transmission capacity of the i-th storage unit / i-th data transmission path. Based on this, This can be expressed as the amount of KV Cache loaded into the i-th storage unit under the current available transmission capacity. The parameters to be optimized, such as loading time, are mapped under certain conditions. This application creates an independent first mapping relationship adapted to the actual transmission resources for each data transmission path, providing a reliable basis for subsequent estimations of loading volumes.

[0070] Preferably, this application may also consider the access response latency of each storage unit to determine the parameters to be optimized for different load mappings. This access response latency can be the inherent access latency of the storage unit. The access response latency levels of different levels of storage units can be different, such as ns (nanoseconds) for CPU DRAM and CXL memory (e.g., 100ns), and ms (milliseconds) for SSDs (e.g., 10ms). The access response latency of the i-th storage unit is denoted as... Based on this, the first mapping relationship corresponding to each data transmission path determined in this application can be expressed as:

[0071] + (2);

[0072] In one possible implementation, this application can determine each first mapping relationship through modeling. The amount of data currently stored in the KV Cache of each storage unit, along with the corresponding available transmission capacity and access response latency, can be input into a predefined transmission optimization model to obtain the first mapping relationship between different loads of the KV Cache in the corresponding storage unit and the parameters to be optimized. Here, the transmission optimization model is essentially a template for determining the first mapping relationship. It reflects the mathematical operation relationship as shown in formula (2) above. The model after inputting the corresponding parameters into the model (such as a transmission delay model) represents the first mapping relationship, but it is not limited to this. It can be seen that in the first mapping relationship shown in formula (2), the parameters to be optimized... Loading the KV Cache from the i-th storage unit Loading time / latency (i.e., the result of the first term on the right side of the equals sign in formula (2)) and access response latency The sum of .

[0073] Step S25: Based on available computing power and model structure parameters, determine the second mapping relationship between the key-value cache generated by the computing unit and the parameters to be optimized;

[0074] The model structure parameters may include one or more combinations of attention structure, number of attention heads, head dimensions, data type, and number of model layers, used to deduce the space usage and bandwidth requirements of the KV cache. This application does not limit the content of the model structure parameters. To quantify the cost of regenerating the KV cache (the consumption of parameters to be optimized, such as time consumption, energy consumption, or resource utilization), the computational unit time required for the computing unit (e.g., GPU) to generate a key-value cache unit (a single KV pair) can be determined based on available computing power and model structure parameters. This involves calculating the required computational unit time (the parameters to be optimized; this embodiment only uses time consumption as an example, and the mapping method for other types of parameters to be optimized is similar). For example, based on the model size, structure, and data type, combined with the matrix (KV operation) computing power (which can be determined based on the complexity of matrix operations), the computing power required to generate a single KV pair can be determined. Then, combined with the currently available computing power of the computing unit, the computational time for generating a single KV pair can be determined, denoted as... .

[0075] Next, the number of KV caches that need to be recalculated by the computing unit, which is the maximum computational cost that the current computing unit can generate for KV caches, can be combined with the time taken to generate a single KV pair to determine the time taken to map this maximum computational cost, thus forming the corresponding second mapping relationship. Optionally, the parameters to be optimized for the computational cost mapping can be determined based on the product of the computing unit's time and the candidate values ​​of the computational cost. That is, the second mapping relationship can be expressed by, but is not limited to, the following formula:

[0076] (3);

[0077] in, This can represent the value of the optimization parameters consumed by the GPU in calculating the number of KV caches to be restored using its currently available computing power, such as computation time. For example, if the computing power required to generate a single KV pair is 100 FLOPs, and the GPU's current available computing power is 70 TFLOPS, the latency for generating 1 million KV pairs is: 1 million × 100 / 70 × 10^12 ≈ 1.43ms. It should be understood that the available computing power of a computing unit is a dynamic value, and the computing power required to generate a single KV pair will also change dynamically. Therefore, for the same amount of KV cache calculation, the value of the corresponding optimization parameters will also change accordingly; that is, the second mapping relationship changes dynamically.

[0078] In one possible implementation, this application can still construct a second mapping relationship through modeling, such as inputting the computation unit's time consumption into a predefined computation optimization model to obtain different computational quantities for the current computation unit to generate key-value caches (taking a GPU as the computation unit as an example, the computational quantity can be denoted as...). The second mapping relationship between the computational optimization model and the transmission optimization model mentioned above can be the same type, such as a template used to characterize the second mapping relationship, reflecting the mathematical operation relationship of the above formula (3), and the model after inputting the positions of each variable in the model with the corresponding parameters (such as the computation time / delay model, etc.) to represent the second mapping relationship, but not limited to this.

[0079] Step S26: Based on the total amount of data to be recovered in the key-value cache of the computing unit, the amount of data in the key-value cache of each storage unit and the corresponding available transmission capacity, determine the optimization constraints.

[0080] Step S27: Based on the various mapping relationships and optimization constraints, determine the total parameters to be optimized under the respective candidate values ​​of each loading amount and computational amount;

[0081] Step S28: Based on the total parameters to be optimized, determine the target values ​​for different loading amounts and computational amounts that satisfy the optimization objective, so as to form a target allocation strategy;

[0082] In this embodiment, compared to a single KVCache recovery path that relies on loading the KVCache or entirely on computation to generate the KVCache, a collaborative approach using these two recovery paths is proposed. This involves precise modeling to balance the amount of data loaded directly from each storage unit and the amount of data recalculated by the computation units to generate the KVCache, thus meeting the KVCache recovery requirements. In responding to the KVCache recovery request, the available transmission capacity corresponding to each storage unit / data transmission path and the available computing power of the current computation unit are obtained according to the method described above. First and second mapping relationships adapted to the current corresponding resources are constructed, quantifying the association between the loading amount, computation amount, and the parameters to be optimized. This transforms the subsequent optimization process from empirical allocation to solving an optimization problem. In this way, the optimal allocation for multi-path KVCache recovery is transformed into a joint optimization problem, i.e., a minimax problem under multiple constraints. By deciding the loading amount of the KVCache in each storage unit and the computation amount of the computation units generating the KVCache, load balancing is achieved, improving KVCache recovery efficiency.

[0083] Therefore, the decision variables for the aforementioned joint optimization problem can be the amount of KV cache loaded from each storage unit and the computational cost of generating the KV cache. Taking multiple storage units including CPU memory, CXL extended content, and local disk disk, and the computational unit being the GPU as an example, the decision variables can be expressed as follows: , , , Based on the content of the parameters to be optimized, a corresponding optimization objective is constructed. For example, if the parameter to be optimized is recovery time, the optimization objective can be to minimize the total recovery time, that is, to minimize the maximum recovery time of the KV Cache loading task and the computation task in parallel execution, which is to minimize the total parameter to be optimized and achieve global optimization. In this case, the total parameter to be optimized can refer to the maximum recovery time among the recovery time of loading the corresponding candidate value key-value cache from each storage unit and the recovery time of the computation unit calculating and generating the corresponding candidate value cache key-value (which can be expressed as max( , The objective function of the joint optimization problem, with the goal of minimizing the maximum time consumption, can be expressed as: min(max( , ), i∈{CPU,CXL,DISK}.

[0084] For the aforementioned joint optimization problem, corresponding optimization constraints can be constructed based on current resource and physical topology limitations. These constraints may include, but are not limited to, data integrity constraints (to avoid loading non-existent KV caches), storage capacity constraints, non-negativity constraints (which may include the indivisibility of loading / computing a single KV pair, or even, when the number of KV caches corresponding to a single token is the smallest data unit (data processing granularity), the KV cache corresponding to each token cannot be indivisibly loaded or computed), and resource upper limit constraints (such as transmission capacity constraints and computing power constraints). These constraints constrain the values ​​of each decision variable during the allocation optimization process, ensuring a balance between the feasibility of the solution and the optimization objective. This prevents allocation strategies from exceeding actual capabilities or violating data integrity requirements, ensuring that the target allocation strategy achieves the optimization objective while satisfying all resource constraints, and maximizing KV cache recovery efficiency, such as minimizing the time required to recover all KV caches. The optimization objective can be determined based on the task requirements of the actual scenario; this application only uses this as an example for illustration.

[0085] In the process of finding the optimal solution to the joint optimization problem, i.e., the decision objective allocation strategy, optimization methods such as uniform sampling or adaptive sampling (prioritizing sampling the interval with a high probability of being optimal) can be used to determine the candidate values ​​of each decision variable and the values ​​of the parameters to be optimized (such as recovery time). For example, the candidate values ​​corresponding to the loading amount of KV Cache in each storage unit and the candidate values ​​of the computation amount of the computing unit to generate KV Cache are combined into a candidate combination. Combined with the values ​​of the parameters to be optimized mapped to each, the total parameters to be optimized of the candidate combination are determined, thereby reducing the computational amount of the candidate combination quantization evaluation and improving the efficiency of quantization evaluation.

[0086] Subsequently, based on the comparison results of the total parameters to be optimized for each of the multiple candidate combinations, a candidate combination that satisfies the optimization objective can be determined. This means that the target values ​​of each loading quantity and the target value of the computational quantity satisfy the optimization objective, thus forming a feasible objective allocation strategy. During the optimal solution selection process, if multiple candidate combinations that meet the optimization objective exist, such as multiple optimal solutions with the same total recovery time, secondary objectives (such as maximizing resource utilization) can be added for further selection, without any restrictions.

[0087] In the process of solving the optimization problem under the above-mentioned optimization constraints, the solution can be based on the PuLP library of Python, and the optimal strategy, i.e. the target allocation strategy, can be output. For example, the optimal allocation amount of KV Cache needs to be loaded from each storage unit (target value) and the number of KV Caches that the GPU needs to recalculate (target value) needs to generate. This application does not restrict the method of determining the optimal allocation of the target allocation strategy.

[0088] Step S29: Based on the target allocation strategy, execute the loading task of loading key-value cache from each storage unit and the calculation task of calculating key-value cache from the computing unit to obtain the target key-value cache set.

[0089] The implementation process of step S29 can be referred to the detailed description of the corresponding embodiment in the context, and will not be repeated here. Optionally, this application can generate corresponding scheduling instructions based on the target allocation strategy to start the KV Cache loading and calculation generation task. While the GPU uses the current computing resources to recalculate and generate KV, the KV Cache in each storage unit is synchronously transmitted through each data transmission path, thereby achieving the masking of calculation and transmission delays and minimizing the overall KV Cache recovery delay. This application will not describe in detail the resource scheduling process based on the target allocation strategy during task execution.

[0090] In summary, the model inference acceleration method proposed in this application can adapt to heterogeneous storage scenarios, supporting, but not limited to, various storage units such as CPU DRAM, CXL, SSD, and remote storage devices. Depending on actual needs, the computer device can also connect to other external storage devices as storage units for the KV Cache corresponding to historical tokens, reducing reliance on high-end graphics cards with large-capacity video memory and lowering overall hardware deployment costs at the system level. In this case, the corresponding first mapping relationship and optimized constraints can be constructed according to the method described above without modifying the core logic, demonstrating strong scalability. Moreover, this application dynamically adapts to resource changes, i.e., it collects dynamic resource parameters in real time and dynamically adjusts the corresponding mapping relationships and optimized constraints to adapt to fluctuations in resource load during model inference, thereby dynamically adjusting the KV Cache allocation strategy to avoid resource waste and improve model inference speed and interactive experience.

[0091] Among them, the resource parameters involved in this application, such as bandwidth, computing power, and latency, are all quantitative indicators that can be collected by hardware and calculated by the model. The above mapping relationship and optimization constraints can be implemented by programming in languages ​​such as Python, so as to be directly integrated into mainstream inference frameworks. Based on the corresponding mainstream inference frameworks, the task processing requirements in the corresponding scenarios can be met, such as significantly improving the concurrency capability, response speed and overall system throughput in long context inference scenarios, and meeting the real-time response requirements.

[0092] In some embodiments, after determining the target KV Cache set, i.e. the target Key Cache subset and the target Value Cache subset, according to the method proposed in the above embodiments, the computing unit (such as GPU) can use the input information (Query, query vector) to be processed, reuse these Value Caches restored to the video memory, quickly complete the attention calculation task, and obtain the attention output for the query vector, which is used to generate the final result of model inference, such as generating the next token. The implementation process of the attention calculation task is not described in detail in this application.

[0093] Preferably, in order to further reduce request latency, multiple computing units can be used to collaboratively complete the attention computation task during the decoding stage, so as to make full use of heterogeneous computing resources, effectively reduce attention computation latency, and improve overall throughput. Therefore, this application can decompose the complete multi-head attention computation task according to the attention head granularity based on the structural characteristics of the independent computation between attention heads, construct a set of parallelizable subtasks, without cross-task dependencies and without complex task synchronization mechanisms, and provide a task slicing basis for the subsequent heterogeneous computing scheduling implemented by multiple computing units, so as to flexibly allocate the independent subtasks to different computing units based on the real-time attention computation resources of each computing unit.

[0094] Among them, multiple computing units refer to multiple hardware units with independent computing power, such as GPU clusters, multiple GPU nodes, heterogeneous combinations of GPUs and CPUs or other AI chips, and heterogeneous combinations of GPUs, CPUs and other AI chips. Each computing unit has its own dedicated cache access channel and attention computing resources, such as the SM core of the GPU and the computing threads of the CPU. This application does not limit this, but only uses two types of computing units (the first computing unit and the second computing unit) as examples to illustrate the heterogeneous attention computing process.

[0095] At this point, the target key-value cache set can include a first KV Cache subset and a second KV Cache subset. The first KV Cache subset can be obtained by the first computing unit (such as a GPU) through loading tasks and by the KV Cache generated and loaded through computing tasks. In other words, it is a locally available cache set obtained by the first computing unit integrating the KV Cache loaded from multiple storage units and the recalculated KV Cache, without the need for cross-unit transfer, resulting in the lowest access latency. The second KV Cache subset can be obtained by the second computing unit (such as a CPU) directly accessing (e.g., DMA) at least one storage unit without going through the first computing unit. In this case, based on the target key-value cache set, the attention output of the model is obtained by executing different assigned subtasks through the first computing unit and at least one second computing unit.

[0096] Therefore, the target key-value cache set obtained in this application includes the KV Cache directly loaded by the first computing unit from each storage unit, the recalculated KV Cache, and at least one KV Cache directly accessed by the second computing unit. The KV Cache accessed by the second computing unit can still be cached in the corresponding storage unit so that the required KV Cache can be read from that storage unit for attention calculation. Based on this, refer to Figure 3 This is a flowchart illustrating the model inference acceleration method proposed in Embodiment 3 of this application. This embodiment describes the implementation process of how to perform multi-head attention computation after obtaining the target KV Cache set through the method in the above embodiments. Figure 3 As shown, the implementation process may include:

[0097] Step S31: Obtain the attention computing resources of each of the multiple computing units.

[0098] Step S32: Decompose the multi-head attention computation task into different sub-tasks, each sub-task corresponding to the computation of one attention head;

[0099] Step S33: Based on the attention computing resources of each of the multiple computing units, determine the sub-tasks allocated to each of the multiple computing units;

[0100] Step S34: Based on the obtained target key-value cache set, the subtasks assigned to each unit are executed by multiple computing units to obtain attention output.

[0101] In this embodiment, the available computing resources for attention computing in each computing unit can be monitored in real time according to the current attention computing mode and data type, and recorded as attention computing resources, such as GPU SM utilization, CPU idle core count, etc. If necessary, dynamic resource status such as GPU local cache bandwidth, CPU direct access latency and other cache access capabilities, and historical task load such as the execution time of the previous round of attention computing can also be obtained to improve the accuracy and reliability of subtask dynamic allocation.

[0102] During task decomposition, this application splits tasks at the "attention head" granularity. Each subtask corresponds to a complete computation of an attention head, such as similarity calculation between the query and key-value pairs, weighted summation, etc. The number of subtasks is consistent with the total number of attention heads in the model. Subsequently, subtasks are dynamically allocated based on the current attention resources of each computing unit. For example, "high-complexity subtasks (such as long sequence attention head computation tasks)" are prioritized for allocation to computing units with sufficient attention computation resources, and subtasks that rely on access to a subset of the second key-value cache are allocated to computing units with high bandwidth for transmission paths to storage units. However, this principle of subtask allocation is not limited to this.

[0103] In one possible implementation, this application can determine the unit throughput of single-head attention computation for each computing unit (such as GPU and CPU) based on the attention computation resources of each computing unit (such as GPU and CPU). That is, the computational throughput for single-head attention (such as matrix operation throughput). Then, according to the ratio between the computational throughput of each computing unit, multiple subtasks are allocated to different computing units. For example, the optimal allocation strategy of subtasks is determined according to the inverse ratio of the computational throughput of different computing units, that is, the number of subtasks allocated to each computing unit. This achieves the optimal allocation of computing load among heterogeneous devices, avoids a single computing unit from becoming the performance bottleneck of the overall execution, and also avoids the GPU being idle while waiting for the CPU to execute subtasks, or the CPU slowing down the GPU's computing efficiency, thereby achieving load balancing.

[0104] For example, if the model contains 32 attention heads, the first computing unit (GPU) has sufficient attention computing resources (e.g., SM utilization of 30%) and holds a subset of the first KV Cache, allocating subtasks for 18 attention heads; two second computing units (e.g., CPU and other AI chips) each hold high-bandwidth access channels (e.g., CXL memory access link bandwidth of 5GB / s), and each second computing unit is allocated subtasks for 7 attention heads. All subtasks can be started and executed simultaneously without resource contention. Compared to the first computing unit executing all subtasks (a complete multi-head attention computing task), this embodiment proposes a heterogeneous parallelization of attention computing through multiple computing units, realizing pipelined collaboration across computing units, making full use of heterogeneous computing resources, reducing attention computing latency, and improving overall throughput.

[0105] Therefore, during the collaborative parallel execution of multiple subtasks, the first computing unit can execute the assigned subtask based on the first KV Cache subset without cross-unit or storage access, leveraging the low latency advantage of local (e.g., video memory) caching to quickly complete the computation. Simultaneously, at least one second computing unit can access the corresponding second KV Cache subset to execute the assigned subtask. This allows the second computing unit to start its assigned subtask while simultaneously accessing the KV Cache belonging to the second KV Cache subset (at least one KV Cache corresponding to a historical token) in the storage unit via an independent transmission path. This enables parallel execution of KV Cache access and attention computation, masking transmission latency. The number of second KV Cache subsets is the same as the number of second computing units, thus determining the number of KV Cache subsets into which the target KV Cache set is divided, without any limitation.

[0106] Taking a GPU as the first computing unit and a CPU as the second computing unit as an example, these two computing units read the KV cache required for their respective attention calculations through independent data channels (transmission links), avoiding interactive blocking and maximizing the parallelism of multi-head attention calculations. During the execution of each subtask, each computing unit synchronizes its attention calculation progress through shared memory or high-speed interconnect networks (such as NVLink), ensuring that the outputs of different attention heads are aligned in the time dimension, providing a foundation for subsequent result concatenation. It is evident that this multi-computing unit collaborative attention calculation shortens the computational latency, significantly reducing TTFT (Time To First Token, the time interval from the user's complete input request (such as the long text prompt word) to the model generating the first output token, which directly determines the user's initial perception of "response speed") and TPOT (Time Per Output Token, the time per token generation, which can also be the average time interval between two adjacent tokens, reflecting the smoothness of long text generation; a high TPOT will make the user feel that the output is "discontinuous") during long text inference, reducing the overall inference latency for long texts and improving the user's interactive experience.

[0107] Optionally, each KV Cache subset can be dynamically adjusted based on real-time monitored access frequency. For example, the frequently accessed second KV Cache subset can be dynamically migrated to the first KV Cache subset, or vice versa, to optimize access efficiency. It should be understood that when computing units expand, such as adding a third computing unit, a corresponding KV Cache subset can be added through the storage mapping table (storage distribution relationship) of the target KV Cache set. This includes directly accessing unrecovered KV Caches in dedicated storage units, dynamically adjusting the subtask allocation strategy according to the method described above. The implementation process is not detailed in this application.

[0108] Among them, the attention calculation result O obtained by performing subtasks in each computing unit j The attention result output by the j-th computation unit can be concatenated according to the order of attention heads to generate a complete attention output, thus completing the overall closed loop of cross-device attention computation. In other words, the KV Cache generated during the attention computation process can be migrated to multiple storage units for storage, and then loaded into the internal storage unit in the next round of inference according to the above method to continue attention computation.

[0109] In some embodiments, the KV Cache in each of the storage units listed above can be obtained from the storage of historical data generated during the model inference process, i.e., the intermediate data generated by computing historical tokens, including the key vector and value vector generated by attention calculation. In order to release the occupation of the storage space (such as video memory resources) of the computing unit (such as the first computing unit GPU mentioned above), so that the first computing unit can bear a larger batch processing scale and more concurrent requests, and significantly improve the multi-session concurrency capability, this application can split these KV Caches into the smallest splitting unit according to the KV Cache corresponding to a single token, and then unload them in parallel to multiple external (relative to video memory) storage units, such as CPU memory, CXL memory, local hard disk and remote storage devices, etc., so as to fully aggregate the bandwidth of heterogeneous storage units, improve the KV Cache unloading efficiency, effectively break through the performance bottleneck caused by the limited bandwidth of a single storage unit, support longer input sequence length and context window under the same hardware conditions, and make full use of the storage resources of external storage units, which can reduce the dependence on high-end graphics cards with large capacity video memory and reduce the overall hardware deployment cost at the system level.

[0110] Based on the above analysis, and referring to Figure 4 This is a flowchart illustrating the model inference acceleration method proposed in Embodiment 4 of this application. This embodiment describes a possible implementation method of the storage process of the KV Cache in each storage unit of the method in the above embodiments, such as... Figure 4 As shown, the stored procedure may include:

[0111] Step S41: Based on the model structure parameters and the number of historical lexical units obtained by model inference, determine the total amount of data to be stored in the key-value cache from the computing unit.

[0112] In this embodiment, after a round of token calculation is completed in model inference, in order to reuse the KV Cache generated in this round of calculation and reduce attention computation, in response to the KV Cache unloading request, the number (size, denoted as KV Cache data unit) of KV Cache corresponding to each token can be determined based on the model structure parameters and the number of historical tokens. This determines the total amount of KV Cache data to be unloaded (such as the product of the number of historical tokens and the number of KV Cache data units), which is usually the KV Cache corresponding to all historical tokens, to avoid blind migration of KV Cache in the future and to ensure that the amount of KV Cache data unloaded matches the storage capacity of multiple storage units.

[0113] Step S42: Obtain the available input bandwidth and available storage capacity of each of the multiple storage units.

[0114] To avoid overloading the migration of KV Cache to each storage unit and prevent excessive bandwidth consumption from affecting inference performance, this application can dynamically sense the system load, such as real-time monitoring of the computing units (computing units of the KV Cache that are not currently being migrated, such as GPUs) and the current bandwidth utilization of each storage unit, thereby determining the available input bandwidth of each storage unit. At the same time, it can monitor the available storage capacity of each storage unit, or determine the capacity based on the detected capacity utilization and the total allocated capacity, etc. This application does not limit the implementation method of step S42.

[0115] Step S43: Based on the total amount of key-value cache data to be stored, the available input bandwidth, and the available storage capacity, determine the storage allocation strategy for multiple storage units; the storage allocation strategy includes the target value of migrating the amount of key-value cache data to each of the multiple storage units.

[0116] Step S44: Based on the storage allocation strategy, the key-value caches to be stored in the computing unit are migrated to different storage units respectively.

[0117] This application aims to migrate all key-value caches to be stored (i.e., unloaded) to multiple external storage units with the fastest migration efficiency (minimizing total migration time). This can be achieved by combining the storage and transmission resources of each storage unit to optimize the allocation of the total amount of data in the key-value caches to be stored. Specifically, it determines how to allocate the data in the shortest time and sets a target value for the amount of key-value cache data to be stored in each storage unit. Therefore, this application transforms the key-value cache migration and allocation into an optimization problem (e.g., constructing a corresponding linear programming model). Using the available input bandwidth and available storage capacity of each storage unit, as well as the total amount of data in the key-value caches to be stored, appropriate migration constraints are determined, such as data integrity, storage capacity, and parallel write rate constraints. The goal is to minimize the total migration time, and a storage allocation strategy that satisfies this optimization objective is determined. The implementation process is not detailed in this application.

[0118] Subsequently, the KV Cache migration task can be executed in parallel, migrating all the KV Caches that need to be migrated to different storage units. For example, based on a multi-stream mechanism, the KV Cache data in the GPU memory can be migrated to multiple storage units in parallel, maximizing the utilization of the parallel bandwidth of each storage unit, reducing the actual migration time and inference waiting time, and achieving adaptive scheduling and efficient utilization of storage resources. It should be noted that KV Caches that need to be unloaded in other stages of model inference can also be migrated to external storage units according to the method described above; the implementation process is not detailed in this application.

[0119] As can be seen, multiple storage units constitute a KV Cache storage pool. The KV Cache is unloaded at each stage of distributed storage to ensure data consistency and integrity. This KV Cache storage pool can receive and respond to KV Cache read and write requests, and supports data migration and parallel access between multiple storage units.

[0120] In some embodiments, to avoid latency spikes caused by temporarily loading the KV cache during model inference, the necessary KV cache can be preloaded into high-speed storage units (such as GPU memory) before inference, as described in the embodiments above regarding the loading process of direct loading and recomputation to generate a collaborative implementation. Prior to this, the amount of KV cache data stored in each external storage unit can be dynamically adjusted as needed, especially when the bandwidth of each storage unit changes dynamically after migration. The data distribution relationship of the KV cache in the storage pool can be adjusted according to, but not limited to, the KV cache warm-up method described below.

[0121] Reference Figure 5 This is a flowchart illustrating the model inference acceleration method proposed in Embodiment 5 of this application. The method in this embodiment can be executed before restoring the KV Cache to the high-speed storage unit in the method described above. Figure 5 As shown, the method proposed in this embodiment may include:

[0122] Step S51: Determine the amount of data for preheating the KV Cache based on the model structure parameters and the number of inference tokens in the model.

[0123] Step S52: Determine the initial data distribution relationship of the preheated KV Cache in multiple storage units;

[0124] In this embodiment, the amount of KV Cache data to be preheated can be determined by parsing the KV Cache preheating request. This can be done by counting the number of tokens and calculating the total amount of KV Cache data to be preheated based on the model structure parameters, thus accurately positioning the preheating range and avoiding excessive resource consumption during preheating or insufficient preheating requiring temporary loading. Then, by traversing the storage units where the KV Cache corresponding to each token resides, the number of tokens corresponding to the KV Cache in each storage unit (i.e., the amount of KV Cache data currently stored) can be counted, generating a corresponding storage mapping table, i.e., the initial data distribution relationship, to clarify the current KV Cache distribution and avoid subsequent duplicate loading or omissions.

[0125] Step S53: Based on the available bandwidth of each of the multiple storage units, the amount of data in the preheated KV Cache, and the initial data distribution relationship, the preheated KV Cache is transmitted in parallel between the multiple storage units to determine the distribution relationship of the preheated KV Cache data stored in each of the multiple storage units, which is used to determine the mapping relationship between the loading amount of the KV Cache loaded from the corresponding storage unit and the parameters to be optimized.

[0126] In one possible implementation, this application can use linear programming to adjust the amount of KV Cache data currently stored in each storage unit to minimize the total warm-up time and fully utilize the read capabilities of each storage unit, avoiding any storage unit becoming a read bottleneck. Therefore, this embodiment uses the data integrity of the KV Cache to be warmed up, the capacity of each storage unit, and the read rate as warm-up constraints to establish a mapping relationship between the KV Cache storage amount in each storage unit and the warm-up time (or other warm-up optimization parameters, such as read / write utilization, read / write rate, etc.) (denoted as the third mapping relationship, which can be represented by a linear rule model). Then, based on the optimization objective of the warm-up time (such as minimizing parallel loading time) and the initial data distribution relationship, a warm-up allocation strategy that satisfies the optimization objective can be determined, i.e., the target value of the KV Cache storage amount in each storage unit.

[0127] Subsequently, based on the preheating allocation strategy, a CPU multi-threaded asynchronous mechanism can be used to transmit KV Cache in parallel between multiple storage units, readjust the initial data distribution relationship of KV Cache, and obtain a new data distribution relationship (such as a new data mapping table). This new data distribution relationship is determined as the preheating KV Cache data distribution relationship stored by each of the multiple storage units. Then, based on this, a first mapping relationship between the loading amount of KV Cache loaded by each storage unit and the parameters to be optimized can be constructed. For example, based on this data distribution relationship, the amount of KV Cache data (storage amount) already stored in each storage unit can be obtained, which is convenient for calculating the loading time in conjunction with the bandwidth parameters. This maximizes the utilization of the parallel bandwidth between multiple storage devices, reduces the total preheating time, and ensures that KV Cache can be loaded directly at high speed during inference. The loading process can be referred to the description of the corresponding part of the above embodiment, and will not be repeated in this embodiment.

[0128] In summary, referring to Figure 6 The flowchart shown illustrates how this application can improve model inference speed and user experience through the KVCache unloading, warm-up, loading, and heterogeneous attention calculation processes described in the above embodiments. For example, Table 1 records the contents of multiple external storage units and their bandwidth for testing purposes.

[0129] equipment bandwidth Single-player configuration with 8 GPUs, PCIe 5.0 x16 512GB / s CPU and memory, dual-channel PCIe 5.0 x16 128GB / s CXL Extended Memory 34GB / s Local hard drive 8GB / s Remote storage 25GB / s

[0130] Table 1

[0131] According to the method of parallel offloading of KV Cache across multiple storage units proposed above in this application, the bandwidth of these storage units can be utilized simultaneously to obtain an aggregate bandwidth of 195 GB / s. Ignoring the impact of individual access response latency, compared to the traditional serial offloading scheme (such as migrating all data to CPU memory), this application improves the data transfer rate of the same amount of KV Cache across multiple storage units by approximately (195 - 128) / 128 = 52%. At this time, the KV Cache offloading and loading time can be shortened by approximately 1 - 128 / 195 = 34%. Specifically, if the proportion of KV Cache loading at the target latency is 0.3, the TTFT can be reduced by 10.2%, and the throughput can be increased by 11.36%; if the proportion is 0.5, the TTFT can be reduced by 17%, and the throughput can be increased by 20.49%; if the proportion is 0.8, the TTFT can be reduced by 27.2%, and the throughput can be increased by 37.36%, etc. As can be seen, the improvement effect achieved varies depending on the proportion of KV cache loading to the target latency, and can be determined according to the actual scenario requirements.

[0132] Furthermore, according to the scheme of this application, if the GPU participates in 30% of the KV Cache calculation and generation, the total recovery time of the KV Cache can be further shortened to 1 – 128 * (1 - 30%) / 195 = 54%. Furthermore, by employing GPU and CPU collaboration to accelerate attention calculation, if the CPU is responsible for 20% of the attention calculation, this collaborative method can shorten the attention calculation time by 20%, and the decoding rate can be increased by 16%, greatly improving the model inference speed.

[0133] Reference Figure 7 This is a schematic diagram of the structure of a model inference acceleration device provided in an embodiment of this application. Figure 7 As shown, the model inference acceleration device may include:

[0134] The resource acquisition module 71 is used to acquire the available computing power of the current computing unit for calculating and generating key-value cache, as well as the available transmission capacity of the data transmission path connecting multiple storage units.

[0135] The mapping relationship determination module 72 is used to determine, based on the available computing power and the available transmission capacity, the mapping relationship between the loading amount of the key-value cache in each of the storage units, the computing amount of the computing unit to generate the key-value cache, and the parameters to be optimized; the parameters to be optimized characterize the recovery efficiency of the key-value cache;

[0136] The target allocation strategy determination module 73 is used to determine the target allocation strategy based on the mapping relationship and the optimization target of the parameter to be optimized; the target allocation strategy includes the target values ​​of the loading amount and the computing amount respectively;

[0137] The task execution module 74 is used to execute a loading task of loading key-value cache from each of the storage units based on the target allocation strategy, and to perform a calculation task of generating key-value cache with the computing unit to obtain a target key-value cache set.

[0138] In one possible implementation, the mapping determination module 72 may include:

[0139] The first mapping relationship determination unit is used to determine the first mapping relationship between the loading amount of the key-value cache in the corresponding storage unit and the parameter to be optimized based on the storage amount of the key-value cache in each storage unit and the corresponding available transmission capacity.

[0140] The second mapping relationship determination unit is used to determine the second mapping relationship between the key-value cache calculated and generated by the computing unit and the parameters to be optimized, based on the available computing power and model structure parameters.

[0141] In one possible implementation, the target allocation strategy determination module 73 may include:

[0142] The optimization constraint determination unit is used to determine optimization constraints based on the total amount of data to be recovered in the key-value cache of the computing unit, the amount of data in the key-value cache of each storage unit and the corresponding available transmission capacity;

[0143] The unit for determining the total parameters to be optimized is used to determine the total parameters to be optimized under the respective candidate values ​​of the loading amount and the computation amount based on the mapping relationships and the optimization constraints.

[0144] The target value determination unit is used to determine the target values ​​of different loading amounts and computation amounts that satisfy the optimization objective based on the total parameters to be optimized, so as to form a target allocation strategy.

[0145] Optionally, if the parameter to be optimized is the recovery time, the total parameter to be optimized may refer to the maximum recovery time among the recovery time of loading the corresponding candidate value key-value cache from each of the storage units and the recovery time of the computing unit calculating and generating the corresponding candidate value cache key-value, and the optimization objective is to minimize the maximum recovery time.

[0146] Based on this, the mapping relationship determination module 72 mentioned above may include:

[0147] The first obtaining unit is used to input the amount of data currently stored in the key-value cache of each storage unit, as well as the corresponding available transmission capacity and access response delay, into a predefined transmission optimization model to obtain a first mapping relationship between the different loading amounts of the key-value cache in the corresponding storage unit and the parameters to be optimized.

[0148] The time consumption determination unit is used to determine the computing unit consumption time required for the computing unit to calculate and generate the key-value cache unit based on the available computing power and model structure parameters.

[0149] The second obtaining unit is used to input the time consumed by the computing unit into a predefined computing optimization model to obtain a second mapping relationship between the different computing amounts of the key-value cache generated by the current computing unit and the parameters to be optimized.

[0150] In some embodiments, the resource acquisition module 71 may include:

[0151] The acquisition unit is used to acquire the computing power utilization rate of the current computing unit, the available output bandwidth of each of the multiple storage units, and the available input bandwidth of the data transmission path of each of the multiple storage units connected to the computing unit.

[0152] An available transmission capacity determination unit is used to compare the available input bandwidth of each of the data transmission paths with the available output bandwidth of the corresponding storage unit to determine the available transmission capacity of the corresponding data transmission path.

[0153] Available computing power determination unit, used to determine the available computing power of the computing unit based on the computing power utilization rate and the total computing power allocated to the computing unit.

[0154] Based on the above analysis, if there are multiple computing units, the target key-value cache set includes the key-value cache loaded by the first computing unit through the loading task, the first key-value cache subset obtained by calculating and loading the key-value cache through the computing task, and the second key-value cache subset obtained by directly accessing at least one of the storage units.

[0155] The attention calculation module is used to execute different sub-tasks decomposed into a multi-head attention calculation task based on the target key-value cache set, through the first calculation unit and at least one second calculation unit, to obtain the attention output of the model; each sub-task corresponds to the calculation of an attention head, and the sub-task is allocated based on the current attention calculation resources of each calculation unit;

[0156] Optionally, the attention calculation module may include:

[0157] The subtask collaborative execution unit is used to execute the assigned subtask based on the first key-value cache subset by the first computing unit, and to execute the assigned subtask by accessing the corresponding second key-value cache subset by at least one second computing unit.

[0158] In some embodiments, the above-described model inference acceleration device may further include:

[0159] The preheating data volume determination module is used to determine the amount of data in the preheating key-value cache based on the model structure parameters and the number of inference tokens in the model;

[0160] The data distribution relationship determination module is used to determine the distribution relationship of the hot key value cache data stored in each of the multiple storage units by transmitting the preheating key value cache in parallel among the multiple storage units based on the available bandwidth of each of the multiple storage units and the amount of data in the preheating key value cache;

[0161] The hotkey value cache data distribution relationship is used to determine the mapping relationship between the amount of key value cache loaded from each of the storage units and the parameters to be optimized.

[0162] The key-value cache in each of the storage units is obtained by storing historical data generated during the model inference process, and the storage module for storing the key-value cache may include:

[0163] The total data volume determination unit is used to determine the total amount of key-value cache to be stored from the computing unit based on the model structure parameters and the number of historical lexical units obtained by model inference.

[0164] The storage resource acquisition unit is used to acquire the available input bandwidth and available storage capacity of each of the current multiple storage units.

[0165] A storage allocation strategy determination unit is used to determine a storage allocation strategy for the plurality of storage units based on the total amount of data to be stored in the key-value cache, the available input bandwidth, and the available storage capacity; the storage allocation strategy includes a target value for migrating the amount of key-value cache data to the plurality of storage units respectively;

[0166] The migration unit is used to migrate the key-value cache to be stored in the computing unit to different storage units based on the storage allocation strategy.

[0167] This application also provides a computer program product including computer-readable instructions. When the computer-readable instructions are executed on a computer device, the computer device implements any of the model inference acceleration methods provided in this application. The computer program product can be stored in a readable storage medium, such as a computer floppy disk, USB flash drive, external hard drive, ROM (Read-Only Memory), RAM (Random Access Memory), magnetic disk, or optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, training device, or network device, etc.) to execute the model inference acceleration methods described in the various embodiments of this application.

[0168] This application also provides a computer-readable storage medium carrying one or more computer programs. When these programs are executed by a computer device, the computer device can implement any of the model inference acceleration methods provided in this application. This application does not limit the product form of the computer-readable storage medium.

[0169] Reference Figure 8 This is a schematic diagram of the hardware structure of a computer device according to an embodiment of this application. The computer device may include at least one data transmission port 81 and a processing device 82. The data transmission port 81 is configured to connect to a storage unit to realize the transmission of KV Cache data between the storage unit and the storage unit. The processing device 82 is configured to implement the various steps of the model inference acceleration method proposed in the embodiments of this application. The implementation process can be referred to the description of the corresponding parts of the above embodiments, and will not be repeated here.

[0170] The storage unit can be one or more combinations of, but not limited to, CPU memory, CXL extended memory, local hard disk, or remote storage devices listed above. It is hardware used to store the KV cache and can be a storage device already deployed in the computer equipment or an external storage device connected via a wireless or wired network. These storage units are all external storage devices for the video memory of the computing unit. The number or type of storage units can be expanded as needed. Different data transmission protocols can be used for different types of storage units. Data transmission ports 81 supporting the corresponding data transmission protocols are connected via wired or wireless communication. This application does not limit the communication connection method between each data transmission port 81 and multiple storage units.

[0171] Reference Figure 9The system architecture diagram shown illustrates that multiple storage units can form a KV cache storage pool, such as a distributed multi-level storage architecture. This pool is responsible for the KV cache being offloaded at each stage of the distributed storage, ensuring data consistency and integrity, receiving and responding to read and write requests from the cache engine, and supporting data migration and parallel access between multiple storage devices. The implementation process can be referred to the corresponding description in the above embodiments. It should be noted that... Figure 9 The multiple storage units are merely an optional example of a storage pool; different storage units can store different tokens (which can be...). Figure 9 The storage size relationship between different KV caches (such as 1, 2, 3, and 4, but not limited to these) can be represented by the length of the box labeled with the number, but is not limited to this.

[0172] In one possible implementation, the processing device 82 may include a cache engine 821 and a model inference engine 822, which work together to implement the model inference acceleration method proposed in this application. Wherein, as Figure 9 As shown, the cache engine 821 can perceive the system load status in real time, monitor the available bandwidth and access latency of the first computing unit (such as the GPU) and various storage units, as well as the computing power utilization of each computing unit, providing a basis for subsequent scheduling and optimization. Based on the real-time perceived load status, a corresponding mapping relationship is constructed to determine the allocation strategy for different nodes, and to execute the scheduling of parallel unloading and loading of KV cache between the GPU and external storage devices, and the scheduling of GPU and CPU collaborative parallel computing attention. In the heterogeneous attention computing process, tasks are split according to single-head attention, and the total query vector Query is split to obtain the sub-query vectors required for each subtask, such as... Figure 9 The query vector Query1 for attention computation performed by the GPU and the query vector Query2 for attention computation performed by the CPU, but not limited to these.

[0173] Since the KV cache processed at each stage is processed according to the token dimension, such as Figure 9 The unloading node splits all the KV caches that need to be unloaded into different subsets. Figure 9 The dashed lines represent at least one KV cache corresponding to each of the tokens 1, 2, 3, and 4. The KV cache transmitted between multiple storage units during the warm-up phase is also a KV cache for at least one token. Figure 9The KV cache of some tokens (represented by number 3) in the local hard drive is migrated to CPU memory, and the KV cache of some tokens (represented by number 4) in the remote storage device is migrated to CXL memory, but this is not limited to this. Similarly, during the loading phase, when the GPU recalculates and generates the KV cache, it can determine that numbers 5 and 6 are tokens that need to be recalculated in order to calculate and generate the corresponding KV cache (the historical KV cache of these two parts of the historical tokens (such as...)). Figure 9 The KV cache in region 5 of the CPU memory and the KV cache in region 6 of the CXL memory are still stored in their respective memory locations and are written to the GPU memory without needing to be loaded or deleted. Figure 9 The regions corresponding to numbers 5 and 6 in the entire recovered KV cache, as well as other regions (such as the region corresponding to number 3, and the region corresponding to the first part of the KV cache in CPU memory), are directly loaded from the storage unit.

[0174] In a heterogeneous attention computing scheme employing GPUs and CPUs, the CPU can directly access the KV cache in memory units, such as the second part of the KV cache in the CPU's memory and the entire KV cache in remote storage devices (e.g., the KV cache in the region corresponding to label 4). This allows the CPU to execute the assigned subtasks and obtain the attention output O2. Simultaneously, the GPU utilizes the recovered KV cache (e.g., ... Figure 9 In the KV cache area of ​​the model inference engine, the assigned subtask is executed to obtain the attention output O1. The attention output is then concatenated according to the order of the attention heads to obtain the final attention output. It should be noted that other computing units can also be scheduled to implement the attention calculation stage. The heterogeneous computing process is similar, and this application will not provide detailed examples of each one.

[0175] Regarding the workflow of the model inference engine in the system architecture, based on the relevant descriptions in the above method embodiments, it can be responsible for generating and temporarily storing the KV cache during the prefill phase, forming an unload request to transfer the KV cache to each storage unit. Based on the batch of tokens to be calculated (the number of tokens to be calculated in the next round of inference), a preheating request is sent to the cache engine, following the steps outlined above. Figure 5The method described in the corresponding embodiment is used to pre-adjust the data distribution relationship of the KV cache that needs to be restored in the storage pool. It can also initiate a KV cache loading request based on context dependencies and drive the GPU to calculate and generate part of the KV cache that needs to be restored. In conjunction with the description of the corresponding parts of the above embodiments, during the execution of this computation task, the cache engine will execute the KV cache loading task between the GPU and multiple storage units based on the target allocation strategy. After the KV cache is restored to the GPU memory, the model inference engine can also drive the GPU to calculate the attention output using the query vector and integrate the attention output calculated by the CPU to obtain the final attention output, which is used to complete subsequent inference tasks.

[0176] Furthermore, the computer device may also include at least one memory, which can be used to store programs / instructions required for executing the model inference acceleration method, so that the processor executes the programs / instructions to implement the model inference acceleration method proposed in the embodiments of this application. Optionally, the memory may also store models deployed on the edge side to perform inference tasks.

[0177] In addition, the computer device may include at least one input component, such as an audio acquisition device, stylus, mouse and keyboard or joystick, for users to operate the input component to input input information (Qurey) representing task requirements to the computer device. The computer device may also include at least one output component, such as an audio player or display screen, for outputting response information (final inference result) to the input information; it may also include acquisition tools / sensors for real-time acquisition of external storage unit performance parameters, such as read and write bandwidth, access latency and available capacity.

[0178] It should be understood that, Figure 8 The structure of the computer device shown does not constitute a limitation on the computer device in the embodiments of this application. In practical applications, the computer device may include more than Figure 8 The more or fewer components shown, or combinations of certain components, are not listed in detail in this application.

[0179] Finally, it should be noted that the device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. In addition, in the accompanying drawings of the device embodiments provided in this application, the connection relationship between modules indicates that they have a communication connection, which can be implemented as one or more communication buses or signal lines.

[0180] In the above embodiments, the invention can be implemented entirely or partially by software, hardware, firmware, or any combination thereof. Through the description of the above embodiments, those skilled in the art can clearly understand that this application can be implemented using software plus necessary general-purpose hardware, or it can be implemented using dedicated hardware including dedicated integrated circuits, dedicated CPUs, dedicated memory, dedicated components, etc. When implemented using software, it can be implemented entirely or partially in the form of a computer program product. The various embodiments in this specification are described in a progressive or parallel manner, with each embodiment focusing on the differences from other embodiments. Similar or identical parts between embodiments can be referred to mutually. For the apparatus and computer equipment disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the description is relatively simple; relevant parts can be referred to in the method section.

Claims

1. A method for accelerating model inference, the method comprising: Obtain the available computing power of the current computing unit for calculating and generating the key-value cache, as well as the available transmission capacity of the data transmission path connecting multiple storage units; Based on the available computing power and the available transmission capacity, determine the mapping relationship between the loading amount of the key-value cache in each of the storage units, the computing amount of the computing unit to generate the key-value cache, and the parameters to be optimized; The parameters to be optimized characterize the recovery efficiency of the key-value cache; Based on the mapping relationship and the optimization objective of the parameters to be optimized, a target allocation strategy is determined; The target allocation strategy includes target values ​​for both the loading amount and the computation amount; Based on the target allocation strategy, a loading task is executed to load key-value cache from each of the storage units, and a calculation task is executed with the computing unit to generate key-value cache, so as to obtain the target key-value cache set.

2. The method according to claim 1, wherein determining the loading amount of the key-value cache in each storage unit, the computational amount of the computing unit calculating the key-value cache, and the mapping relationship between the parameters to be optimized based on the available computing power and the available transmission capacity, comprises: Based on the storage amount of the key-value cache in each storage unit and the corresponding available transmission capacity, a first mapping relationship between the loading amount of the key-value cache in the corresponding storage unit and the parameter to be optimized is determined. Based on the available computing power and model structure parameters, a second mapping relationship is determined between the key-value cache generated by the computing unit and the parameters to be optimized.

3. The method according to claim 2, wherein determining the target allocation strategy based on the mapping relationship and the optimization objective of the parameter to be optimized includes: Based on the total amount of data to be recovered in the key-value cache of the computing unit, the amount of data in the key-value cache of each storage unit and the corresponding available transmission capacity, the optimization constraints are determined. Based on the mapping relationships and optimization constraints, determine the total parameters to be optimized under the respective candidate values ​​of the loading amount and the computation amount; Based on the total parameters to be optimized, target values ​​for different loading amounts and computation amounts that satisfy the optimization objective are determined to form a target allocation strategy.

4. The method according to claim 3, wherein the parameter to be optimized is the recovery time; The total parameter to be optimized refers to the maximum time taken between the recovery time of loading the corresponding candidate value key-value cache from each of the storage units and the recovery time of the computing unit calculating and generating the corresponding candidate value cache key-value. The optimization objective is to minimize the maximum time consumption.

5. The method according to claim 3, wherein determining a first mapping relationship between the loading amount of the key-value cache in each of the storage units and the corresponding available transmission capacity, and determining a second mapping relationship between the key-value cache calculated by the computing unit and the parameters to be optimized, based on the available computing power and model structure parameters, comprises: Input the amount of data currently stored in the key-value cache of each storage unit, along with the corresponding available transmission capacity and access response latency, into a predefined transmission optimization model to obtain the first mapping relationship between the different loading amounts of the key-value cache in the corresponding storage unit and the parameters to be optimized. Based on the available computing power and model structure parameters, determine the computing unit time required for the computing unit to generate the key-value cache unit; The computation time of the computing unit is input into a predefined computation optimization model to obtain a second mapping relationship between the different computation amounts of the current computing unit in generating the key-value cache and the parameters to be optimized.

6. The method according to any one of claims 1-5, wherein obtaining the available computing power of the current computing unit for calculating and generating the key-value cache, and the available transmission capacity of the data transmission path connecting multiple storage units, comprises: The computing power utilization rate of the current computing unit, the available output bandwidth of each of the multiple storage units, and the available input bandwidth of the data transmission path of each of the multiple storage units connected to the computing unit are collected. The available input bandwidth of each of the data transmission paths is compared with the available output bandwidth of the corresponding storage unit to determine the available transmission capacity of the corresponding data transmission path; Based on the computing power utilization rate and the total computing power allocated to the computing unit, the available computing power of the computing unit is determined.

7. The method according to any one of claims 1-5, wherein the number of computing units is multiple, and the target key-value cache set includes a first key-value cache obtained by the first computing unit loading the key-value cache through the loading task and the key-value cache calculated and loaded by the computing task, and a second key-value cache subset obtained by directly accessing at least one of the storage units; The method further includes: Based on the target key-value cache set, the multi-head attention computation task is decomposed into different sub-tasks by the first computing unit and at least one second computing unit, respectively, to obtain the attention output of the model; Each of the subtasks corresponds to the computation of an attention head, and the subtasks are allocated based on the current attention computation resources of each computation unit; The execution process of the different sub-tasks includes: The first computing unit executes the assigned subtask based on the first key-value cache subset, and at least one second computing unit accesses the corresponding second key-value cache subset to execute the assigned subtask.

8. The method according to any one of claims 1-5, further comprising: The amount of data to be cached for preheating key values ​​is determined based on the model structure parameters and the number of lexical units in the model inference. Based on the available bandwidth of each of the multiple storage units and the amount of data in the preheating key-value cache, the preheating key-value cache is transmitted in parallel among the multiple storage units to determine the distribution relationship of the hot key-value cache data stored in each of the multiple storage units. The hotkey value cache data distribution relationship is used to determine the mapping relationship between the amount of key value cache loaded from each of the storage units and the parameters to be optimized.

9. The method according to any one of claims 1-5, wherein, The key-value cache in each of the aforementioned storage units is obtained by storing historical data generated during the model inference process, and the storage process of the key-value cache includes: Based on the model structure parameters and the number of historical lexical units obtained from model inference, the total amount of data to be stored in the key-value cache from the computing unit is determined. Obtain the available input bandwidth and available storage capacity of each of the current storage units; Based on the total amount of data to be stored in the key-value cache, the available input bandwidth, and the available storage capacity, a storage allocation strategy for the plurality of storage units is determined; the storage allocation strategy includes a target value for migrating the amount of key-value cache data to each of the plurality of storage units. Based on the storage allocation strategy, the key-value caches to be stored in the computing unit are migrated to different storage units.

10. A computer device, comprising: At least one data transmission port is configured to connect to a storage unit to enable the transmission of key-value caches between the storage unit and the storage unit; The processing device is configured as follows: Obtain the available computing power of the current computing unit for calculating and generating the key-value cache, as well as the available transmission capacity of the data transmission path connecting the multiple storage units; Based on the available computing power and the available transmission capacity, determine the mapping relationship between the loading amount of the key-value cache in each of the storage units, the computing amount of the computing unit to generate the key-value cache, and the parameters to be optimized; The parameters to be optimized characterize the recovery efficiency of the key-value cache; Based on the mapping relationship and the optimization objective of the parameters to be optimized, a target allocation strategy is determined; The target allocation strategy includes target values ​​for both the loading amount and the computation amount; Based on the target allocation strategy, a loading task is executed to load key-value cache from each of the storage units, and a calculation task is executed with the computing unit to generate key-value cache, so as to obtain the target key-value cache set.

Images