Inference acceleration method and device, equipment, storage medium and program product

Abstract

The invention relates to a reasoning acceleration method and device, equipment, a storage medium and a program product. The method comprises the following steps: receiving a user request, and obtaining a request feature of the user request; determining a target multiplexing strategy from each multiplexing strategy according to the request characteristics; the multiplexing strategy is used for indicating a multiplexing path of the key value cache; and executing the target reuse strategy according to the user request to obtain a reasoning result. According to the method and the device, when different user requests are received, processing can be performed according to a selective multiplexing strategy, and the reasoning acceleration effect is ensured to be maximized.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular to a reasoning acceleration method, apparatus, device, storage medium, and program product. Background Technology

[0002] As AI inference services continue to develop, the number of parameters is also increasing. This can lead to severe computing resource shortages on resource-constrained edge devices or in private scenarios. Currently, KVcache reuse technology is the main means of alleviating computing resource shortages, but it adds extra pressure to KVcache storage, resulting in a decrease in reuse performance. Summary of the Invention

[0003] Therefore, it is necessary to provide a reasoning acceleration method, apparatus, device, storage medium, and program product that can guarantee reusability performance to address the above-mentioned technical problems.

[0004] Firstly, this application provides a reasoning acceleration method, the method comprising:

[0005] Receive user requests and obtain the request characteristics of the user requests;

[0006] Based on the request characteristics, the target reuse strategy is determined from various reuse strategies; the reuse strategy is used to indicate the reuse path of the key-value cache;

[0007] Execute the target reuse strategy in response to the user request to obtain the inference result.

[0008] In one embodiment, the reuse path includes one of a recalculation path and a loading path; the recalculation path includes recalculating the key-value cache, forward computation, and decoding computation in sequence, and the loading path includes loading the key-value cache and decoding computation in sequence.

[0009] In one embodiment, a target reuse strategy is determined from various reuse strategies based on request characteristics, including:

[0010] Consult the lookup table and determine the reuse strategy corresponding to the request feature in the lookup table as the target reuse strategy; the lookup table includes the correspondence between each request feature and each reuse strategy.

[0011] In one embodiment, the method further includes:

[0012] Get the pre-filling phase calculation time and key-value cache transmission time corresponding to each user request;

[0013] Obtain the correspondence between request characteristics and the computation time of the pre-filling stage and the key-value cache transfer time;

[0014] Based on the correspondence, a lookup table is generated according to the calculation time of the pre-filling stage and the key-value cache transmission time.

[0015] In one embodiment, a lookup table is generated based on the pre-filling phase calculation time and the key-value cache transfer time, including:

[0016] Statistical analysis was performed on the computation time of the pre-filling stage and the key-value cache transfer time to obtain estimated results for the computation time of the pre-filling stage and the key-value cache transfer time.

[0017] Based on the estimated calculation time of the pre-filling stage and the estimated key-value cache transmission time, a reuse strategy for the corresponding request features is obtained with the goal of minimizing the target latency.

[0018] In one embodiment, the target delay includes the TTFT delay; the estimated result of the prefill stage calculation time includes the average of the calculation times of each prefill stage; and the estimated result of the key-value buffer transfer time includes the average of the transfer times of each key-value buffer.

[0019] In one embodiment, the request characteristics include at least one of request length, batch size, and context length.

[0020] Secondly, this application also provides a reasoning acceleration device, the device comprising:

[0021] The feature acquisition module is used to receive user requests and acquire the request features of the user requests;

[0022] The strategy determination module is used to determine the target reuse strategy from various reuse strategies based on request characteristics; the reuse strategy is used to indicate the reuse path of the key-value cache.

[0023] The strategy execution module is used to execute the target reuse strategy in response to user requests and obtain the inference results.

[0024] Thirdly, this application also provides an electronic device, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps in the above-described inference acceleration method.

[0025] Fourthly, this application also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps in the above-described inference acceleration method.

[0026] Fifthly, this application also provides a computer program product, including a computer program that, when executed by a processor, implements the steps in the above-described inference acceleration method.

[0027] The aforementioned inference acceleration method, apparatus, device, storage medium, and program product, upon receiving a user request, acquire the request characteristics of the user request, and then determine a target reuse strategy from various reuse strategies based on the request characteristics. The reuse strategy is used to indicate the reuse path of the key-value cache. Then, the target reuse strategy is executed for the user request to obtain the inference result. This application embodiment, upon receiving different user requests, can process them according to a selective reuse strategy, selectively reusing and loading the key-value cache (KVcache) based on the reuse path, ensuring maximum inference acceleration, replacing the original fixed KVcache loading and reuse process, and guaranteeing improved reuse performance. Attached Figure Description

[0028] To more clearly illustrate the technical solutions in the embodiments of this application or related technologies, the drawings used in the description of the embodiments of this application or related technologies will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0029] Figure 1 This is a schematic diagram of the traditional KVcache reuse method in an AI inference system.

[0030] Figure 2 This is a flowchart illustrating an inference acceleration method in one embodiment;

[0031] Figure 3 This is a flowchart illustrating the process of generating a lookup table in one embodiment;

[0032] Figure 4 This is a schematic diagram of the overall architecture for selective KVcache reuse in one embodiment.

[0033] Figure 5 This is a flowchart illustrating the inference acceleration method in another embodiment;

[0034] Figure 6 This is a schematic diagram illustrating the specific process of an inference acceleration method in one embodiment;

[0035] Figure 7 This is a structural block diagram of the inference acceleration device in one embodiment;

[0036] Figure 8 This is a diagram of the internal structure of an electronic device in one embodiment. Detailed Implementation

[0037] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0038] It should be noted that the terms "first," "second," etc., used in this application can be used to describe various elements, but these elements are not limited by these terms. These terms are only used to distinguish the first element from the second element. The terms "comprising" and "having," and any variations thereof, used in this application, are intended to cover non-exclusive inclusion. The term "multiple" used in this application refers to two or more. The term "and / or" used in this application refers to one of the solutions, or any combination of multiple solutions. In addition, the character " / " generally indicates that the preceding and following objects are in an "or" relationship.

[0039] With the continuous development of AI inference, large language models (LLMs) have ushered in a new era of generative applications. To ensure the accuracy and logical consistency of LLM responses, the number of parameters is constantly increasing, with most LLMs exceeding 1 billion parameters, and some even reaching hundreds of billions. Deploying these large-scale models on resource-constrained edge GPUs or in private scenarios can lead to severe computational resource shortages, increasing the computational pressure on GPUs. KVcache reuse technology is currently the main means of alleviating computational resource shortages. It can save intermediate results during LLM inference in the form of KVcache to GPU memory or other storage media, and reload them onto the GPU when needed, reusing existing computation results, thereby avoiding redundant calculations and alleviating the pressure on LLM forward computation.

[0040] The KVcache reuse mechanism currently used in AI inference systems does not guarantee acceleration. Because KVcache is generally large, KVcache reuse requires multi-level storage. After GPU inference computation, the KVcache results need to be unloaded to DRAM or SSD. When subsequent computations reuse the KVcache, it must first be loaded from DRAM or SSD into GPU memory before reuse, introducing additional transmission latency for LLM inference. In resource-constrained scenarios with insufficient memory bandwidth, the transmission latency of this loading process may exceed the prefill time for recalculating the KVcache. In this case, KVcache reuse does not save time but instead increases the TTFT latency of the AI ​​inference system; furthermore, it also leads to unnecessary DRAM read pressure.

[0041] While the KVcache reuse mechanism alleviates the computational burden on the GPU, it also adds extra pressure to the KVcache storage, increasing DRAM read pressure, some of which is even unnecessary (for example, some of this pressure is not even a valid workload). Figure 1 The traditional KVcache reuse method in the AI ​​inference system shown requires storing the KVcache on DRAM, a larger storage space, to ensure a higher KVcache hit rate and reduce GPU memory usage. This forces the GPU to load the KVcache from DRAM to GPU memory when using it, introducing additional transmission latency and potentially causing a decrease in TTFT performance after KVcache reuse. Therefore, the current mainstream KVcache reuse mechanism cannot guarantee the acceleration effect of AI inference, largely because the negative benefits of KVcache loading and reuse have not been considered.

[0042] Based on the aforementioned traditional technologies, this application's embodiments design a novel AI inference system with selective KVcache reuse, replacing the original KVcache usage mechanism. This allows the AI ​​inference system to maintain high inference performance regardless of the request type or hardware conditions, reducing the average TTFT latency. Specifically, this application provides an AI inference acceleration system based on selective KVcache reuse. This solution adds a selective KVcache reuse module to the original AI inference framework, ensuring the performance improvement effect of reuse. It should be noted that the selective KVcache reuse module in this application's embodiments can be a single module, responsible for loading all KVcache entries.

[0043] This application employs a basic AI inference system and a basic KVcache reuse mechanism, encompassing hardware such as GPUs, CPUs, and DRAMs, as well as mainstream AI inference frameworks. Regarding the selective KVcache reuse module, for example, this application designs three new modules for selective KVcache reuse based on mainstream AI inference frameworks such as VLLM. These modules may include the following three sub-modules: a KVcache processing information collection module (also called a KVcache processing information collector, or simply a collector), a selective KVcache reuse strategy module, and a KVcache reuse implementation module. The KVcache processing information collection module can detect the transmission latency between the GPU and DRAM for KVcache generated by the GPU's computation of different user requests, while also recording the time spent computationating the KVcache. The selective KVcache reuse strategy module can calculate the sweet spot for accelerating KVcache reuse based on the collector's information, generating the optimal reuse strategy for different request characteristics. The KVcache multiplexing implementation module can be used to enable AI inference systems to process according to a selective KVcache multiplexing strategy, maximizing the acceleration effect of AI inference systems, and also reducing the pressure of invalid DRAM reads.

[0044] It should be noted that the selective KVcache multiplexing module, KVcache processing information collection module, selective KVcache multiplexing strategy module, and KVcache multiplexing implementation module in the embodiments of this application can refer to software modules / hardware modules; optionally, software modules and hardware modules can refer to the various functional modules. Furthermore, the above-mentioned software modules and hardware modules can be implemented entirely or partially through software, hardware, or a combination thereof. The above modules can be embedded in the processor of the electronic device in hardware form or independently of it, or they can be stored in the memory of the electronic device in software form, so that the processor can call and execute the operations corresponding to the above modules.

[0045] This application proposes a selective KVcache reuse algorithm. Based on the loading and calculation information of previously requested KVcache entries, it outputs a "reuse strategy" and automatically determines the optimal reuse strategy for a new request. This application employs a selective KVcache reuse mechanism to ensure that the used KVcache reuse strategy improves the TTFT performance of AI inference without causing performance degradation. This application uses a dynamic KVcache loading mechanism, abandoning the traditional strategy of loading all cache entries upon a cache hit, and only loading KVcache entries that offer time savings. By replacing the original fixed KVcache loading and reuse process with a selective KVcache reuse module and mechanism, the average TTFT of the AI ​​inference system is significantly shortened, improving the acceleration effect of large AI models using KVcache.

[0046] Exemplary examples of this application involve cloud computing, AI inference systems, intelligent computing clusters, and GPUs, such as AI inference and high-performance computing. Optionally, this application can be used in scenarios such as large-scale data center networks and large-model AI inference. Further, the application scenarios of this application include, but are not limited to, AI inference services in intelligent computing centers and edge-side inference services with lower GPU computing power.

[0047] It should be noted that the beneficial effects or technical problems solved by the embodiments of this application are not limited to this one, but may also be other implicit or related problems. For details, please refer to the description of the embodiments below.

[0048] Before introducing the specific embodiments of this application, the technical terms involved in this application will be explained:

[0049] KVcache (Key-Value Cache) is a mechanism that caches intermediate computation results (Key and Value) during the AI ​​inference computation process. It can be reused to avoid repeated computation for the same user request, thereby significantly improving inference efficiency.

[0050] AI (Artificial Intelligence) inference systems are hardware and software systems used to deploy and implement inference for large language models. They primarily include GPUs (Graphics Processing Units), CPUs (Central Processing Units), system DRAM (Dynamic Random Access Memory), SSDs (Solid State Disk or Solid State Drive) storage, and AI inference frameworks that enable large-scale language inference capabilities. Generally, they support mainstream KVcache reuse mechanisms.

[0051] Prefill computation: This is one of the computational stages in the AI ​​inference process, also known as the forward computation process. This stage involves a large number of matrix additions and multiplications, placing extremely high demands on the GPU's computing power. Therefore, Kvcache reuse mainly occurs in this stage to alleviate the computational burden on the GPU.

[0052] Decoding computation: This is the second computational stage in the AI ​​inference process, and also the stage for generating the token needed by the user. This computational stage has relatively low requirements for GPU computing power.

[0053] TTFT (Time to first token): An important metric for evaluating the performance of AI inference systems, measuring the latency between when a user initiates a request and when the first token result is received. One of the evaluation criteria for AI inference acceleration is the reduction of TTFT.

[0054] The technical solution of this application and how the technical solution of this application solves the above-mentioned technical problems are described in detail below with specific embodiments. These specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments. The embodiments of this application will now be described with reference to the accompanying drawings.

[0055] In one exemplary embodiment, such as Figure 2 As shown, a method for accelerating inference is provided. Taking the application of this method to an AI inference system as an example, the method includes the following steps 202 to 206. Wherein:

[0056] Step 202: Receive user request and obtain the request characteristics of user request.

[0057] In this context, "user request" can refer to user requests received by the AI ​​inference system. For example, the AI ​​inference system can receive user requests over a period of time and perform large-scale model inference calculations.

[0058] Specifically, when the AI ​​inference system receives a user request, it can obtain the request characteristics of the user request. In this embodiment, the request characteristics can be used to characterize the request type of the user request. For example, the request characteristics can be used to classify user requests, and the same reuse strategy can be adopted for user requests of the same type.

[0059] Optionally, the KVcache processing information collection module in the AI ​​inference system collects the request features corresponding to user requests and adds the request features to the request information database. In some embodiments, the request features may include at least one of request length, batch size, and context length, which is not limited in this application.

[0060] Step 204: Based on the request characteristics, determine the target reuse strategy from the various reuse strategies; the reuse strategy is used to indicate the reuse path of the key-value cache.

[0061] Specifically, the AI ​​inference system can determine a target reuse strategy from various reuse strategies based on request characteristics. For example, the reuse strategy corresponding to the request characteristics among multiple reuse strategies can be used as the target reuse strategy; that is, the target reuse strategy refers to the currently selected reuse strategy. For instance, the reuse strategy can refer to the optimal reuse strategy corresponding to the request characteristics. This application, by using reuse strategies corresponding to different user requests, enables the AI ​​inference system to process different user requests according to a selective KVcache reuse strategy, ensuring that the acceleration effect of the AI ​​inference system for all requests is maximized.

[0062] For example, the selective KVcache reuse strategy module in the AI ​​inference system can generate various reuse strategies. For instance, the selective KVcache reuse strategy module calculates the sweet spot for accelerating KVcache reuse for user requests based on information collected by the KVcache processing information collection module, and generates reuse strategies corresponding to different request characteristics.

[0063] Furthermore, the reuse strategy in this embodiment is used to indicate the reuse path of the key-value cache (KVcache). This reuse path can refer to a selective process of loading and reusing KVcache, i.e., selective reuse and loading of KVcache. Based on the reuse strategy indicating the reuse path, this application replaces the original fixed KVcache loading and reuse process with a selective KVcache reuse mechanism, significantly reducing the average latency of the AI ​​inference system and improving the acceleration effect of large AI models when using KVcache.

[0064] Step 206: Execute the target reuse strategy for the user request to obtain the inference result.

[0065] Specifically, once a target reuse strategy is determined, it can be executed in response to user requests to obtain inference results. For example, the inference results may include a token. Optionally, once a target reuse strategy is determined, the KVcache reuse implementation module in the AI ​​inference system can process the data according to the target reuse strategy and output the inference results.

[0066] In the aforementioned inference acceleration method, the selective KVcache reuse mechanism can process different user requests according to a selective reuse strategy. Based on the reuse path, the key-value cache KVcache is selectively reused and loaded, maximizing the inference acceleration effect. This replaces the original fixed Kvcache loading and reuse process, ensuring improved reuse performance.

[0067] Regarding the reuse path in the embodiments of this application, in one embodiment, the reuse path may include one of a recalculation path and a loading path; the recalculation path includes recalculating the key-value cache, forward calculation, and decoding calculation in sequence, and the loading path includes loading the key-value cache and decoding calculation in sequence.

[0068] Specifically, the reuse path in this application can be a recalculation path or a loading path. Once the target reuse strategy is determined, either the recalculation path or the loading path can be selected. For example, if the target reuse strategy is a KVcache recalculation strategy (referred to as the recalculation strategy), then the recalculation path is selected. Similarly, if the target reuse strategy is a KVcache loading strategy (referred to as the loading strategy), then the loading path is selected.

[0069] The recalculation path can include sequentially recalculating the key-value cache, forward computation, and decoding computation. When using the recalculation path, the GPU no longer loads the KVcache but recalculates it, then performs forward computation and decoding computation sequentially, outputs the token, and the user receives the token, reducing unnecessary DRAM read pressure. The loading path can include sequentially loading the key-value cache and decoding computation. When using the loading path, the GPU reads and loads the corresponding KVcache from DRAM, skips forward computation, directly performs decoding computation, and outputs the token, which the user receives.

[0070] It's understandable that when the chosen target reuse strategy is the KVcache loading strategy, the KVcache loading time is less than the KVcache recalculation time, thus reusing KVcache can achieve AI inference acceleration. Furthermore, regarding the stage under this strategy where forward computation is skipped and decoding computation is performed directly to output the token, since the decoding computation requires very little computing power, this stage does not need to use KVcache.

[0071] In some embodiments, determining a target reuse strategy from various reuse strategies based on request characteristics includes: consulting a lookup table and determining the reuse strategy corresponding to the request characteristics in the lookup table as the target reuse strategy; the lookup table includes the correspondence between each request characteristic and each reuse strategy.

[0072] Specifically, the AI ​​inference system can determine the target reuse strategy by consulting a lookup table to identify the reuse strategy corresponding to the request features in the table. The lookup table includes the correspondence between each request feature and each reuse strategy. Optionally, the correspondence between each request feature and each reuse strategy can refer to a one-to-one correspondence between multiple request features and multiple reuse strategies.

[0073] For example, the lookup table in this application may refer to the lookup table of "request characteristics - best reuse strategy". When the request characteristics of the user request are obtained, the lookup table of "request characteristics - best reuse strategy" can be consulted to determine the target reuse strategy, so that the target reuse strategy can be implemented.

[0074] In one embodiment, such as Figure 3 As shown, the method may also include:

[0075] Step 302: Obtain the pre-filling stage calculation time and key-value cache transmission time corresponding to each user request.

[0076] Specifically, the KVcache information collection module can collect the prefill stage computation time and key-value cache transfer time required for user requests. The prefill stage computation time can refer to the Prefill computation time (also known as KVcache computation time, or recalculation time). The key-value cache transfer time can refer to the KVcache transfer time; optionally, the KVcache transfer time is the DRAM→GPU transfer time. For example, the KVcache transfer time can be obtained by directly measuring the DRAM→GPU transfer time, or by approximately estimating it by measuring the GPU→DRAM transfer time in reverse. It should be noted that, generally, the GPU→DRAM transfer time is the same as the DRAM→GPU transfer time.

[0077] Once the pre-filling stage computation time and key-value cache transmission time are obtained, a request information database can be formed; at the same time, request characteristics such as request length can also be added to this request information database.

[0078] Understandably, initially, the KVcache database in DRAM is empty, requiring a recalculation strategy for all new user requests. The KVcache data obtained during the pre-filling calculation process for new user requests is stored in the DRAM database, allowing for the collection of KVcache data over a period of time. During this process, the request characteristics of each user request, along with the corresponding recalculation time (i.e., the pre-filling phase calculation time), can be recorded. As time progresses, new user requests can be matched with the KVcache database, indicating that existing KVcache data can be reused. In this case, a loading strategy can be employed to load the KVcache data from the DRAM database onto the GPU, recording the request characteristics of each user request and the corresponding KVcache transfer time.

[0079] Step 304: Obtain the correspondence between request characteristics and the calculation time of the pre-filling stage and the key-value cache transmission time.

[0080] Specifically, given the pre-filling stage calculation time and key-value cache transfer time, the correspondence between request characteristics, recalculation time (for multiple user requests), and KVcache transfer time (for multiple user requests) can be obtained.

[0081] Step 306: Based on the correspondence, generate a lookup table according to the calculation time of the pre-filling stage and the key-value cache transmission time.

[0082] Specifically, based on the above correspondence, a lookup table can be generated according to the calculation time of the pre-filling stage and the key-value cache transmission time; for example, various reuse strategies can be generated to form a lookup table, which can include the correspondence between various request features and various reuse strategies.

[0083] The aforementioned inference acceleration method can generate optimal reuse strategies for different request characteristics, enabling the AI ​​inference system to process requests according to a selective KVcache reuse strategy, thereby maximizing the acceleration effect of the AI ​​inference system.

[0084] In one embodiment, a lookup table is generated based on the pre-filling stage computation time and the key-value cache transmission time, including: performing statistical processing on the pre-filling stage computation time and the key-value cache transmission time respectively to obtain the estimated results of the pre-filling stage computation time and the key-value cache transmission time; and obtaining a reuse strategy for the corresponding request feature based on the estimated results of the pre-filling stage computation time and the key-value cache transmission time, with the goal of minimizing the target latency.

[0085] Specifically, based on the collected information, embodiments of this application can generate selective KVcache reuse strategies for different request characteristics. Specifically, statistical processing can be performed on the pre-filling stage computation time and the key-value cache transfer time to obtain estimated results for the pre-filling stage computation time and the key-value cache transfer time, thereby obtaining a lookup table. For example, the estimated result for the pre-filling stage computation time can refer to a statistical estimate of the KVcache computation time, and the estimated result for the key-value cache transfer time can refer to a statistical estimate of the KVcache transfer time.

[0086] For example, the selective KVcache reuse strategy module extracts statistical estimates of KVcache transmission time and KVcache computation time under different request characteristics from the request information database of the KVcache processing information collection module, and combines these latency information to deduce a reference table of "request characteristics - best reuse strategy".

[0087] In some embodiments, the estimated computation time for the pre-filling stage may include the average computation time of each pre-filling stage; the estimated key-value cache transfer time may include the average transfer time of each key-value cache. Specifically, for each request feature, the corresponding recomputation time and KVcache transfer time can be estimated, and then the average of the times corresponding to all request features can be used as the time estimate for that request feature.

[0088] Furthermore, based on the estimated computation time of the pre-filling stage and the estimated key-value cache transfer time, a reuse strategy for the corresponding request feature can be obtained with the shortest possible target latency as the objective. The target latency can be estimated as an evaluation metric to generate selective KVcache reuse strategies for different request features. The reuse strategy implemented in this application can reduce the latency of the AI ​​processing system. Optionally, the target latency may include the TTFT latency. It is understood that the TTFT performance will differ under different strategies. Under the recalculation strategy, the TTFT value is the recalculation time; under the loading strategy, the TTFT value is the KVcache transfer time. Therefore, this embodiment selects the strategy with the shortest time as the optimal reuse strategy for the request feature.

[0089] Based on the aforementioned inference acceleration method, after obtaining the "request characteristics - optimal reuse strategy" lookup table, when a new user request is received, the KVcache reuse implementation module can apply the optimal KVcache reuse mechanism (KVcache loading or KVcache recalculation) to that user request according to the lookup table, ensuring the high performance of the AI ​​inference system. As the request volume continues to increase, the database collected by the KVcache processing information collection module will become richer, and the selective KVcache reuse strategy module will also be updated and iterated, achieving automatic acceleration strategy optimization.

[0090] For example Figure 4 Taking the overall architecture of selective KVcache reuse shown as an example, this embodiment of the application, based on mainstream AI inference frameworks such as VLLM, designs three new modules for selective KVcache reuse: a KVcache processing information collection module, a selective KVcache reuse strategy module, and a KVcache reuse implementation module. Further, as... Figure 5 As shown, taking QP (Queue Pair) task management as an example, the inference acceleration method can include the following process: The AI ​​inference system receives user requests over a period of time and performs large model inference calculations. The KVcache information collection module collects information related to KVcache calculation and transmission from the initially received user requests, such as the prefill calculation time (i.e., KVcache calculation time) and KVcache transmission time required by the KVcache corresponding to the user request, forming a request information database. At the same time, request characteristics such as request length can also be added to this request information database.

[0091] The selective KVcache reuse strategy module can extract statistical estimates of KVcache transmission time (i.e., KVcache transmission time estimates) and KVcache computation time (i.e., KVcache computation time estimates) under different request characteristics from the request information database of the KVcache processing information collection module. Combining this latency information, it infers a lookup table of "request characteristics - optimal reuse strategy." For example, based on the collected information, it estimates using TTFT as the evaluation metric and generates selective KVcache reuse strategies for different request characteristics. The lookup table can be updated in real time when KVcache loading occurs.

[0092] Upon receiving a new user request, the KVcache reuse implementation module can extract the request characteristics and then implement a reuse strategy (selecting a loading path or a recalculation path) for the request based on the "request characteristics - best reuse strategy" lookup table. For example, it can implement the best KVcache reuse mechanism (KVcache loading or KVcache recalculation) for the request to ensure the high performance of the AI ​​inference system.

[0093] To further illustrate the scheme of this application, the following will be combined with... Figure 6 Please provide an explanation, such as Figure 6 As shown, each KVcache has a corresponding Token. Retrieving user requests and searching the KVcache database for a matching KVcache means matching the user request's Token with the Token in the KVcache database. When the new user request's Token is the same as the Token corresponding to a certain KVcache in the KVcache database, it means that the KVcache has been hit. Figure 6 The metadata in the file refers to the request characteristics of each user request and the corresponding KVcache transfer time.

[0094] Among them, regarding Figure 6 The KVcache information collection module then records metadata for use by the selective KVcache reuse strategy module. It's worth noting that during the initialization phase, information such as KVcache computation time and KVcache transfer time needs to be accumulated to provide samples for strategy generation. Regarding... Figure 6 The selective KVcache reuse strategy module selects the best reuse strategy. This can be understood as the module recalculating the KVcache if there is no KVcache hit.

[0095] This application employs a selective KVcache reuse mechanism to ensure that the KVcache reuse strategy improves the TTFT performance of AI inference without causing performance degradation. It adopts a dynamic KVcache loading mechanism, abandoning the traditional strategy of loading all caches upon a cache hit, and only loading KVcache entries that offer time savings. By replacing the original fixed KVcache loading and reuse process with a selective KVcache reuse mechanism, the average TTFT of the AI ​​inference system is significantly shortened, improving the acceleration effect of large AI models when using KVcache.

[0096] It should be understood that although the steps in the flowcharts of the embodiments described above are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the embodiments described above may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages in other steps. It is understood that the steps in different embodiments can be freely combined as needed, and all non-contradictory solutions formed by such combinations are within the scope of protection of this application.

[0097] Based on the same inventive concept, this application also provides a reasoning acceleration device for implementing the reasoning acceleration method described above. The solution provided by this device is similar to the solution described in the above method; therefore, the specific limitations in one or more of the reasoning acceleration device embodiments provided below can be found in the limitations of the reasoning acceleration method described above, and will not be repeated here.

[0098] In one exemplary embodiment, such as Figure 7 As shown, a reasoning acceleration device is provided, comprising:

[0099] The feature acquisition module 901 is used to receive user requests and acquire the request features of the user requests;

[0100] The strategy determination module 902 is used to determine the target reuse strategy from various reuse strategies based on the request characteristics; the reuse strategy is used to indicate the reuse path of the key-value cache.

[0101] The strategy execution module 903 is used to execute the target reuse strategy in response to user requests and obtain inference results.

[0102] In one embodiment, the reuse path includes one of a recalculation path and a loading path; the recalculation path includes recalculating the key-value cache, forward computation, and decoding computation in sequence, and the loading path includes loading the key-value cache and decoding computation in sequence.

[0103] In one embodiment, the strategy determination module 902 is used to consult a lookup table and determine the reuse strategy corresponding to the request feature in the lookup table as the target reuse strategy; the lookup table includes the comparison relationship between each request feature and each reuse strategy.

[0104] In one embodiment, the device further includes:

[0105] The time information acquisition module is used to acquire the pre-filling stage calculation time and key-value cache transmission time corresponding to each user request;

[0106] The correspondence determination module is used to obtain the correspondence between request features and the calculation time of the pre-filling stage and the key-value cache transmission time;

[0107] The lookup table generation module is used to generate a lookup table based on the correspondence, the calculation time of the pre-filling stage, and the key-value cache transmission time.

[0108] In one embodiment, the lookup table generation module includes:

[0109] The time estimation module is used to perform statistical processing on the pre-filling stage calculation time and the key-value cache transfer time respectively, and obtain the estimated results of the pre-filling stage calculation time and the key-value cache transfer time.

[0110] The strategy generation module is used to obtain a reuse strategy for the corresponding request features based on the estimated calculation time of the pre-filling stage and the estimated key-value cache transmission time, with the goal of minimizing the target latency.

[0111] In one embodiment, the target delay includes the TTFT delay; the estimated result of the prefill stage calculation time includes the average of the calculation times of each prefill stage; and the estimated result of the key-value buffer transfer time includes the average of the transfer times of each key-value buffer.

[0112] In one embodiment, the request characteristics include at least one of request length, batch size, and context length.

[0113] Each module in the aforementioned inference acceleration device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in the electronic device, or stored in the memory of the electronic device as software, so that the processor can call and execute the operations corresponding to each module.

[0114] In one exemplary embodiment, an electronic device is provided, which may be a server, and its internal structure diagram may be as follows: Figure 8As shown, this electronic device includes a processor, memory, input / output (I / O) interfaces, and a communication interface. The processor, memory, and I / O interfaces are connected via a system bus, and the communication interface is also connected to the system bus via the I / O interfaces. The processor provides computational and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and a database. The internal memory provides the environment for the operating system and computer programs stored in the non-volatile storage media. The database stores KVcache and other related data. The I / O interfaces are used for exchanging information between the processor and external devices. The communication interface is used for communicating with external terminals via a network connection. When the computer program is executed by the processor, it implements an inference acceleration method.

[0115] Those skilled in the art will understand that Figure 8 The structure shown is only a block diagram of a part of the structure related to the present application and does not constitute a limitation on the electronic device to which the present application is applied. The specific electronic device may include more or fewer components than shown in the figure, or combine certain components, or have different component arrangements.

[0116] In one embodiment, an electronic device is also provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps in the above-described method embodiments.

[0117] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon that, when executed by a processor, implements the steps in the above method embodiments.

[0118] In one embodiment, a computer program product is provided, including a computer program that, when executed by a processor, implements the steps in the above method embodiments.

[0119] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, artificial intelligence (AI) processors, etc., and are not limited to these.

[0120] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.

[0121] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. A method for accelerating inference, characterized in that, The method includes: Receive a user request and obtain the request characteristics of the user request; Based on the request characteristics, a target reuse strategy is determined from various reuse strategies; the reuse strategy is used to indicate the reuse path of the key-value cache. The target reuse strategy is executed in response to the user request to obtain the inference result.

2. The method according to claim 1, characterized in that, The reuse path includes one of a recalculation path and a loading path; the recalculation path includes recalculating the key-value cache, forward calculation, and decoding calculation in sequence, and the loading path includes loading the key-value cache and decoding calculation in sequence.

3. The method according to claim 1, characterized in that, Based on the request characteristics, a target reuse strategy is determined from various reuse strategies, including: Consult a lookup table and determine the reuse strategy corresponding to the request feature in the lookup table as the target reuse strategy; the lookup table includes the correspondence between each request feature and each reuse strategy.

4. The method according to claim 3, characterized in that, The method further includes: Obtain the pre-filling stage calculation time and key-value cache transmission time corresponding to each user request; Obtain the correspondence between the request characteristics and the calculation time of the pre-filling stage, and the key-value cache transmission time; Based on the correspondence, the lookup table is generated according to the calculation time of the pre-filling stage and the key-value cache transmission time.

5. The method according to claim 4, characterized in that, Based on the aforementioned correspondence, the lookup table is generated according to the calculation time of the pre-filling stage and the key-value cache transmission time, including: Statistical processing was performed on the calculation time of the pre-filling stage and the transmission time of the key-value cache, respectively, to obtain the estimated results of the calculation time of the pre-filling stage and the estimated results of the transmission time of the key-value cache. Based on the estimated calculation time of the pre-filling stage and the estimated transmission time of the key-value cache, the reuse strategy corresponding to the request characteristics is obtained with the goal of minimizing the target latency.

6. The method according to claim 5, characterized in that, The target delay includes the TTFT delay; the estimated result of the pre-fill stage calculation time includes the average of the calculation times of each pre-fill stage; the estimated result of the key-value cache transmission time includes the average of the transmission times of each key-value cache.

7. The method according to any one of claims 1 to 6, characterized in that, The request characteristics include at least one of request length, batch size, and context length.

8. A reasoning acceleration device, characterized in that, The device includes: The feature acquisition module is used to receive user requests and acquire the request features of the user requests; The strategy determination module is used to determine a target reuse strategy from various reuse strategies based on the request characteristics; the reuse strategy is used to indicate the reuse path of the key-value cache. The strategy execution module is used to execute the target reuse strategy in response to the user request and obtain the inference result.

9. An electronic device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 7.

11. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 7.

Images