Cache preloading method and device based on KV Cache, equipment and medium
By monitoring the GPU's idle state and preloading the KV cache into the GPU memory, the service response latency problem of large language models was solved, achieving efficient use of the cache and improving inference efficiency and response speed.
Patent Information
- Application Number
- CN202511397564.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-09-28
- Publication Date
- 2025-12-12
AI Technical Summary
The service response latency problem of large language models in existing technologies is mainly caused by loading the necessary cache only after the request arrives.
By monitoring the GPU's idle state and preloading the KV cache into the video memory, data is prepared in advance using idle resources, thus achieving cache preloading.
It significantly improves the efficiency and response speed of large model inference and solves the service response latency problem.
Smart Images
Figure CN121116865A_ABST
Abstract
Description
TECHNICAL FIELD
[0001] The present application relates to the technical field of artificial intelligence, and in particular to a cache preloading method and device based on KV Cache, equipment and a medium. BACKGROUND
[0002] With the wide application of large language models (LLM), inference efficiency has become a key factor affecting user experience and deployment cost. KV Cache technology caches the attention vectors of historical tokens to avoid repeated calculations, significantly improving the efficiency of autoregressive generation.
[0003] The prior art only loads the necessary cache after the request arrives, which increases the service response delay. Therefore, how to reduce the service response delay of large language models has become a technical problem to be solved. SUMMARY
[0004] Therefore, the purpose of the present application is to provide a cache preloading method and device based on KV Cache, which can monitor the idle state of the GPU and preload the KV Cache to the video memory, thereby preparing data in advance using idle resources and solving the problem of large model inference response delay. The specific scheme is as follows:
[0005] In a first aspect, the present application provides a cache preloading method based on KV Cache, applied to a target cache management system, comprising:
[0006] receiving a target prompt prefix specified by a target user through a preset application programming interface and a business priority of a large language model prompt corresponding to the target prompt prefix;
[0007] real-time statistics of target feature information corresponding to each target prompt prefix, and generating a statistical priority score corresponding to each target prompt prefix according to the target feature information; wherein the target feature information includes access frequency, cache hit rate and business value coefficient corresponding to each target prompt prefix;
[0008] weighting and fusing the business priority corresponding to each target prompt prefix and the corresponding statistical priority score to obtain a target priority score corresponding to each target prompt prefix;
[0009] determining whether the local GPU is in a preset idle state, and if the local GPU is in the preset idle state, determining a to-be-loaded KV Cache from a target KV Cache corresponding to each target prompt prefix based on the target priority score, and preloading the to-be-loaded KV Cache to a target video memory in a sequence from high to low of the target priority score.
[0010] Optionally, before the real-time statistics of the target feature information corresponding to each target prompt prefix and the generation of the statistical priority score corresponding to each target prompt prefix based on the target feature information, the method further comprises:
[0011] calculating the target KV Cache corresponding to each target prompt prefix by using a target calculation method, storing the target KV Cache corresponding to each target prompt prefix to a preset storage space of the local, and recording a business identifier, a creation time and a predicted use probability corresponding to each target KV Cache.
[0012] Optionally, the generation of the statistical priority score corresponding to each target prompt prefix based on the target feature information comprises:
[0013] determining a first weight coefficient corresponding to the access frequency, a second weight coefficient corresponding to the cache hit rate and a third weight coefficient corresponding to the business value coefficient;
[0014] weighting and fusing the access frequency, the cache hit rate and the business value coefficient corresponding to each target prompt prefix based on the first weight coefficient, the second weight coefficient and the third weight coefficient, to generate the statistical priority score corresponding to each target prompt prefix.
[0015] Optionally, the determination of whether the local GPU is in the preset idle state comprises:
[0016] real-time monitoring of a running state of the GPU to obtain a GPU utilization rate of the GPU in a target number of continuous time windows, and if the GPU utilization rate of the GPU in the target number of continuous time windows is not greater than a preset GPU utilization rate threshold, determining that the target GPU is in the preset idle state.
[0017] Optionally, the preloading of the to-be-loaded KV Cache to the target video memory comprises:
[0018] Determine the target loading speed corresponding to the to-be-loaded KV Cache according to the idle duration of the target GPU, and pre-load each to-be-loaded KV Cache to the target GPU in turn according to the target loading speed and a progressive loading strategy.
[0019] Optionally, after the to-be-loaded KV Cache is pre-loaded to the target GPU in turn according to the order of the target priority score from high to low, the method further comprises:
[0020] Obtain a target business processing request sent by the target user, and determine whether the target GPU contains a to-be-used KV Cache corresponding to the target business processing request.
[0021] Optionally, after the to-be-loaded KV Cache is pre-loaded to the target GPU in turn according to the order of the target priority score from high to low, the method further comprises:
[0022] If the target GPU does not contain the to-be-used KV Cache corresponding to the target business processing request, the to-be-used KV Cache corresponding to the target business processing request is calculated in real time, and the target business processing request is processed by using the to-be-used KV Cache and the target large model.
[0023] In a second aspect, the present application provides a cache pre-loading device based on KV Cache, applied to a target cache management system, comprising:
[0024] A data receiving module is configured to receive a target prompt prefix specified by a target user through a preset application programming interface and a business priority of a large language model prompt word corresponding to the target prompt prefix.
[0025] A feature information statistical module is configured to statistically determine target feature information corresponding to each target prompt prefix in real time, and generate a statistical priority score corresponding to each target prompt prefix according to the target feature information.
[0026] A score obtaining module is configured to weight and fuse the business priority and the corresponding statistical priority score corresponding to each target prompt prefix to obtain a target priority score corresponding to each target prompt prefix.
[0027] The cache preloading module is configured to determine whether a local GPU is in a preset idle state. If the local GPU is in the preset idle state, the cache preloading module is configured to determine, based on the target priority score, a to-be-loaded KV Cache from a target KV Cache corresponding to each target prompt prefix, and preloads each to-be-loaded KV Cache to a target video memory in a descending order of the target priority score.
[0028] In a third aspect, the present application provides an electronic device, comprising:
[0029] A memory configured to save a computer program;
[0030] A processor configured to execute the computer program to implement the foregoing KV Cache-based cache preloading method.
[0031] In a fourth aspect, the present application provides a computer-readable storage medium configured to save a computer program, which, when executed by a processor, implements the foregoing KV Cache-based cache preloading method.
[0032] The present application first receives a target prompt prefix specified by a target user through a preset application programming interface and a business priority of a large language model prompt corresponding to the target prompt prefix, then statistically determines target feature information corresponding to each target prompt prefix in real time, and generates a statistical priority score corresponding to each target prompt prefix according to the target feature information; wherein the target feature information includes an access frequency, a cache hit rate and a business value coefficient corresponding to each target prompt prefix, then the business priority and the corresponding statistical priority score corresponding to each target prompt prefix are weighted and fused to obtain a target priority score corresponding to each target prompt prefix, finally it is determined whether a local GPU is in a preset idle state. If the local GPU is in the preset idle state, a to-be-loaded KV Cache is determined based on the target priority score from a target KV Cache corresponding to each target prompt prefix, and each to-be-loaded KV Cache is preloaded to a target video memory in a descending order of the target priority score. As can be seen, the present application realizes dynamic adaptive optimization of the cache strategy by statistically determining the access frequency, the cache hit rate and the business value coefficient and generating the statistical priority score; a hybrid decision model with business intention and real-time efficiency is formed by weighted fusion of the business weight and the statistical score, and the cache priority determination is refined; by monitoring the idle state of the GPU and preloading the KV Cache to the video memory according to the priority, the idle resources are used to prepare data in advance, the response delay problem caused by the lack of preloading mechanism is solved, and finally the cache is directly called during request processing, which significantly improves the inference efficiency and response speed. Attached Figure Description
[0033] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.
[0034] Figure 1 This application discloses a flowchart of a cache preloading method based on KV Cache.
[0035] Figure 2 This is a schematic diagram of a specific cache preloading method based on KV Cache disclosed in this application;
[0036] Figure 3 This is a schematic diagram of a cache preloading device based on KV Cache disclosed in this application;
[0037] Figure 4 This is a structural diagram of an electronic device disclosed in this application. Detailed Implementation
[0038] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.
[0039] Current cache loading methods can increase service response latency. To address this, this application provides a cache preloading method based on KV Cache. By monitoring the GPU's idle state and preloading the KV Cache into GPU memory, it utilizes idle resources to prepare data in advance, thus solving the response latency problem for large model inference.
[0040] See Figure 1 As shown, this embodiment of the invention discloses a cache preloading method based on KV Cache, applied to a target cache management system, including:
[0041] Step S11: Receive the target prompt word prefix specified by the target user through a preset application programming interface and the service priority of the large language model prompt word corresponding to the target prompt word prefix.
[0042] The system includes a business awareness module, a cache management module, a resource monitoring module, and a model service. The cooperation process of each part is shown in Figure 2 The business awareness module analyzes high-frequency business scenarios, provides the ability to actively configure high-priority businesses, and provides cache data. The cache management module then actively caches the data and preloads the KVCache before the user inputs the request.
[0043] The functions of each module are as follows:
[0044] Business awareness module: Collect user income, analyze high-frequency business scenarios, and provide the ability to actively configure high-priority businesses.
[0045] Cache management module: Responsible for storing, querying, and preloading cache data.
[0046] Resource monitoring module: Real-time collection of GPU (Graphics Processing Unit) utilization, video memory usage, and other indicators.
[0047] Model service: Basic service for large models, supports querying KV Cache, and responds to customer needs.
[0048] It can be understood that the process of receiving the target prompt prefix is responsible for the business awareness module, and the business awareness module specifically includes:
[0049] Business cache configuration module: Receives the cache prompt prefix specified by the business and its priority weight, and forms a business cache whitelist.
[0050] Statistical cache decision module: Real-time analysis of request flow, dynamic determination of high-value cache content based on access frequency, business value, cache hit rate, and other indicators.
[0051] Hybrid cache strategy engine: Weighted fusion of business specified weights and statistical indicators to generate the final cache priority score.
[0052] It should be noted that this embodiment supports active caching of commonly used prompt prefixes, supports cross-session persistent storage, and improves cache reuse rate. Through a hybrid strategy of business specification and statistical driving, the cache hit rate is improved, especially in scenarios where multiple users share the same prompt prefix.
[0053] Step S12, real-time statistics of each target prompt prefix corresponding to the target feature information, and generating a statistical priority score corresponding to each target prompt prefix according to the target feature information; wherein the target feature information includes the access frequency, cache hit rate, and business value coefficient corresponding to each target prompt prefix.
[0054] In the embodiment, before the real-time statistics of the target feature information corresponding to each target prompt prefix and the generation of the statistical priority score corresponding to each target prompt prefix according to the target feature information, the method further comprises: calculating the target KV Cache corresponding to each target prompt prefix by using a target calculation method, storing the target KV Cache corresponding to each target prompt prefix to a preset storage space of a local device, and recording the business identifier, the creation time, and the predicted use probability corresponding to each target KV Cache.
[0055] The above process is responsible for the cache management module, specifically, the cache management module has the following mechanisms:
[0056] Active cache mechanism:
[0057] The commonly used prompt prefix (i.e., target prompt prefix) specified for the business is pre-calculated and stored persistently;
[0058] The business identifier, the creation time, the last access time, and the predicted use probability are recorded for each cache item.
[0059] Idle preloading mechanism:
[0060] The GPU utilization is monitored, and when the utilization is lower than a threshold T2 in N consecutive time windows, it is determined that it is an idle period; in the embodiment, the data of the time window can be set according to the actual application requirement, which is not specifically limited here.
[0061] According to the cache priority score, the high-value cache item is selected and preloaded to the video memory;
[0062] The progressive preloading strategy is adopted to avoid resource contention at the end of the idle period.
[0063] In addition, in the embodiment, the process of generating the statistical priority score corresponding to each target prompt prefix according to the target feature information can specifically include: determining a first weight coefficient corresponding to the access frequency, a second weight coefficient corresponding to the cache hit rate, and a third weight coefficient corresponding to the business value coefficient; and performing weighted fusion on the access frequency, the cache hit rate, and the business value coefficient corresponding to each target prompt prefix based on the first weight coefficient, the second weight coefficient, and the third weight coefficient to generate the statistical priority score corresponding to each target prompt prefix.
[0064] Specifically, the statistical indicators of each prompt prefix are calculated in real time:
[0065] Access frequency: the number of requests per unit time;
[0066] Cache hit rate: the reuse ratio of the prefix cache;
[0067] Business value coefficient: weighted value related to business income / importance;
[0068] Statistical priority score: W_stat = • Access frequency • Cache hit rate • Business value coefficient;
[0069] wherein the above i.e. the first weight coefficient, i.e. the second weight coefficient, i.e. the third weight coefficient.
[0070] In addition, the business specified cache content in the embodiment is as follows:
[0071] The business party specifies the prompt prefix and business priority (e.g. 1-10 levels) to be cached through an API, i.e. a preset application programming interface;
[0072] The system allocates a basic weight W_base (i.e. business priority) for each business cache item;
[0073] Example: the customer service robot system sets "Hello, I am AI assistant" as a cache item with priority 8.
[0074] Through the business priority mechanism, the cache of key businesses is given priority protection, meeting the service quality requirements of enterprise-level applications; and the cache decision of the embodiment is not only based on historical statistics, but also combines business value and prediction models to achieve more accurate resource allocation; in addition, through active caching of the prompt prefix, the limitation of existing technologies to only single inference batch is broken, realizing cross-session KV Cache persistent reuse. Under the same hardware conditions, the system can improve throughput and reduce unit inference cost.
[0075] Step S13: weighting and fusing the business priority and the corresponding statistical priority score of each target prompt prefix to obtain a target priority score corresponding to each target prompt prefix.
[0076] In the embodiment, the mixed priority (i.e. target priority score): W_final = • W_base • W_stat.
[0077] Through the business priority mechanism, the cache of key businesses is given priority protection, meeting the service quality requirements of enterprise-level applications.
[0078] In step S14, it is determined whether the local GPU is in a preset idle state. If the local GPU is in the preset idle state, a to-be-loaded KV Cache is determined from a target KV Cache corresponding to each target prompt prefix based on the target priority score, and each to-be-loaded KV Cache is sequentially preloaded into a target video memory according to an order from high to low of the target priority score.
[0079] In this embodiment, the process of determining whether the local GPU is in the preset idle state can specifically include: monitoring the running state of the GPU in real time to obtain the GPU utilization of the GPU in a target number of continuous time windows, and determining that the target GPU is in the preset idle state if the GPU utilization of the GPU in the target number of continuous time windows is all not greater than a preset GPU utilization threshold.
[0080] In addition, in this embodiment, the process of sequentially preloading each to-be-loaded KV Cache into the target video memory can specifically include: dynamically determining a target loading speed corresponding to the to-be-loaded KV Cache according to the idle duration of the target GPU, and sequentially preloading each to-be-loaded KV Cache into the target video memory according to the target loading speed and a progressive loading strategy.
[0081] That is, the GPU idle determination: GPU_util < T2 and request_queue_length < Q_min continuously exceeds T_duration.
[0082] Preloading priority: selecting cache items in descending order of W_final.
[0083] Preloading rate control: dynamically adjusting the loading speed according to the idle duration.
[0084] Through the preloading mechanism in the GPU idle period, the delay is reduced, and the response speed of high-priority services is guaranteed.
[0085] In addition, in a specific implementation, after sequentially preloading each to-be-loaded KV Cache into the target video memory according to an order from high to low of the target priority score, the method further includes: obtaining a target service processing request sent by a target user, determining whether the target video memory contains a to-be-used KV Cache corresponding to the target service processing request, and processing the target service processing request by using the to-be-used KV Cache and a target large model if the target video memory contains the to-be-used KV Cache corresponding to the target service processing request.
[0086] In another specific embodiment, if the target memory does not contain the to-be-used KV Cache corresponding to the target business processing request, the to-be-used KV Cache corresponding to the target business processing request is calculated in real time, and the target business processing request is processed by using the to-be-used KV Cache and the target large model.
[0087] It can be seen that, by statistically calculating the access frequency, cache hit rate and business value coefficient in real time and generating a statistical priority score, the application realizes dynamic adaptive optimization of the cache strategy; by weighting and fusing the business weight and the statistical score, a hybrid decision model with business intention and real-time efficiency is formed, and the cache priority determination is refined; by monitoring the GPU idle state and preloading the KV Cache to the memory according to the priority, the idle resources are used to prepare data in advance, the response delay problem caused by the lack of preloading mechanism is solved, and finally the cache is directly called during request processing, which significantly improves the inference efficiency and response speed.
[0088] Referring to Figure 3 The embodiment of the application discloses a cache preloading device based on KV Cache, applied to a target cache management system, comprising:
[0089] The data receiving module 11 is configured to receive a target prompt prefix specified by a target user through a preset application programming interface and a business priority of a large language model prompt corresponding to the target prompt prefix;
[0090] The feature information statistical module 12 is configured to statistically calculate target feature information corresponding to each target prompt prefix in real time, and generate a statistical priority score corresponding to each target prompt prefix according to the target feature information; wherein the target feature information includes an access frequency, a cache hit rate and a business value coefficient corresponding to each target prompt prefix;
[0091] The score acquisition module 13 is configured to weight and fuse the business priority and the corresponding statistical priority score corresponding to each target prompt prefix to obtain a target priority score corresponding to each target prompt prefix.
[0092] The cache preloading module 14 is configured to determine whether the local GPU is in a preset idle state, and if the local GPU is in a preset idle state, determine a to-be-loaded KV Cache from the target KV Cache corresponding to each target prompt prefix based on the target priority score, and sequentially preload each to-be-loaded KV Cache to the target memory according to the order from high to low of the target priority score.
[0093] In some specific embodiments, the feature information statistical module 12 further comprises:
[0094] a data storage unit, configured to calculate the target KV Cache corresponding to each of the target prompt prefixes by using a target calculation method, store the target KV Cache corresponding to each of the target prompt prefixes to a preset storage space locally, and record the service identifier, creation time, and predicted use probability corresponding to each of the target KV Cache.
[0095] In some specific embodiments, the feature information statistics module 12 can specifically include:
[0096] a weight coefficient determination unit, configured to determine a first weight coefficient corresponding to the access frequency, a second weight coefficient corresponding to the cache hit rate, and a third weight coefficient corresponding to the service value coefficient;
[0097] a priority score generation unit, configured to perform weighted fusion on the access frequency, the cache hit rate, and the service value coefficient corresponding to each of the target prompt prefixes based on the first weight coefficient, the second weight coefficient, and the third weight coefficient, to generate the statistical priority score corresponding to each of the target prompt prefixes.
[0098] In some specific embodiments, the cache preloading module 14 can specifically include:
[0099] a state monitoring unit, configured to monitor the running state of the GPU in real time to obtain the GPU utilization of the GPU in a target number of continuous time windows, and determine that the target GPU is in the preset idle state if the GPU utilization of the GPU in the target number of continuous time windows is not greater than a preset GPU utilization threshold.
[0100] In some specific embodiments, the cache preloading module 14 can specifically include:
[0101] a cache preloading unit, configured to dynamically determine a target loading speed corresponding to the to-be-loaded KV Cache according to the idle duration of the target GPU, and preloading each of the to-be-loaded KV Cache to a target video memory according to the target loading speed and a progressive loading strategy.
[0102] In some specific embodiments, the cache preloading module 14 further includes:
[0103] The first request processing unit is used to obtain the target service processing request sent by the target user, determine whether the target video memory contains the KV Cache to be used corresponding to the target service processing request, and if the target video memory contains the KV Cache to be used corresponding to the target service processing request, then the target service processing request is processed using the KV Cache to be used and the target big model.
[0104] In some specific embodiments, the cache preloading module 14 further includes:
[0105] The second request processing unit is configured to, if the target video memory does not contain the KV Cache to be used corresponding to the target service processing request, calculate the KV Cache to be used corresponding to the target service processing request in real time, and process the target service processing request using the KV Cache to be used and the target large model.
[0106] Furthermore, embodiments of this application also disclose an electronic device, Figure 4 This is a structural diagram of an electronic device 20 according to an exemplary embodiment. The content of the diagram should not be construed as limiting the scope of this application.
[0107] Figure 4 This is a schematic diagram of the structure of an electronic device 20 provided in an embodiment of this application. Specifically, the electronic device 20 may include: at least one processor 21, at least one memory 22, a power supply 23, a communication interface 24, an input / output interface 25, and a communication bus 26. The memory 22 stores a computer program, which is loaded and executed by the processor 21 to implement the relevant steps in the KV Cache-based cache preloading method disclosed in any of the foregoing embodiments. Alternatively, the electronic device 20 in this embodiment may specifically be an electronic computer.
[0108] In this embodiment, the power supply 23 is used to provide operating voltage for each hardware device on the electronic device 20; the communication interface 24 can create a data transmission channel between the electronic device 20 and external devices, and the communication protocol it follows can be any communication protocol applicable to the technical solution of this application, and is not specifically limited here; the input / output interface 25 is used to acquire external input data or output data to the outside world, and its specific interface type can be selected according to specific application needs, and is not specifically limited here.
[0109] In addition, the memory 22, as a carrier for resource storage, can be a read-only memory, random access memory, disk or optical disk, etc. The resources stored thereon can include operating system 221, computer program 222, etc., and the storage method can be temporary storage or permanent storage.
[0110] The operating system 221 is configured to manage and control the various hardware devices and computer programs 222 on the electronic device 20, which can be Windows Server, Netware, Unix, Linux, etc. The computer programs 222 can further include computer programs capable of performing other specific tasks in addition to the computer programs capable of performing the KV Cache-based cache preloading method disclosed by the electronic device 20 in any of the foregoing embodiments.
[0111] Further, the present application also discloses a computer readable storage medium for storing a computer program; wherein the computer program is executed by a processor to implement the foregoing disclosed KV Cache-based cache preloading method. The specific steps of the method can refer to the corresponding content disclosed in the foregoing embodiments, which will not be repeated here.
[0112] The embodiments in the present specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts of each embodiment can be referred to each other. For the device disclosed in the embodiments, since it corresponds to the method disclosed in the embodiments, the description is relatively simple, and the relevant parts can refer to the method part.
[0113] The skilled person can further realize that the units and algorithm steps of the examples described in combination with the embodiments disclosed herein can be realized by electronic hardware, computer software or a combination of both. In order to clearly illustrate the interchangeability of hardware and software, the components and steps of the examples have been described in general terms in the foregoing description. Whether the functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. The skilled person can use different methods to realize the described functions for each specific application, but such implementation should not be considered beyond the scope of the present application.
[0114] The steps of the method or algorithm described in combination with the embodiments disclosed herein can be directly implemented by hardware, a software module executed by a processor, or a combination of both. The software module can be placed in a random access memory (RAM), a memory, a read-only memory (ROM), an electrically programmable ROM, an electrically erasable programmable ROM, a register, a hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art.
[0115] Finally, it needs to be pointed out that in this document, relational terms such as first and second and the like can only be intended to distinguish one entity or operation from another entity or operation without necessarily requiring or implying any actual such relationship or order between such entities or operations. Moreover, the terms "comprising", "including", or any other variant thereof are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but can also include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without more limitations, an element defined by the statement "comprising a" does not exclude the existence of additional identical elements in the process, method, article, or apparatus including the stated element.
[0116] The above detailed description of the technical solutions provided by the present application has been provided, and the principles and implementation modes of the present application have been described by applying specific examples. The above description of the examples is only for the purpose of helping to understand the method of the present application and its core idea; at the same time, for those skilled in the art, according to the idea of the present application, the specific implementation mode and application range will be changed, and the above description of the content of the specification should not be understood as a limitation of the present application.
Claims
1. A KV Cache based cache preloading method, characterized in that, The application is applied to a target cache management system, comprising: receiving a target prompt prefix specified by a target user through a preset application programming interface and a business priority of a large language model prompt corresponding to the target prompt prefix; real-time statistics of target feature information corresponding to each target prompt prefix, and generating a statistical priority score corresponding to each target prompt prefix according to the target feature information; wherein the target feature information includes access frequency, cache hit rate and business value coefficient corresponding to each target prompt prefix; weighting and fusing the business priority and the corresponding statistical priority score corresponding to each target prompt prefix to obtain a target priority score corresponding to each target prompt prefix; determining whether the local GPU is in a preset idle state, if the local GPU is in a preset idle state, determining the target KV Cache to be loaded from the target KV Cache corresponding to each target prompt prefix based on the target priority score, and preloading each target KV Cache to the target video memory in order according to the order from high to low of the target priority score.
2. The KV Cache based cache preloading method of claim 1, wherein, Before the real-time statistics of the target feature information corresponding to each target prompt prefix, and the generation of the statistical priority score corresponding to each target prompt prefix according to the target feature information, it further comprises: calculating the target KV Cache corresponding to each target prompt prefix by using a target calculation method, storing the target KV Cache corresponding to each target prompt prefix to a preset storage space in the local, and recording the business identifier, creation time and predicted use probability corresponding to each target KV Cache.
3. The KV Cache based cache preloading method of claim 1, wherein, The generation of the statistical priority score corresponding to each target prompt prefix according to the target feature information comprises: determining a first weight coefficient corresponding to the access frequency, a second weight coefficient corresponding to the cache hit rate and a third weight coefficient corresponding to the business value coefficient; weighting and fusing the access frequency, the cache hit rate and the business value coefficient corresponding to each target prompt prefix based on the first weight coefficient, the second weight coefficient and the third weight coefficient to generate the statistical priority score corresponding to each target prompt prefix.
4. The KV Cache based cache preloading method of claim 1, wherein, The determination of whether the local GPU is in a preset idle state comprises: real-time monitoring the running state of the GPU to obtain the GPU utilization rate of the GPU in a target number of continuous time windows, if the GPU utilization rate of the GPU in the target number of continuous time windows is not greater than a preset GPU utilization rate threshold, it is determined that the target GPU is in the preset idle state.
5. The KV Cache based cache preloading method of claim 1, wherein, The preloading of each target KV Cache to the target video memory comprises: Determine the target loading speed corresponding to the to-be-loaded KV Cache according to the idle duration of the target GPU, and pre-load each to-be-loaded KV Cache to the target GPU memory in turn according to the target loading speed and a progressive loading strategy.
6. The KV Cache based cache preloading method according to any one of claims 1 to 5, characterized in that, After pre-loading each to-be-loaded KV Cache to the target GPU memory in turn according to the order from high to low of the target priority scores, the method further comprises: Obtain a target service processing request sent by the target user, and determine whether the target GPU memory contains a to-be-used KV Cache corresponding to the target service processing request.
7. The KV Cache based cache preloading method of claim 6, wherein, After pre-loading each to-be-loaded KV Cache to the target GPU memory in turn according to the order from high to low of the target priority scores, the method further comprises: If the target GPU memory does not contain the to-be-used KV Cache corresponding to the target service processing request, calculate the to-be-used KV Cache corresponding to the target service processing request in real time, and process the target service processing request by using the to-be-used KV Cache and the target large model.
8. A KV Cache based cache preloading apparatus, characterized by, The application is applied to a target cache management system, which comprises: A data receiving module is configured to receive a target prompt prefix and a business priority of a large language model prompt corresponding to the target prompt prefix specified by a target user through a preset application programming interface. A feature information statistical module is configured to statistically determine target feature information corresponding to each target prompt prefix in real time, and generate a statistical priority score corresponding to each target prompt prefix according to the target feature information. A score obtaining module is configured to weight and fuse the business priority and the corresponding statistical priority score corresponding to each target prompt prefix to obtain a target priority score corresponding to each target prompt prefix. A cache pre-loading module is configured to determine whether a local GPU is in a preset idle state, and if the local GPU is in the preset idle state, determine a to-be-loaded KV Cache from target KV Caches corresponding to each target prompt prefix based on the target priority score, and pre-load each to-be-loaded KV Cache to a target GPU memory in turn according to the order from high to low of the target priority score.
9. An electronic device, comprising: The application comprises: A memory is configured to save a computer program. A processor is configured to execute the computer program to implement the cache pre-loading method based on KV Cache according to any one of claims 1 to 7.
10. A computer-readable storage medium, characterized in that, The application is configured to save a computer program, and the computer program is executed by a processor to implement the cache pre-loading method based on KV Cache according to any one of claims 1 to 7.
Citation Information
Cited By
Cache data scheduling method and electronic equipment
CN122332073A