Video memory management method and device for large language model reasoning, medium and product

Abstract

The invention provides a video memory management method and device for large language model reasoning, a medium and a product. The method and device are applied to service equipment deployed with a large language model. The method comprises the following steps: allocating physical video memory resources on the service equipment so as to respectively map the physical video memory resources to a first video memory resource pool of which a cache object is a key value cache and a second video memory resource pool of which a cache object is an intermediate activation value; and aiming at the reasoning task submitted to the large language model, if it is judged that idle video memory resources of any one of the first video memory resource pool and the second video memory resource pool are not enough to cache a corresponding cache object for the reasoning task, sending the cache object to the first video memory resource pool and the second video memory resource pool, and if not, temporarily borrowing at least a part of idle video memory resources of another video memory resource pool to any video memory resource pool.

Description

Technical Field

[0001] One or more embodiments of this specification relate to the field of artificial intelligence technology, and in particular, to a method, device, medium, and product for managing video memory for large language model reasoning. Background Art

[0002] Large Language Model (LLM) is a type of neural network model based on deep learning, designed for understanding and generating natural language text. By learning the statistical laws and semantic structure of language through large-scale training data sets, LLM can show excellent performance in a variety of natural language processing tasks, including text generation, machine translation, sentiment analysis, and question-answering systems. Due to its strong generalization ability and versatility, LLM has been widely used in the field of artificial intelligence.

[0003] LLM requires a large amount of GPU (Graphics Processing Unit) memory during the inference process. If the memory resources are not properly scheduled, it may be necessary to deploy relatively more memory resources, resulting in excessively high infrastructure costs, or the inference speed and throughput performance may be affected due to insufficient memory resources. Summary of the invention

[0004] In view of this, one or more embodiments of this specification provide the following technical solutions:

[0005] According to a first aspect of one or more embodiments of this specification, a method for managing video memory for large language model inference is proposed, which is applied to a service device deployed with a large language model; the method includes:

[0006] Allocating physical video memory resources on the service device to be mapped to a first video memory resource pool whose cache objects are key-value caches and a second video memory resource pool whose cache objects are intermediate activation values, respectively;

[0007] For the reasoning task submitted to the large language model, if it is determined that the free memory resources of any one of the first memory resource pool and the second memory resource pool are insufficient to cache the corresponding cache object for the reasoning task, at least a portion of the free memory resources of the other memory resource pool are temporarily loaned to any one of the memory resource pools.

[0008] According to a second aspect of one or more embodiments of this specification, an electronic device is proposed, comprising: a processor; a memory for storing processor-executable instructions; wherein the processor implements the steps of the method described in the first aspect by running the executable instructions.

[0009] According to a third aspect of one or more embodiments of the present specification, a computer-readable storage medium is provided, on which computer instructions are stored, and when the instructions are executed by a processor, the steps of the method described in the first aspect are implemented.

[0010] According to a fourth aspect of one or more embodiments of the present specification, a computer program product is provided, comprising a computer program / instruction, which implements the steps of the method described in the first aspect when executed by a processor.

[0011] As can be seen from the above embodiments, this specification creates a corresponding first video memory resource pool and a second video memory resource pool for the key-value cache and the intermediate activation value, respectively, and mutual interference between the key-value cache and the intermediate activation value can be avoided when the video memory resources are sufficient. At the same time, by uniformly managing the first video memory resource pool and the second video memory resource pool, this specification can dynamically schedule and flexibly manage idle video memory resources according to actual needs, so that the key-value cache can temporarily borrow the video memory resources originally allocated to the second video memory resource pool for the intermediate activation value, and the intermediate activation value can also temporarily borrow the video memory resources originally allocated to the first video memory resource pool for the key-value cache, which greatly improves the scheduling flexibility and utilization of video memory resources, thereby helping to shorten the reasoning time of LLM and improve the throughput performance of LLM under the condition of certain infrastructure costs. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] Figure 1 The present invention is a flowchart of a method for managing video memory for large language model reasoning provided by an exemplary embodiment.

[0013] Figure 2 A flowchart of video memory scheduling in the Prefill phase is provided by an exemplary embodiment.

[0014] Figure 3 The present invention is a flowchart of performing video memory scheduling in the Decode stage provided by an exemplary embodiment.

[0015] Figure 4 The diagram is a schematic diagram of the architecture of a scheduling and management system for video memory resources provided by an exemplary embodiment.

[0016] Figure 5 It is a structural schematic diagram of a device provided by an exemplary embodiment.

[0017] Figure 6 It is a block diagram of a video memory management device for large language model reasoning provided by an exemplary embodiment. DETAILED DESCRIPTION

[0018] The reasoning process of the Transformer-based large language model (LLM) is usually divided into two main stages: the Prefill stage and the Decode stage. In the Prefill stage, the LLM generates the first output token and a tensor called the Key-Value Cache (KV Cache) based on the prompt word (prompt) input by the user. In the Decode stage, the LLM will repeatedly perform the Auto-Regression step. Each time the Auto-Regression step is performed, the LLM will take the previous output token and the generated KV Cache as input to generate the next output token and the updated KV Cache. For example, when the Auto-Regression step is performed for the first time, the previous output token is the first output token generated in the Prefill stage, and the generated KV Cache is the KV Cache generated in the Prefill stage; when the Auto-Regression step is performed for the second time, the previous output token is the output token generated by the first execution of the Auto-Regression step, and the generated KV Cache is the KV Cache updated by the first execution of the Auto-Regression step; and so on, until the LLM generates a complete output.

[0019] Since the generated KV Cache needs to be continuously used to generate output tokens during the Decode phase, the KV Cache will also be cached in the GPU's video memory to improve its access performance. In addition to the KV Cache, the LLM reasoning process also requires a large amount of video memory resources to cache weights, intermediate activations (ACTs), etc.

[0020] Weights are part of the LLM model parameters, which are determined after LLM training is completed. That is, during the LLM reasoning process, the amount of memory resources occupied by weights is fixed. Therefore, for service devices that deploy LLM, the memory resources required by the weights of the deployed LLM can be calculated in advance, and the physical memory resources on the service device can be accurately allocated accordingly. Then, the remaining physical memory resources need to be reasonably allocated and scheduled between KV Cache and ACT.

[0021] In related technologies, the physical video memory resources of KV Cache or ACT may be insufficient, thereby affecting the smooth execution of LLM on current and subsequent reasoning tasks, such as increasing the first word delay or reducing throughput performance.

[0022] In view of the deficiencies in the related art, this specification proposes a video memory management solution for large language model reasoning, which will be described in detail below with reference to the accompanying drawings.

[0023] Figure 1 FIG. 1 is a flowchart of a method for managing video memory for large language model reasoning provided by an exemplary embodiment. Figure 1 As shown, the method is applied to a service device deployed with a large language model; the method comprises:

[0024] Step 102: Allocate physical video memory resources on the service device to map them to a first video memory resource pool whose cache objects are key-value caches and a second video memory resource pool whose cache objects are intermediate activation values.

[0025] The technical solution of this specification is applicable to all types of LLM in the relevant technology, that is, this specification does not limit this.

[0026] For example, from the perspective of the field: the LLM of this specification can be a general field big model or a vertical field big model, and this specification does not limit this. Among them, the so-called general field big model refers to a general big model that is trained by widely collecting data samples from multiple fields and can handle reasoning tasks in multiple fields. The so-called vertical field big model refers to a big model that is able to handle reasoning tasks in a specific target field by fine-tuning the general field big model based on the data samples of the general field big model.

[0027] For another example, from the perspective of modality: the LLM in this specification can be a traditional text reasoning model for natural language tasks, and its applicable prompt can specifically be text data. Of course, the LLM in this specification can also be a multimodal LLM, and its applicable prompt can include one or more types of data such as text, audio, picture, video, etc.

[0028] As mentioned above, in the LLM reasoning scenario, the cache objects that need to occupy physical video memory resources may include weights, KVCache, and ACT. The physical video memory resources required by the weights can be allocated first, and the remaining physical video memory resources can be allocated to the first video memory resource pool and the second video memory resource pool through the technical solution of this specification, so as to be used for caching KVCache and ACT respectively. Of course, if the physical video memory resources for weights and the physical video memory resources for KV Cache and ACT on the service device are independent of each other, then there is no need to consider the weight issue, and the physical video memory resources can be directly allocated and scheduled for KV Cache and ACT.

[0029] The operation of allocating physical video memory resources to the first video memory resource pool and the second video memory resource pool may be an initial allocation operation after the LLM is deployed to the service device, and adjustments are made on this basis in the subsequent reasoning process. Of course, if there is indeed a relevant demand in the reasoning process, such as finding that the initial allocation scheme is not suitable in the reasoning process, such as causing too frequent dynamic scheduling, then theoretically it can also be reallocated, and this specification does not limit this.

[0030] The physical memory resources on the service device may include GPU memory. Of course, if the service device uses other types of chips for LLM reasoning, the physical memory resources may also come from the other types of chips.

[0031] Taking GPU as an example, the service device can be equipped with one or more graphics cards. If there is only one graphics card, the physical graphics memory resources allocated to the first graphics memory resource pool and the second graphics memory resource pool come from the graphics memory on this graphics card. If multiple graphics cards are included, then: in one case, each graphics card can be managed separately, that is, for the graphics memory on each graphics card, it is allocated as a separate physical graphics memory resource, and each physical graphics memory resource has a corresponding first graphics memory resource pool and a second graphics memory resource pool. Therefore, in this scenario, the secondment of graphics memory resources between the first graphics memory resource pool and the second graphics memory resource pool will not involve cross-graphics cards. In another case, the video memories of multiple graphics cards can be allocated as a unified physical video memory resource. Then the physical video memory resources allocated to the first video memory resource pool may come from one or more graphics cards, and the second video memory resource pool is similar. Therefore, in this scenario, the loan of video memory resources between the first video memory resource pool and the second video memory resource pool may involve cross-graphics cards. For example, the physical video memory resources originally used by the first video memory resource pool come from graphics card ①, while the physical video memory resources loaned by the second video memory resource pool come from graphics card ②.

[0032] Among them, in the case where there are multiple graphics cards at the same time, each graphics card can be used as a worker, and by forming a shard of the LLM instance on each worker, the parallel processing of the reasoning task can be realized. In the process of parallel processing, the video memory management scheme of this specification is orthogonal to various parallel strategies in the related technology, so no matter which LLM instance adopts the parallel strategy, the video memory management scheme of this specification can be superimposed. For example, parallel strategies can include TP (Tensor Parallelism), PP (Pipeline Parallelism), EP (Expert Parallelism), etc., and this specification does not limit this.

[0033] For the allocation operation of physical video memory resources, the processing method in the related art can be adopted, and this specification does not limit this. Taking the first video memory resource pool as an example, a page table T1 corresponding to the first video memory resource pool can be established, and the mapping relationship between the logical video memory address of the first video memory resource pool and the physical video memory address of the allocated physical video memory resource is recorded in the page table T1. Similarly, the second video memory resource pool can also have a corresponding page table T2, which is used to record the mapping relationship between the logical video memory address of the second video memory resource pool and the physical video memory address of the allocated physical video memory resource.

[0034] As for the proportion or quantity of physical video memory resources actually allocated to the first video memory resource pool and the second video memory resource pool when allocating physical video memory resources to the two video memory resource pools, this involves the allocation strategy actually adopted, and this specification does not limit this.

[0035] For example, it is usually difficult to estimate the number of tokens contained in the output generated by LLM for prompt, and the amount of memory resources occupied by KVCache in the Decode stage will increase as the number of tokens increases, so it is usually difficult to accurately calculate the amount of physical memory resources required by KV Cache. Therefore, the amount of physical memory resources required by ACT can be estimated first and the corresponding physical memory resources can be allocated to the second memory resource pool, and then the remaining physical memory resources can be allocated to the first memory resource pool.

[0036] For any reasoning task, the memory usage of ACT in the Prefill stage is positively correlated with the prompt length of the reasoning task. For example, when the prompt length is 1K, the memory usage of ACT in the Prefill stage is 2G, and when the prompt length is 8K, the memory usage of ACT in the Prefill stage is 16G. In addition, the memory usage of ACT in the Decode stage is the size of the space occupied by an output token (specifically the previous output token). The specific value is related to the embedding dimension (Embedding Dimension) and data type (such as 32-bit floating point number float32, 16-bit floating point number float16 or integer int8, etc.) actually used by the output token.

[0037] It can be seen that if the prompt length is different, the memory resource usage of the ACT generated by different reasoning tasks in the Prefill stage is not the same. But what can be determined is that for a certain LLM, since it sets the maximum prompt word length, that is, it limits the maximum length of the prompt of each reasoning task, the memory resource usage of the ACT generated by it in the Prefill stage will not exceed the memory resource usage corresponding to the maximum prompt word length. Therefore, the maximum prompt word length set by the large language model deployed on the above-mentioned service device can be determined, and then the maximum prompt word length is used as the assumed prompt word length (that is, it is assumed that the length of the prompt of each reasoning task received by the LLM during the reasoning process is the maximum prompt word length), and matching physical memory resources are allocated to the second memory resource pool.

[0038] Specifically, for example, for a certain LLM, when the length of prompt is the above-mentioned maximum prompt word length, the corresponding video memory resource occupancy is Z. Then: assuming that the number of inference tasks contained in each batch during the preset batching is n, the amount of video memory resources allocated to the second video memory resource pool may be no less than n*Z, so that the second video memory resource pool can provide video memory resources for caching ACT for at least n inference tasks at the same time.

[0039] Of course, in the actual reasoning process, the prompt of the reasoning task often does not really reach the maximum prompt word length, and may even differ greatly from the maximum prompt word length. For each batch, it may cause a waste of video memory resources in the second video memory resource pool. Therefore, the assumed prompt word length can be set to an actual length less than the above-mentioned maximum prompt word length (that is, assuming that the length of the prompt of each reasoning task received by the LLM during the reasoning process is the actual length), and matching physical video memory resources are allocated to the second video memory resource pool, and the remaining physical video memory resources are allocated to the first video memory resource pool.

[0040] For example, the above actual length is obtained by the data statistics of historical reasoning tasks, and the prompt word length of historical reasoning tasks that is not less than the preset ratio is not greater than the actual length. For example, assuming that the preset ratio is 90%, and through statistical analysis of historical reasoning tasks, it is found that although the maximum prompt word length of LLM is 4K, at least 90% of the prompt lengths of historical reasoning tasks are not greater than 1K, then the above actual length can be set to 1K, that is, the video memory resource occupancy of the ACT generated by these historical reasoning tasks in the Prefill stage does not exceed 2G, which is far less than the 8G required for a prompt with a length of 4K, so the video memory resource occupancy Z of the ACT generated by each reasoning task in the Prefill stage can be set to 2G; if the number of reasoning tasks in the batch is set to n=5, the amount of physical video memory resources allocated to the second video memory resource pool is 5*2G=10G, which is far less than 5*8G=40G. In the actual reasoning process, even if there are a small number of reasoning tasks with prompt lengths exceeding 1K in the batch, since the prompt lengths of most reasoning tasks are no more than 1K, the overall ACT memory resource usage usually does not exceed the memory resources in the second memory resource pool. Of course, if there is indeed a shortage of free memory resources in the second memory resource pool, then the technical solution of this specification can be used to borrow the free memory resources required from the first memory resource pool, which will be described in detail below.

[0041] If it is considered that the video memory resources can be borrowed at any time during the reasoning process through the technical solution of this specification, then taking the above-mentioned setting of the video memory resource occupancy Z of ACT to 2G as an example, no matter how much the batch size is set in the actual reasoning process, the batch size can be defaulted to 1, so the amount of physical video memory resources initially allocated to the second video memory resource pool can be 2G, and can be borrowed from the first video memory resource pool at any time according to actual conditions. Of course, if the amount of physical video memory resources initially allocated to the second video memory resource pool is determined according to the actual size of the batch (such as 5 listed above) as described above, the occurrence of borrowing operations can be minimized to avoid the additional overhead caused by it.

[0042] Of course, the above-mentioned actual length can also be set in other ways, and does not necessarily need to be obtained from the data statistics of historical reasoning tasks. For example, it can be set to an empirical value, or even to an arbitrary value. In fact, as mentioned above, even if the actual length is set to be relatively small, resulting in fewer physical video memory resources allocated to the second video memory resource pool, if there is indeed a situation where there is insufficient free video memory resources in the second video memory resource pool, the technical solution of this specification can be used to temporarily borrow video memory resources to meet the ACT's demand for video memory resources.

[0043] Step 104, for the reasoning task submitted to the large language model, if it is determined that the free memory resources of any one of the first memory resource pool and the second memory resource pool are insufficient to cache the corresponding cache object for the reasoning task, at least a portion of the free memory resources of the other memory resource pool are temporarily loaned to any one of the memory resource pools.

[0044] In the process of executing the reasoning task through LLM, if KV Cache is generated, the memory resources in the first memory resource pool are used for caching, and if ACT is generated, the memory resources in the second memory resource pool are used for caching. When the memory resources in the first memory resource pool and the second memory resource pool are both sufficient, the cache operations for KV Cache and ACT are independent of each other and do not interfere with each other. On this basis, this manual also manages the first memory resource pool and the second memory resource pool in a unified manner, so as to realize the temporary transfer of idle memory resources between the two memory resource pools to meet the cache requirements of the reasoning task.

[0045] Whether in the Prefill stage or the Decode stage, there may be a need to temporarily schedule idle video memory resources. The following will list the situations where these two stages may have such needs, but this specification does not limit this.

[0046] Let’s take the Prefill stage as an example.

[0047] After receiving the inference task, the video memory resource requirements of the KV Cache and ACT of the inference task in the Prefill stage can be calculated respectively; if the free amount of video memory resources in any of the first video memory resource pool and the second video memory resource pool is less than the calculated video memory resource requirements corresponding to the corresponding cache object, it is determined that the free video memory resources in any of the video memory resource pools are insufficient to cache the corresponding cache object for the inference task, and the technical solution of this specification can be used to temporarily borrow the free video memory resources. If after the borrowing, the free amount of video memory resources in the first video memory resource pool and the second video memory resource pool can be satisfied that both are not less than the calculated video memory resource requirements corresponding to the corresponding cache object, or the free amount of video memory resources in the first video memory resource pool and the second video memory resource pool can be satisfied that both are not less than the calculated video memory resource requirements corresponding to the corresponding cache object without borrowing, then the corresponding inference task can be advanced to the Prefill stage.

[0048] For example, Figure 2 FIG. 1 is a flowchart of a video memory scheduling method in the Prefill phase provided by an exemplary embodiment. Figure 2 As shown, the video memory scheduling in the Prefill phase may include the following steps:

[0049] In step 202, an inference task is received.

[0050] Here, taking any inference task as an example, the processing logic of video memory scheduling is introduced.

[0051] In step 204, the video memory resource requirements P1 and S1 of the KV Cache and ACT are calculated respectively.

[0052] As mentioned above, based on the prompt in the received inference task, specifically the actual length of the prompt, and combined with the data type and other attributes set by the LLM itself, the memory resource requirements of the KV Cache and ACT corresponding to the prompt in the Prefill stage can be calculated, that is, how much memory resources each needs to occupy. Assume that the memory resource requirement of the KV Cache is P1 and the memory resource requirement of the ACT is S1.

[0053] In step 206, it is determined whether P1≤P0 and S1≤S0 are satisfied.

[0054] As mentioned above, a first video memory resource pool for caching KV Cache and a second video memory resource pool for caching ACT can be created respectively. Here, the first video memory resource pool is referred to as the KV video memory resource pool and the second video memory resource pool is referred to as the ACT video memory resource pool. Through the continuous management of the KV video memory resource pool and the ACT video memory resource pool, the amount of free video memory resources in each of the two video memory resource pools can be determined respectively. Free video memory resources refer to video memory resources that have not been occupied in the corresponding video memory resource pool, or video memory resources that are marked as free in the corresponding video memory resource pool. Assume that the amount of free video memory resources in the KV video memory resource pool is P0, and the amount of free video memory resources in the ACT video memory resource pool is S0.

[0055] If P1≤P0 is satisfied, it indicates that the free memory resources in the KV memory resource pool are sufficient to cache the KV Cache generated by the current reasoning task in the Prefill stage. Similarly, if S1≤S0 is satisfied, it indicates that the free memory resources in the ACT memory resource pool are sufficient to cache the ACT generated by the current reasoning task in the Prefill stage. Then, if P1≤P0 and S1≤S0 are satisfied at the same time, you can jump directly to step 210, that is, advance the current reasoning task to the Prefill stage for processing.

[0056] If P1>P0, it means that the free memory resources in the KV memory resource pool are insufficient to cache the KV Cache generated by the current inference task in the Prefill stage, and the current inference task will not be advanced to the Prefill stage. Similarly, if S1>S0, it means that the free memory resources in the ACT memory resource pool are insufficient to cache the ACT generated by the current inference task in the Prefill stage, and the current inference task will not be advanced to the Prefill stage.

[0057] In step 208a, if P1>P0 and the ACT video memory resource pool has sufficient memory, idle video memory resources are temporarily transferred from the ACT video memory resource pool to the KV video memory resource pool.

[0058] As mentioned above, P1>P0 indicates that there are insufficient free memory resources in the KV memory resource pool. At this time, if the free memory resources in the ACT memory resource pool are sufficient to cache ACT on the one hand, and sufficient to be transferred to the KV memory resource pool on the other hand, it indicates that the ACT memory resource pool is sufficient. For example, assuming P1=4G, the KV Cache of the inference task in the Prefill stage needs to occupy 4G of memory resources. If P0=3G, the free memory resources in the KV memory resource pool are 3G, which is not enough to cache the 4G KVCache, and there is still a lack of 1G of memory resources. Then it is necessary to determine whether the free memory resources of the ACT video memory resource pool are sufficient: assuming S0 = 5G, that is, the free memory resources of the ACT video memory resource pool are 5G; if S1 = 2G, that is, the ACT of the inference task in the Prefill stage needs to occupy 2G of video memory resources, then it can be determined that S0-S1 = 3G>1G, indicating that in addition to being able to cache ACT, the ACT video memory resource pool also has sufficient free memory resources that can be transferred to the KV video memory resource pool for caching KV Cache, so it is determined that the ACT video memory resource pool is sufficient. If S0 = 2.5G, then although the free memory resources of the ACT video memory resource pool are sufficient to cache ACT, the remaining 0.5G free memory resources are not enough to fill the above 1G video memory resources required by KVCache, so it is determined that the ACT video memory resource pool is not sufficient. If S0 < 2G, the ACT video memory resource pool cannot even cache ACT, and it should also be determined that the ACT video memory resource pool is not sufficient.

[0059] On the premise that the ACT memory resource pool is sufficient, a portion of the idle memory resources in the ACT memory resource pool can be temporarily transferred to the KV memory resource pool. The amount of the transferred memory resources depends on the difference between the idle memory resources in the KV memory resource pool and the memory resource requirements of the KV Cache of the inference task in the Prefill phase. For example, in the above example, P1=4G, P0=3G, and the difference can be determined to be P1-P0=1G, so 1G of idle memory resources can be temporarily transferred from the ACT memory resource pool to the KV memory resource pool.

[0060] In step 208b, if S1>S0 and the KV memory resource pool has sufficient memory, idle memory resources are temporarily transferred from the KV memory resource pool to the ACT memory resource pool.

[0061] As mentioned above, S1>S0 indicates that the free memory resources in the ACT memory resource pool are insufficient. At this time, if the free memory resources in the KV memory resource pool are sufficient for caching the KV Cache on the one hand, and sufficient for loaning to the ACT memory resource pool on the other hand, it indicates that the KV memory resource pool is sufficient. For example, assuming S1=8G, the ACT of the inference task in the Prefill stage needs to occupy 8G of memory resources. If S0=5G, the free memory resources in the ACT memory resource pool are 5G, which is not enough to cache the 8G ACT, and there is still a lack of 3G of memory resources. Then it is necessary to determine whether the free memory resources of the KV memory resource pool are sufficient: assuming P0 = 15G, that is, the free memory resources of the KV memory resource pool are 15G; if P1 = 10G, that is, the KV Cache of the inference task in the Prefill stage needs to occupy 10G of memory resources, then it can be determined that P0-P1 = 5G> 3G, indicating that in addition to caching the KV Cache, the KVCache memory resource pool also has sufficient free memory resources that can be transferred to the ACT memory resource pool for caching the ACT, so it is determined that the KV memory resource pool is sufficient. If P0 = 12G, then although the free memory resources of the KV memory resource pool are sufficient to cache the KV Cache, the remaining 2G free memory resources are not enough to fill the above 3G memory resources required by the ACT, so it is determined that the KV Cache memory resource pool is not sufficient. If P0 is less than 10G, the KV memory resource pool cannot even cache the KV Cache, and it should also be determined that the KV memory resource pool is insufficient.

[0062] On the premise that the KV memory resource pool is sufficient, a portion of the idle memory resources in the KV memory resource pool can be temporarily loaned to the ACT memory resource pool. The amount of loaned memory resources depends on the difference between the idle memory resources in the ACT memory resource pool and the memory resource requirements of the ACT during the prefill phase of the inference task. For example, in the above example, S1=8G, S0=5G, and the difference can be determined to be S1-S0=3G, so 3G of idle memory resources can be temporarily loaned from the KV memory resource pool to the ACT memory resource pool.

[0063] In step 208c, for other situations, the inference task needs to be in a waiting state and cannot be advanced to the Prefill stage through the scheduling of video memory resources.

[0064] Other situations may include:

[0065] P1>P0 and S1>S0, that is, the free memory resources of the KV memory resource pool are insufficient to cache the KV Cache, and the free memory resources of the ACT memory resource pool are insufficient to cache the ACT.

[0066] P1>P0 and S1<S0, but the ACT video memory resource pool is insufficient. In this regard, the description of the relevant example is provided above and will not be repeated here.

[0067] S1>S0 and P1<P0, but the KV memory resource pool is insufficient. In this regard, the description of the relevant example is provided above and will not be repeated here.

[0068] In step 210 , the inference task is advanced to the Prefill stage.

[0069] In the batching scenario, multiple inference tasks of the same batch can be simultaneously advanced to the Prefill stage. After all inference tasks in the batch have completed the processing of the Prefill stage, each inference task will enter the subsequent Decode stage.

[0070] By analyzing the memory resource usage of KV Cache and ACT in the Prefill stage and the Decode stage, we can find their characteristics:

[0071] For KV Cache, it needs to occupy more video memory resources in the Prefill stage. Since the stored KV Cache needs to be accessed and updated when the autoregressive step is executed in the Decode stage, not only will the KV Cache generated in the Prefill stage still occupy the original video memory resources after entering the Decode stage, but it will also gradually occupy more video memory resources with each execution of the autoregressive step. For example, the video memory resources occupied by KV Cache in the Prefill stage are 4G, and in the Decode stage, they will be further gradually increased to 8G on this basis.

[0072] For ACT, it needs to occupy more video memory resources in the Prefill stage, but it is reduced to only storing the previous output token in the Decode stage. For example, ACT occupies 2G of video memory resources in the Prefill stage, and only occupies the storage space of one output token in the Decode stage.

[0073] It can be seen that for ACT, if the idle video memory resources of the first video memory resource pool are temporarily loaned to the second video memory resource pool for the reasoning task before the Prefill stage, then after the Prefill stage, the loaned video memory resources will be returned from the second video memory resource pool to the first video memory resource pool, so that this part of the video memory resources can be used for caching KVCache in the Decode stage, or for caching KV Cache of other reasoning tasks in the Prefill stage.

[0074] The above lists the temporary scheduling requirements for idle video memory resources in the Prefill phase and the corresponding video memory scheduling process. The following is an explanation of the Decode phase.

[0075] As mentioned above, when the reasoning task is in the Decode stage, as the output token is continuously generated, the memory resource occupancy of the KV Cache also increases, so the memory resource consumption of the first memory resource pool can be determined in real time: if the memory resources of the first memory resource pool have been exhausted or the proportion of idle memory resources is less than the warning ratio, it is determined that the idle memory resources of the first memory resource pool are insufficient to cache the corresponding cache object for the reasoning task, and the idle memory resources can be temporarily borrowed through the technical solution of this specification.

[0076] For example, Figure 3 FIG. 1 is a flowchart of a video memory scheduling method in the Decode stage provided by an exemplary embodiment. Figure 3 As shown, the video memory scheduling in the Decode stage may include the following steps:

[0077] In step 302, an autoregressive step is performed.

[0078] As mentioned earlier, when performing the autoregressive step, LLM takes the previous output token and the cached KV Cache as input to generate the next output token and the updated KV Cache.

[0079] In step 304, an output token is generated.

[0080] The output token here is the so-called "next output token" generated by this round of autoregressive step.

[0081] In step 306, it is determined whether it is an end token.

[0082] The output generated by LLM includes several output tokens. When the output token generated by a round of autoregressive steps is the end token, LLM confirms that the complete output has been generated and does not repeat the autoregressive steps. In short, the end token indicates the termination of the output, which may be called differently in different LLMs, and this specification does not limit this.

[0083] If the output token generated by this round of autoregressive step is not the end token, proceed to step 308 .

[0084] In step 308 , it is determined whether the P0 ratio is less than 10%. If so, the process proceeds to step 310 , otherwise, the process returns to step 302 .

[0085] As mentioned above, when the proportion of free video memory resources in the first video memory resource pool is less than the warning ratio, it can be borrowed from the second video memory resource pool. The 10% here is an exemplary warning ratio, so that when the proportion of free video memory resources P0 in the KV video memory resource pool is less than 10% and not exhausted, the free video memory resources are temporarily borrowed from the ACT video memory resource pool in advance. Then for the reasoning task, the KV video memory resource pool can always provide enough free video memory resources, and there is no need to temporarily wait for the temporary borrowing of free video memory resources from the ACT video memory resource pool to the KV video memory resource pool, which will not delay the generation of tokens and ensure the efficient execution of the reasoning task. Among them, this specification does not limit the value of the warning ratio. For example, it can be adjusted according to the number of reasoning tasks processed in parallel: when the number of reasoning tasks processed in parallel is more, the overall consumption speed of video memory resources is often relatively faster, so the value of the warning ratio can be relatively increased, so that there is enough time to complete the temporary borrowing operation in time.

[0086] Of course, even after the free memory resources in the KV memory resource pool are exhausted, the free memory resources can be temporarily borrowed from the ACT memory resource pool to the KV memory resource pool. Compared with not adopting the temporary borrowing and waiting for other reasoning tasks to complete and release the occupied memory resources, the temporary borrowing processing solution can obviously shorten the waiting time of the reasoning task, thereby speeding up the execution speed of the reasoning task.

[0087] In step 310, at least a portion of idle video memory resources in the ACT video memory resource pool is temporarily transferred to the KV video memory resource pool.

[0088] and Figure 2 Similar to the illustrated embodiment, if it is necessary to temporarily borrow idle video memory resources from the ACT video memory resource pool, it should be ensured that there are sufficient idle video memory resources in the ACT video memory resource pool: both to meet the ACT's own cache requirements and to be borrowed to the KV video memory resource pool. As mentioned above, during the process of LLM processing the reasoning task, the demand for video memory resources of ACT in the Decode stage drops sharply. Therefore, when the reasoning task completes the processing of the Prefill stage, most of the video memory resources occupied by it in the ACT video memory resource pool can be turned into idle resources and can be temporarily borrowed to the KV video memory resource pool for caching KVCache, thereby improving the utilization of video memory resources and the processing efficiency of reasoning tasks.

[0089] Since the number of tokens contained in the output generated by LLM for prompt cannot be accurately estimated, when the free memory resources of the KV memory resource pool are exhausted or nearly exhausted in the Decode stage, it is impossible to determine how many output tokens will be generated later. Therefore, a certain borrowing strategy is needed to determine the amount of free memory resources to be transferred from the ACT memory resource pool to the KV memory resource pool.

[0090] For example, a fixed borrowing amount may be predefined so that the corresponding idle video memory resources are temporarily borrowed from the ACT video memory resource pool according to the borrowing amount each time. This specification does not limit the specific value of the borrowing amount, for example, it may be an empirical value defined according to actual conditions.

[0091] For another example, according to the business scenario to which the reasoning task belongs, the memory resource occupancy of the KV Cache based on the data statistics of historical reasoning tasks in the business scenario can be determined; then, the difference between the free memory resources of the KV memory resource pool and the calculated memory resource occupancy is calculated, and the free memory resources corresponding to the difference in the ACT memory resource pool are temporarily loaned to the KV memory resource pool.

[0092] Since the questions raised by users in the same business scenario often have certain regularities, and the content of the corresponding answers is often within a certain length range, the amount of video memory resources occupied by KV Cache can be estimated to a certain extent by statistically analyzing the data of historical reasoning tasks in the business scenario. Among them, this manual does not limit the division method and granularity of business scenarios. For example, the business scenarios can be divided into corresponding business scenarios according to dimensions such as education, life, emotion, technology, and literature.

[0093] Of course, the above estimation based on business scenarios is bound to have certain deviations. If the estimated occupancy is too large, the amount of idle video memory resources transferred from the ACT video memory resource pool to the KV video memory resource pool will also be relatively large, and the extra idle video memory resources will actually be wasted in the KV video memory resource pool; if the estimated occupancy is too small, it only needs to be transferred again, which will increase certain expenses but help improve the utilization of video memory resources.

[0094] For example, if it is the first borrowing of the inference task in the Decode stage, the difference calculated and estimated by the above business scenario can be used to temporarily transfer the idle video memory resources corresponding to the difference in the ACT video memory resource pool to the KV video memory resource pool. If it is not the first borrowing of the inference task in the Decode stage, it means that no matter what method is used for estimation before, the idle video memory resources borrowed are too few, and it is necessary to temporarily transfer the idle video memory resources from the ACT video memory resource pool to the KV video memory resource pool again or more times. As the number of borrowings increases, the number of output tokens generated by LLM for the inference task also increases, and it is getting closer and closer to the end of the output. It can be seen that: in the case of non-first borrowing, although it can still be borrowed according to the borrowing amount used in the first borrowing, as the number of borrowings increases, the probability of wasting the borrowed idle video memory resources (output ends before they are used up) will continue to increase. Therefore, in order to reduce the waste of video memory resources, in the case of non-first borrowing, the amount of idle video memory resources borrowed can be gradually reduced when the idle video memory resources in the ACT video memory resource pool are temporarily borrowed to the KV video memory resource pool, so that even if waste occurs, the wasted video memory resources are relatively less. For reference, for example, in the case of non-first borrowing, the amount of resources borrowed each time can be 50% of the amount of resources borrowed the previous time: if the amount of resources borrowed the first time is X, the amount of resources borrowed the second time is 0.5X, the amount of resources borrowed the third time is 0.25X, and so on.

[0095] As mentioned above, if the idle video memory resources of the first video memory resource pool are temporarily transferred to the second video memory resource pool for the inference task in the Prefill phase, the transferred video memory resources can be returned from the second video memory resource pool to the first video memory resource pool after the Prefill phase is over. Since the KV Cache still needs to be accessed and updated in the Decode phase, if the idle video memory resources of the second video memory resource pool are temporarily transferred to the first video memory resource pool for the inference task, the transferred video memory resources can be returned from the first video memory resource pool to the second video memory resource pool after the Decode phase of the inference task is over.

[0096] When temporarily borrowing and returning idle video memory resources, it can be achieved by managing the page table of the corresponding video memory resource pool. For example, when temporarily borrowing at least a part of the idle video memory resources of another video memory resource pool to any video memory resource pool, the mapping relationship between at least a part of the logical video memory addresses recorded in the page table of the other video memory resource pool and the allocated physical video memory addresses can be released and remapped to the page table of any video memory resource pool.

[0097] For example, it is assumed that the first video memory resource pool has a corresponding page table T1, which records the mapping relationship between the logical video memory addresses AL1-ALm and the physical video memory addresses A1-Am, and the second video memory resource pool has a corresponding page table T2, which records the mapping relationship between the logical video memory addresses BL1-BLk and the physical video memory addresses B1-Bk. If it is necessary to temporarily transfer the idle video memory resources corresponding to the physical video memory addresses Aa-Ai from the first video memory resource pool to the second video memory resource pool, it is possible to: release the mapping relationship between the physical video memory addresses Aa-Ai and the logical video memory addresses ALa-ALi from the page table T1, and record the mapping relationship between the physical video memory addresses Aa-Ai and the logical video memory addresses BL(k+a)-BL(k+i) in the page table 2.

[0098] Similarly, if it is necessary to return the idle video memory resources borrowed by any video memory resource pool to another video memory resource pool, the mapping relationship between the corresponding logical video memory address and the physical video memory address recorded in the page table of any video memory resource pool can be released, and remapped to the page table of the other video memory resource pool. Continuing with the above example, the mapping relationship between the physical video memory address Aa-Ai and the logical video memory address BL(k+a)-BL(k+i) can be released from page table 2, and the mapping relationship between the physical video memory address Aa-Ai and the logical video memory address ALa-ALi can be recorded in page table T1.

[0099] Figure 4 FIG. 1 is a schematic diagram of an architecture of a scheduling and management system for video memory resources provided by an exemplary embodiment. Figure 4As shown, taking any worker 0 of the LLM on the service device as an example, the worker 0 can be deployed on a graphics card on the service device.

[0100] From the perspective of physical hardware, it involves the physical video memory on the graphics card where worker0 is located.

[0101] From the software logic point of view, the scheduling and management system involves a unified physical memory pool established for the above-mentioned physical video memory, and the unified physical memory pool is used to maintain the physical video memory resources in the above-mentioned physical video memory. The video memory interface layer can provide interfaces for allocating, releasing, borrowing, etc. physical video memory resources, so that the dynamic allocation and reuse management layer can schedule and manage the physical video memory resources maintained in the unified physical memory pool through the video memory interface layer. In addition, the Prefill processing module is used to perform Prefill stage processing on the inference task, and the Decode processing module is used to perform Decode stage processing on the inference task.

[0102] The dynamic allocation and reuse management layer is the core of the above scheduling management system, which implements the scheduling management of video memory resources through pre-defined strategies. For example:

[0103] 1) Dynamic mapping management strategy

[0104] As mentioned above, by maintaining the page tables corresponding to the first video memory resource pool and the second video memory resource pool, that is, remapping the mapping relationship between the logical video memory address and the physical video memory address, the physical video memory resources can be dynamically mapped to the first video memory resource pool or the second video memory resource pool while maintaining the virtual pointer (that is, the logical video memory address) unchanged. This not only improves the scheduling flexibility of the video memory resources, but the entire process does not involve the reallocation of physical video memory resources, and can reduce the consumption caused by video memory scheduling.

[0105] Of course, as a theoretically feasible technical solution, even if the physical video memory resources are released and reallocated, as long as the physical video memory resources can be seconded, the processing efficiency of the reasoning task can still be improved, and this specification does not exclude this technical solution.

[0106] 2) Cache strategy

[0107] The cache strategy describes how to cache KV Cache and ACT. Specifically, as mentioned above, independent video memory resource pools can be established for KV Cache and ACT, namely the first video memory resource pool and the second video memory resource pool, to cache KV Cache and ACT respectively.

[0108] The cache strategy may also involve how to initialize and allocate physical video memory resources to the first video memory resource pool and the second video memory resource pool, which has been described in detail above and will not be repeated here.

[0109] 3) Asynchronous strategy

[0110] As mentioned above, when KV Cache is cached in the Decode phase, since the overall usage of KV Cache for video memory resources is increasing and cannot be accurately estimated, it is very likely that the idle video memory resources in the first video memory resource pool are insufficient. In this regard, the technical solution of this specification can temporarily transfer the idle video memory resources in the second video memory resource pool to the first video memory resource pool.

[0111] Among them, the asynchronous strategy describes the process of borrowing idle video memory resources. Specifically, by setting the warning ratio as mentioned above, when the proportion of idle video memory resources in the first video memory resource pool is less than the preset ratio, the above-mentioned temporary borrowing operation can be performed in advance, so that the service device can asynchronously execute: ① Generate an output token for the reasoning task; ② Temporarily borrow the idle video memory resources from the second video memory resource pool to the first video memory resource pool. In this way, the time-consuming overhead caused by the borrowing operation on the video memory resources can be hidden in the calculation process for the output token. Then, as long as there are enough idle video memory resources in the first video memory resource pool, the output token can be continuously generated without having to suspend the generation of output tokens due to the exhaustion of idle video memory resources in the first video memory resource pool, and waiting for the service device to dispatch idle video memory resources to the first video memory resource pool through temporary borrowing or other means, which helps to speed up the processing of reasoning tasks.

[0112] 4) Pre-borrowing strategy

[0113] This pre-borrowing strategy is associated with the above-mentioned asynchronous strategy, and its related description has actually been covered in the above description of the asynchronous strategy. The core is: in the Decode stage, before the free memory resources in the first memory resource pool are exhausted, a certain amount of free memory resources are borrowed from the second memory resource pool in advance to ensure that output tokens are continuously generated for the inference task.

[0114] 5) Borrowing strategy

[0115] This borrowing strategy describes how much video memory resources are borrowed each time when temporary borrowing is performed for idle video memory resources. As mentioned above, before the Prefill stage, the video memory resource usage of KV Cache and ACT in the Prefill stage can be accurately calculated for the prompt of the reasoning task, so the amount of video memory resources that need to be borrowed can be accurately calculated based on this; and, in the Decode stage, if there is insufficient idle video memory resources in the first video memory resource pool, a resource amount can be set according to experience and borrowed accordingly each time, or the video memory resource usage of KVCache can be estimated based on business scenarios and historical reasoning task data and the amount of video memory resources that need to be borrowed can be determined accordingly, and the borrowing amount can be gradually reduced when borrowing multiple times, etc., which will not be described one by one here.

[0116] In fact, in the scenario where LLM processes inference tasks, the service device usually receives a large number of inference tasks, which are added to the task queue. The service device can obtain the length of the prompt contained in these inference tasks, and calculate the memory resource usage of the KV Cache and ACT of each inference task in the Prefill stage, and then combine the perceived amount of free memory resources in the first memory resource pool and the second memory resource pool based on the following example. Figure 2 The method shown is to temporarily transfer idle video memory resources between the first video memory resource pool and the second video memory resource pool as needed. In addition, the service device can also reasonably schedule the reasoning tasks in the task queue, such as reasonably adjusting the execution order between the reasoning tasks, and merging the reasoning tasks suitable for simultaneous processing into the same batch, etc.

[0117] As can be seen from the above embodiments, this specification creates a corresponding first video memory resource pool and a second video memory resource pool for the key-value cache and the intermediate activation value, respectively, and mutual interference between the key-value cache and the intermediate activation value can be avoided when the video memory resources are sufficient. At the same time, by uniformly managing the first video memory resource pool and the second video memory resource pool, this specification can dynamically schedule and flexibly manage idle video memory resources according to actual needs, so that the key-value cache can temporarily borrow the video memory resources originally allocated to the second video memory resource pool for the intermediate activation value, and the intermediate activation value can also temporarily borrow the video memory resources originally allocated to the first video memory resource pool for the key-value cache, which greatly improves the scheduling flexibility and utilization of video memory resources, thereby helping to shorten the reasoning time of LLM and improve the throughput performance of LLM under the condition of certain infrastructure costs.

[0118] Among them, by reasonably scheduling the video memory resources before the Prefill stage, when the video memory resources of one of the first video memory resource pool and the second video memory resource pool are insufficient, more reasoning tasks can enter the Prefill stage as soon as possible without or with reduced waiting time based on temporary loan, thereby generating and returning the first output token faster, significantly reducing the first word delay of the reasoning task. This also enables the service device to process more reasoning tasks at the same time under the premise of certain video memory resources, thereby increasing the batch capacity and achieving a relatively higher throughput.

[0119] By properly scheduling video memory resources in the Decode stage, the smooth processing of reasoning tasks can be ensured, and waiting caused by insufficient video memory resources or even task failure caused by waiting too long can be reduced or avoided. The processing time of the Decode stage is shortened, which shortens the overall processing time of the reasoning task and speeds up the processing of the reasoning task. In particular, through real-time monitoring and pre-adjustment of idle video memory resources, it is possible to completely avoid the reasoning task from suspending the generation of output tokens due to insufficient video memory resources in the Decode stage, and to speed up the processing of the reasoning task and return the generated output as much as possible. In this way, the video memory resources occupied by KV Cache and ACT in the Decode stage can also be released as soon as possible and reused for other reasoning tasks.

[0120] In particular, the technical solution of the present specification can make full use of the characteristic that the amount of video memory resources occupied by ACT in the Prefill stage and the Decode stage varies greatly: after a certain reasoning task completes the Prefill stage, since its ACT only needs to store an output token in the video memory in the Decode stage, most of the video memory resources occupied by its generated ACT in the Prefill stage can be temporarily transferred to the KV video memory resource pool (the first video memory resource pool) for caching KVCache, thereby greatly improving the utilization rate of this part of the video memory resources, and helping to achieve the technical effects of accelerating the processing speed of reasoning tasks and improving the overall throughput of reasoning tasks as mentioned above.

[0121] Figure 5 is a schematic structural diagram of a device provided by an exemplary embodiment. Figure 5At the hardware level, the device includes a processor 502, an internal bus 504, a network interface 506, a memory 508, and a non-volatile memory 510, and may also include hardware required for other functions. One or more embodiments of this specification may be implemented based on software, such as the processor 502 reading the corresponding computer program from the non-volatile memory 510 into the memory 508 and then running it. Of course, in addition to the software implementation, one or more embodiments of this specification do not exclude other implementations, such as logic devices or a combination of software and hardware, etc., that is, the execution subject of the following processing flow is not limited to each logic unit, but can also be hardware or logic devices.

[0122] Please refer to Figure 6 , the memory management device for large language model reasoning can be applied to Figure 5 The device shown in the figure is used to implement the technical solution of this specification. The device is applied to a service device deployed with a large language model, and the device may include:

[0123] The resource allocation unit 601 allocates the physical video memory resources on the service device to be mapped to a first video memory resource pool whose cache objects are key-value caches and a second video memory resource pool whose cache objects are intermediate activation values, respectively;

[0124] The resource borrowing unit 602, for the reasoning task submitted to the large language model, if it is determined that the free memory resources of any one of the first memory resource pool and the second memory resource pool are insufficient to cache the corresponding cache object for the reasoning task, then at least a part of the free memory resources of the other memory resource pool is temporarily borrowed to any one of the memory resource pools.

[0125] Optionally, the determining that the free video memory resources of any one of the first video memory resource pool and the second video memory resource pool are insufficient to cache the corresponding cache object for the inference task includes:

[0126] After receiving the inference task, respectively calculating the video memory resource requirements of the key value cache and the intermediate activation value of the inference task in the pre-filling stage;

[0127] If the free memory resources of any of the first memory resource pool and the second memory resource pool are less than the calculated memory resource demand corresponding to the corresponding cache object, it is determined that the free memory resources of any of the memory resource pools are insufficient to cache the corresponding cache object for the inference task.

[0128] Optionally, also include:

[0129] The task processing unit advances the inference task to the pre-filling stage if the free amounts of video memory resources in the first video memory resource pool and the second video memory resource pool are not less than the calculated required amounts of video memory resources corresponding to the corresponding cache objects.

[0130] Optionally, the determining that the free video memory resources of any one of the first video memory resource pool and the second video memory resource pool are insufficient to cache the corresponding cache object for the inference task includes:

[0131] When the inference task is in the decoding stage, if the video memory resources of the first video memory resource pool have been exhausted or the proportion of free video memory resources is less than the warning ratio, it is determined that the free video memory resources of the first video memory resource pool are insufficient to cache the corresponding cache object for the inference task.

[0132] Optionally, the resource secondment unit 602 is specifically configured to:

[0133] According to the business scenario to which the inference task belongs, determining the memory resource usage of the key-value cache based on the data statistics of the historical inference tasks in the business scenario;

[0134] The difference between the idle amount of video memory resources in the first video memory resource pool and the calculated occupied amount of video memory resources is calculated, and the idle video memory resources in the second video memory resource pool corresponding to the difference are temporarily transferred to the first video memory resource pool.

[0135] Optional,

[0136] The temporarily borrowing the idle video memory resources corresponding to the difference in the second video memory resource pool to the first video memory resource pool comprises: if it is the first borrowing of the inference task in the decoding stage, temporarily borrowing the idle video memory resources corresponding to the difference in the second video memory resource pool to the first video memory resource pool;

[0137] The resource borrowing unit 602 is further specifically configured to: if it is not the first borrowing of the inference task in the decoding stage, when temporarily borrowing the idle video memory resources in the second video memory resource pool to the first video memory resource pool, gradually reduce the borrowing amount of the idle video memory resources.

[0138] Optionally, also include:

[0139] A resource returning unit, wherein if the idle video memory resources of the second video memory resource pool are temporarily loaned to the first video memory resource pool for the reasoning task, then after the decoding phase of the reasoning task is completed, the loaned video memory resources are returned from the first video memory resource pool to the second video memory resource pool; if the idle video memory resources of the first video memory resource pool are temporarily loaned to the second video memory resource pool for the reasoning task before the pre-filling phase, then after the pre-filling phase is completed, the loaned video memory resources are returned from the second video memory resource pool to the first video memory resource pool.

[0140] Optionally, the resource allocation unit 601 is specifically configured to:

[0141] Determining the maximum prompt word length set by the large language model;

[0142] Taking the maximum prompt word length as the assumed prompt word length, physical video memory resources are allocated to the second video memory resource pool, and the remaining physical video memory resources are allocated to the first video memory resource pool; or, taking the actual length less than the maximum prompt word length as the assumed prompt word length, physical video memory resources are allocated to the second video memory resource pool, and the remaining physical video memory resources are allocated to the first video memory resource pool.

[0143] Optionally, the actual length is obtained by statistics of data of historical reasoning tasks, and the length of prompt words of the historical reasoning tasks that is not less than a preset ratio is not greater than the actual length.

[0144] Optionally, the first video memory resource pool and the second video memory resource pool respectively record logical video memory addresses, and each logical video memory address is mapped to an allocated physical video memory address; the resource borrowing unit 602 is specifically used to:

[0145] The mapping relationship between at least a part of the logical video memory addresses recorded in the page table of the other video memory resource pool and the allocated physical video memory addresses is released, and the addresses are remapped to the page table of any video memory resource pool.

[0146] Based on the same concept as the above method, this specification also provides an electronic device, including: a processor; a memory for storing processor executable instructions; wherein the processor implements the steps of the method described in any of the above embodiments by running the executable instructions.

[0147] Based on the same concept as the above method, this specification also provides a computer-readable storage medium on which computer instructions are stored. When the instructions are executed by a processor, the steps of the method described in any of the above embodiments are implemented.

[0148] Based on the same concept as the above method, this specification also provides a computer program product, including a computer program / instruction, which implements the steps of the method described in any of the above embodiments when executed by a processor.

Claims

1. A method for managing video memory for large language model inference, applied to a service device deployed with a large language model; the method comprises: Allocating physical video memory resources on the service device to be mapped to a first video memory resource pool whose cache objects are key-value caches and a second video memory resource pool whose cache objects are intermediate activation values, respectively; For the reasoning task submitted to the large language model, if it is determined that the free memory resources of any one of the first memory resource pool and the second memory resource pool are insufficient to cache the corresponding cache object for the reasoning task, at least a portion of the free memory resources of the other memory resource pool are temporarily loaned to any one of the memory resource pools.

2. The method according to claim 1, wherein determining that the free video memory resources of any of the first video memory resource pool and the second video memory resource pool are insufficient to cache the corresponding cache object for the inference task comprises: After receiving the inference task, respectively calculating the video memory resource requirements of the key value cache and the intermediate activation value of the inference task in the pre-filling stage; If the free memory resources of any of the first memory resource pool and the second memory resource pool are less than the calculated memory resource demand corresponding to the corresponding cache object, it is determined that the free memory resources of any of the memory resource pools are insufficient to cache the corresponding cache object for the inference task.

3. The method according to claim 2, further comprising: If the free amounts of video memory resources in the first video memory resource pool and the second video memory resource pool are not less than the calculated required amounts of video memory resources corresponding to the corresponding cache objects, the inference task is advanced to the pre-filling stage.

4. The method according to claim 1, wherein determining that the free video memory resources of any of the first video memory resource pool and the second video memory resource pool are insufficient to cache the corresponding cache object for the inference task comprises: When the inference task is in the decoding stage, if the video memory resources of the first video memory resource pool have been exhausted or the proportion of free video memory resources is less than the warning ratio, it is determined that the free video memory resources of the first video memory resource pool are insufficient to cache the corresponding cache object for the inference task.

5. The method according to claim 4, wherein temporarily transferring at least a portion of idle video memory resources of another video memory resource pool to any of the video memory resource pools comprises: According to the business scenario to which the inference task belongs, determining the memory resource usage of the key-value cache based on the data statistics of the historical inference tasks in the business scenario; The difference between the idle amount of video memory resources in the first video memory resource pool and the calculated occupied amount of video memory resources is calculated, and the idle video memory resources in the second video memory resource pool corresponding to the difference are temporarily transferred to the first video memory resource pool.

6. The method according to claim 5, The temporarily transferring the idle video memory resources corresponding to the difference in the second video memory resource pool to the first video memory resource pool includes: If it is the first time that the inference task is borrowed in the decoding stage, the idle video memory resources corresponding to the difference in the second video memory resource pool are temporarily borrowed to the first video memory resource pool; The temporarily loaning at least a portion of the idle video memory resources of another video memory resource pool to any of the video memory resource pools also includes: if it is not the first borrowing of the inference task in the decoding stage, when the idle video memory resources in the second video memory resource pool are temporarily loaned to the first video memory resource pool, the borrowing amount of the idle video memory resources is gradually reduced.

7. The method according to claim 1, further comprising: If the idle video memory resources of the second video memory resource pool are temporarily loaned to the first video memory resource pool for the inference task, then after the decoding phase of the inference task is completed, the loaned video memory resources are returned from the first video memory resource pool to the second video memory resource pool; If the idle video memory resources of the first video memory resource pool are temporarily loaned to the second video memory resource pool for the inference task before the pre-filling stage, the loaned video memory resources are returned from the second video memory resource pool to the first video memory resource pool after the pre-filling stage ends.

8. The method according to claim 1, wherein allocating physical video memory resources on the service device comprises: Determining the maximum prompt word length set by the large language model; Taking the maximum prompt word length as the assumed prompt word length, physical video memory resources are allocated to the second video memory resource pool, and the remaining physical video memory resources are allocated to the first video memory resource pool; or, taking the actual length less than the maximum prompt word length as the assumed prompt word length, physical video memory resources are allocated to the second video memory resource pool, and the remaining physical video memory resources are allocated to the first video memory resource pool.

9. The method according to claim 8, wherein: The actual length is obtained by statistics of data of the historical reasoning task, and the length of the prompt word of the historical reasoning task which is not less than a preset ratio is not greater than the actual length.

10. The method according to claim 1, wherein the first video memory resource pool and the second video memory resource pool respectively record logical video memory addresses, and each logical video memory address is mapped to an allocated physical video memory address; the temporarily transferring at least a portion of idle video memory resources of another video memory resource pool to any of the video memory resource pools comprises: The mapping relationship between at least a part of the logical video memory addresses recorded in the page table of the other video memory resource pool and the allocated physical video memory addresses is released, and the addresses are remapped to the page table of any video memory resource pool.

11. An electronic device, comprising: processor; A memory for storing processor-executable instructions; wherein the processor implements the steps of the method according to any one of claims 1 to 10 by executing the executable instructions.

12. A computer-readable storage medium having computer instructions stored thereon, which, when executed by a processor, implement the steps of the method according to any one of claims 1 to 10.

13. A computer program product, comprising a computer program / instruction, which, when executed by a processor, implements the steps of the method according to any one of claims 1 to 10.

Images