A large model inference cache management method and device

Abstract

A method for managing large model inference caches includes: a scheduler determining a candidate inference request queue; forming a batch processing queue based on the candidate inference request queue, the batch processing queue being used for the current round of large model computation; allocating a first cache space and a second cache space based on the storage space occupancy on high-bandwidth memory; determining a decision queue; the decision queue being used to acquire a portion of the first cache space and prefetch the second cache space when forming the batch processing queue; the decision queue including synchronous exchange instructions and asynchronous exchange instructions; sending the decision queue to a first executor; sending the formed batch processing queue to a second executor; the first executor executing the synchronous exchange instruction before the start of the current round of large model computation, swapping requests from the candidate inference request queue into the allocated first cache space, so that the scheduler obtains the formed batch processing queue; and executing the asynchronous exchange instruction during the current round of large model computation, swapping requests from the candidate inference request queue into the second cache space.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence, and in particular to a method and apparatus for managing large model inference caches. Background Technology

[0002] Large model inference is divided into two phases: the phase that generates the first output token is called the Prefill phase, and the phase that generates all subsequent output tokens until inference is complete is called the Decoding phase. In the Decoding phase, many iterations are performed, with each iteration generating one output token, until inference ends.

[0003] Key-value (KV) caching is a technique used in AI large-scale model inference to cache intermediate results of model inference, enabling fast access in subsequent inference processes. The cache used to store intermediate results is called a key-value cache.

[0004] In large-scale model inference systems, scheduling primarily occurs on the host side. The CPU groups multiple requests into a batch and sends it to the accelerator for execution. A typical batching strategy operates on a sequence-by-sequence basis, selecting a specified number of requests per iteration to form a batch, which is then sent to the accelerator for inference computation. Computation mainly occurs on the device side, and the key-value cache used is stored in the accelerator's high-bandwidth memory (HBM). The next round of batching only begins after all requests in the batch have completed their inference computation. A significant drawback of this approach is that the next round of batching can only proceed after all requests in the batch have been inferred and computed by the accelerator. Since the lengths of individual requests vary, considerable time and resources are wasted waiting for the longer requests in the batch to complete their computation, impacting the throughput of the large-scale model system. Summary of the Invention

[0005] The large model inference cache management method provided in this application overlaps the data exchange time during the large model inference process with the inference computation time of the computing acceleration processor by scheduling, thereby reducing the latency consumption of data exchange and improving end-to-end throughput.

[0006] In a first aspect, embodiments of this application provide a large model inference cache management method, including a scheduler, a first executor, and a second executor. The method includes: in the scheduler: determining a candidate inference request queue; forming a batch processing queue according to the candidate inference request queue, the batch processing queue being used for the current round of large model computation; allocating a first cache space and a second cache space according to the storage space occupancy on the High Bandwidth Memory (HBM), the first cache space being the cache space required for the current round of large model computation, and the second cache space being used to provide the cache space required for future large model computation; determining a decision queue; the decision queue being used to acquire a portion of the first cache space and prefetch the second cache space when forming the batch processing queue; the decision queue including synchronous exchange instructions and asynchronous exchange instructions; and sending the decision queue... The batch processing queue is sent to the first executor; the assembled batch processing queue is sent to the second executor; in the first executor: before the start of the current round of large model computation, the synchronous exchange instruction is executed to swap the requests in the candidate inference request queue into the allocated first cache space, so that the scheduler obtains the assembled batch processing queue; during the current round of large model computation, the asynchronous exchange instruction is executed to swap the requests in the candidate inference request queue into the allocated second cache space; while the first executor executes the asynchronous exchange instruction, the scheduler calls the kernel function of the large model computation; in the second executor: in response to the kernel function of the large model inference called by the scheduler, the current round of large model computation is executed according to the assembled batch processing queue, and the result of the current round of computation is stored in the first cache space.

[0007] Therefore, the method provided in this application manages the batch processing queue through a scheduler and reasonably allocates the space usage of HBM. It adopts a reasonable decision queue scheduling to schedule the first executor to perform data exchange of KV cache and the large model inference calculation of the second executor. This overlaps the execution time of data exchange with the time of large model inference calculation, reducing the latency consumption of data exchange and thus improving end-to-end throughput.

[0008] In some possible implementations, determining the candidate inference request queue includes: placing multiple pending inference requests into the candidate inference request queue in priority order; traversing the candidate inference request queue and determining the status of each inference request in the candidate inference request queue, the status including "waiting for computation", "being swapped out", "swapped out", "being swapped in", and "swapped in"; the status is used to indicate the current processing status of the inference request.

[0009] Therefore, in the method provided in the embodiments of this application, the scheduler can monitor different state changes of inference requests, and perform scheduling management by forming batch processing queues and decision queues according to the different states of inference requests at different times.

[0010] In some possible implementations, a batch processing queue is formed based on the candidate inference request queue, including: calculating the maximum allowed batch size based on the storage space on the high-bandwidth memory and the large model; determining the cache size of the batch processing queue, wherein the cache size of the batch processing queue is less than or equal to the maximum batch size; removing inference requests whose cache space meets the cache size requirement from the candidate inference request queue and adding them to the batch processing queue in a first-in-first-out order, wherein the cache space occupied by the inference requests that meet the cache size requirement should be less than or equal to the cache size of the batch processing queue.

[0011] Therefore, in the method provided in this application embodiment, the scheduler can determine the size of the batch queue based on the storage space on the high-bandwidth memory, and determine the inference requests to be grouped into the batch queue based on the size of the batch queue.

[0012] In some possible implementations, the batch processing queue is formed based on the candidate inference request queue, and further includes: adding inference requests that meet the status requirements in the candidate inference request queue to the batch processing queue; the inference requests that meet the status requirements include inference requests with a status of "waiting for computation" or "already swapped in".

[0013] Therefore, in the method provided in this application embodiment, the scheduler can batch process queues according to the status of inference request volume, avoiding wasting time and resources due to waiting for long requests to be swapped in.

[0014] In some possible implementations, allocating a first cache space and a second cache space based on the storage space occupancy on the high-bandwidth memory (HBM) includes: determining whether the space occupancy of the high-bandwidth memory has reached a saturation threshold; if the space occupancy of the high-bandwidth memory has not reached the saturation threshold, allocating a first cache space on the high-bandwidth memory, the size of which matches the cache space required for the current large model computation and the space occupied by inference requests in the corresponding batch processing queue.

[0015] In this way, the scheduler can reasonably allocate the cache space required for the current round of large model computation based on the space usage of HBM, so as to avoid full load.

[0016] In some possible implementations, allocating the first cache space and the second cache space based on the storage space occupancy on the high-bandwidth memory (HBM) further includes: when the space occupancy of the high-bandwidth memory reaches a saturation threshold, allocating destination space for the preemptible inference requests in the candidate inference request queue; the preemptible inference requests include the lowest priority inference requests in the candidate inference request queue; the destination space is CPU memory space.

[0017] Therefore, when the space usage of HBM is saturated, the scheduler will move some content to the host side and move it back when the space on the device side is not tight, so as to improve throughput.

[0018] In some possible implementations, future large-scale model computation includes the next round of large-scale model computation, allocating a first cache space and a second cache space based on the storage space occupancy on the high-bandwidth memory (HBM), including: determining whether the space occupancy of the high-bandwidth memory has reached a saturation threshold based on the first cache space; if the space occupancy of the high-bandwidth memory has not reached the saturation threshold, allocating a second cache space on the high-bandwidth memory, the second cache space providing swappable cache space for inference requests from batch processing queues required for future large-scale model computation.

[0019] In this way, the scheduler can reserve cache space for inference requests of batch processing queues required for future large model computations based on the space usage of HBM, so that cache space for data exchange and large model inference computation exists on HBM at the same time, reducing the blocking and waiting time caused by space occupation.

[0020] In some possible implementations, the queue of pending inference requests includes inference requests with a status of "swapped out". Allocating a second cache space on high-bandwidth memory includes: allocating a second cache space in the second cache space for "swapped out" inference requests, where "swapped out" inference requests are inference requests that have been swapped out to CPU memory.

[0021] In this way, the scheduler can reserve the buffer space required for swapping in the batch queue for future large model calculations based on the status of inference requests, avoiding the waste of time caused by waiting for long output requests to complete.

[0022] In some possible implementations, the asynchronous swap instruction includes an asynchronous swap-in instruction, the synchronous swap instruction includes a synchronous swap-in instruction, a decision queue is determined, the decision queue is used to acquire a portion of the first cache space when forming a batch processing queue, and prefetch the second cache space, including: when the space occupancy of the high-bandwidth memory has not reached the saturation threshold, the decision queue includes asynchronous swap-in instructions and synchronous swap-in instructions; execution requires waiting for the instructions to complete; the number of asynchronous swap-in instructions is one or more; the decision queue is used to acquire part or all of the first cache space when forming a batch processing queue through synchronous swap-in instructions, and prefetch the second cache space through asynchronous swap-in instructions.

[0023] This application utilizes an asynchronous swap-in decision mechanism in non-essential scenarios, pre-fetching the data beforehand so that inference computation does not require waiting for the data transfer to complete. In later iterations, if the same asynchronous swap-in request is encountered again, a synchronous swap-in decision is issued, waiting for the previous asynchronous swap-in operation to complete before inference computation. Since the asynchronous swap-in operation has already begun before the current iteration, the waiting time during this blocking phase is significantly reduced.

[0024] In some possible implementations, the decision queue further includes a first temporary queue for storing synchronous swap-in instructions. Determining the decision queue includes: adding synchronous swap-in instructions from the first temporary queue to the decision queue before the current round of large model computation begins; adding asynchronous swap-in instructions to the decision queue during the current round of large model computation, wherein the number of asynchronous swap-in instructions is one or more; acquiring a portion of the first cache space in the batch processing queue before the current round of large model computation begins using synchronous swap-in instructions; and prefetching a second cache space during the current round of large model computation using one or more asynchronous swap-in instructions.

[0025] This provides a group decision queue approach. When a sequence that needs to be swapped in is encountered, an asynchronous swap-in instruction is issued first, and a synchronous swap-in instruction is issued in the next round to actually group it into the batch, thereby realizing the pre-fetching of swap-in instructions and shortening the waiting time.

[0026] In some possible implementations, the asynchronous swapping instruction includes an asynchronous swap-out instruction, the synchronous swapping instruction includes a synchronous swap-out instruction, a decision queue is determined, the decision queue is used to acquire a portion of the first cache space when batch processing the queue, and a second cache space is prefetched, including: when the space occupancy of the high-bandwidth memory reaches a saturation threshold, determining the decision queue includes asynchronous swap-out instructions and synchronous swap-out instructions; the number of asynchronous swap-out instructions is one or more; before the current round of large model computation begins, the decision queue swaps out inference requests in the candidate inference request queue with the status of "being swapped out" from the high-bandwidth memory to the destination space through synchronous swap-out instructions; during the current round of large model computation, inference requests that are to be preempted in the candidate inference request queue are pre-swapped out from the high-bandwidth memory through one or more asynchronous swap-out instructions; and the space occupancy of the high-bandwidth memory is released to restore the state that has not reached the saturation threshold.

[0027] This application utilizes an asynchronous swap-out decision mechanism in non-essential scenarios, pre-fetching data so that inference calculations do not require waiting for the data transfer to complete. In later iterations, if a request that has already undergone an asynchronous swap-out is encountered again, a synchronous swap-out decision is then issued to release cache space. Since the asynchronous swap-out operation has already begun before the current iteration, the waiting time during this blocking iteration is significantly reduced.

[0028] In some possible implementations, the decision queue further includes a second temporary queue for storing synchronous swap-out instructions. Determining the decision queue includes: adding synchronous swap-out instructions from the second temporary queue to the decision queue before the current round of large model computation begins; adding one or more asynchronous swap-out instructions to the decision queue during the current round of large model computation; releasing the cache space required for the current round of large model computation by using synchronous swap-out instructions when batch processing the queue before the current round of large model computation begins; and pre-releasing the cache space required for the next round of large model computation by using one or more asynchronous swap-out instructions during the current round of large model computation.

[0029] This provides a group decision queue approach. When a sequence that needs to be swapped out is encountered, an asynchronous swapping instruction is issued first, and a synchronous swapping instruction is issued in the next round to actually group it into the batch, thereby realizing the pre-fetching of swapping instructions and shortening the waiting time.

[0030] In some possible implementations, sending the decision queue to the first executor and sending the assembled batch processing queue to the second executor includes: issuing a synchronous swap-in instruction to the first executor before the current round of large model calculation begins; sending the assembled batch processing queue to the second executor after the swap-in is completed; and sending one or more asynchronous swap-in instructions to the first executor during the current round of large model calculation.

[0031] This allows for the use of reasonable swap-in decisions to schedule the first executor to perform data exchange in the KV cache, while controlling the large model inference calculation of the second executor, even when the space occupancy of the high-bandwidth memory has not reached the saturation threshold. This overlaps the execution time of data exchange with the time of large model inference calculation during the large model inference process.

[0032] In some possible implementations, the decision queue is sent to the first executor and the assembled batch processing queue is sent to the second executor, including: issuing a synchronous swap instruction to the first executor before the current round of large model calculation begins; and issuing one or more asynchronous swap instructions to the first executor during the current round of large model calculation.

[0033] This allows for the use of reasonable swapping decisions to schedule the first executor to perform data exchange in the KV cache, while controlling the large model inference calculation of the second executor, even when the space occupancy of the high-bandwidth memory has not reached the saturation threshold. This overlaps the execution time of data exchange with the time of large model inference calculation during the large model inference process.

[0034] In some possible implementations, a synchronous swap instruction is executed before the current round of large model computation begins, swapping requests from the candidate inference request queue into the allocated first cache space, so that the scheduler obtains the assembled batch processing queue. This includes: the first executor executing a synchronous swap-in instruction to swap inference requests with a status of "swapping in" into the allocated first cache space, and updating the status to "swapped in" after execution; and executing an asynchronous swap-in instruction to place inference requests with a status of "swapped out" into the allocated second cache space, and updating the status of inference requests with a status of "swapped out" to "swapping in".

[0035] In this way, the first executor can perform the swap-in operation based on the status of the inference request and update the status of the inference request, providing timely status feedback to the scheduler and avoiding the waste of time and resources caused by waiting for long swap-in requests to complete.

[0036] In some possible implementations, the method further includes: the scheduler determining a grouped batch queue based on inference requests with a status of "swapped in" or "waiting to compute".

[0037] In this way, the first executor can perform the swap-in operation based on the status of the inference request and update the status of the inference request, providing timely status feedback to the scheduler and avoiding the waste of time and resources caused by waiting for long request batches to complete.

[0038] In some possible implementations, a synchronous swap instruction is executed before the current round of large model computation begins, swapping requests in the candidate inference request queue into the allocated first cache space, so that the scheduler obtains the grouped batch processing queue. This includes: before the current round of large model computation begins, the first executor executes a synchronous swap-out instruction, swapping out all inference requests with the status "swapping out" from the high-bandwidth memory to the allocated destination space, and updating the status to "swapped out" after execution; during the current round of large model computation, an asynchronous swap-out instruction is executed, swapping out the preempted inference requests from the high-bandwidth memory, and updating the status of the preempted inference requests to "swapping out".

[0039] In this way, the first executor can perform the swapping operation based on the status of the inference request and update the status of the inference request, providing timely status feedback to the scheduler and avoiding the waste of time and resources caused by waiting for long request swapping to complete.

[0040] In some possible implementations, the method further includes: after the current round of large model computation is completed, the scheduler iterates to determine the candidate inference request queues required for the next round of large model computation; based on the candidate inference request queues required for the next round of large model computation, a batch processing queue required for the next round of large model computation is formed; when forming the batch processing queues required for the next round of large model computation, a second cache space is used; the decision queue also includes a synchronous wait instruction; the synchronous wait instruction is used to wait for the return information of the current round of large model computation.

[0041] Secondly, embodiments of this application provide a system including a scheduling module, a first execution module, and a second execution module. The scheduling module determines a candidate inference request queue; assembles a batch processing queue based on the candidate inference request queue, the batch processing queue being used for the current round of large model computation; allocates a first cache space and a second cache space based on the storage space occupancy on the High Bandwidth Memory (HBM), the first cache space being the cache space required for the current round of large model computation, and the second cache space being used to provide the cache space required for future large model computation; determines a decision queue; the decision queue is used to acquire a portion of the first cache space and prefetch the second cache space when assembling the batch processing queue; the decision queue includes synchronous exchange instructions and asynchronous exchange instructions; sends the decision queue to the first execution module; and sends the assembled batch processing queue to the second execution module. The first execution module executes asynchronous exchange instructions while the scheduler module calls the kernel function for large model computation. The first execution module 2 receives batch processing queues and decision queues. Based on the decision queues, it executes asynchronous and synchronous exchange instructions round by round, swapping the batch processing queues into the allocated cache space. After swapping, a second batch processing queue is obtained, which is the request queue required for the current round of large model computation. While executing asynchronous exchange instructions, it calls the kernel function for large model inference. The second execution module responds to the kernel function for large model inference called by the scheduler, executes the current round of large model computation based on the assembled batch processing queues, and stores the computation results of the current round in the first cache space.

[0042] Thirdly, embodiments of this application provide a computing device, including: at least one memory for storing a program; and at least one processor for executing the program stored in the memory, wherein when the program stored in the memory is executed, the processor is configured to execute the method as described in any one of the first aspects.

[0043] Fourthly, embodiments of this application provide a computing device cluster, including at least one computing device, each computing device including a processor and a memory; the processor of the at least one computing device is configured to execute a request stored in the memory of the at least one computing device, such that the computing device cluster performs the method as described in any one of the first aspects.

[0044] Fifthly, embodiments of this application provide a computer storage medium storing a request that, when executed on a computer, causes the computer to perform the method provided in any of the first aspects.

[0045] In a sixth aspect, embodiments of this application provide a computer program product that, when run on a processor, causes the processor to perform the method described in any one of the first aspects. Attached Figure Description

[0046] To more clearly illustrate the technical solutions of the various embodiments disclosed in this specification, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only a few embodiments disclosed in this specification. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0047] The accompanying drawings used in the description of the embodiments or prior art are briefly introduced below.

[0048] Figure 1 This is a hardware system architecture diagram for large model inference cache management proposed in an embodiment of this application;

[0049] Figure 2 This is a schematic diagram of the logical framework of the large model inference cache management method proposed in the embodiments of this application;

[0050] Figure 3 A scheduling flowchart for a large model inference cache management method provided in an embodiment of this application;

[0051] Figure 4a This is a batch processing method provided in the embodiments of this application;

[0052] Figure 4b This is a batch processing method provided in the embodiments of this application;

[0053] Figure 4c This is a batch processing method provided in the embodiments of this application;

[0054] Figure 5 The execution flowchart of the first executor in the large model inference cache management method provided in the embodiments of this application;

[0055] Figure 6a A schematic diagram of the scheduling process for the large model inference cache management method provided in Embodiment 1 of this application;

[0056] Figure 6b A schematic diagram of the execution flow of the large model inference cache management method provided in Embodiment 1 of this application;

[0057] Figure 7 This is a schematic diagram illustrating an application scenario of the large model inference cache management method provided in Embodiment 2 of this application;

[0058] Figure 8 This is a flowchart illustrating the preemptive scenario provided in Embodiment 3 of this application;

[0059] Figure 9 This application provides a large model inference cache management device.

[0060] Figure 10 A computing device provided in an embodiment of this application;

[0061] Figure 11 This application provides a computing device cluster as an embodiment;

[0062] Figure 12 One possible implementation method is shown. Detailed Implementation

[0063] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions in the embodiments of this application will be described below with reference to the accompanying drawings.

[0064] In the description of the embodiments of this application, the words "exemplary," "for example," or "for instance" are used to indicate examples, illustrations, or explanations. Any embodiment or design described as "exemplary," "for example," or "for instance" in the embodiments of this application should not be construed as being more preferred or advantageous than other embodiments or designs. Specifically, the use of the words "exemplary," "for example," or "for instance" is intended to present the relevant concepts in a specific manner.

[0065] In the description of the embodiments in this application, the term "and / or" is merely a description of the association relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, B existing alone, and A and B existing simultaneously. Furthermore, unless otherwise stated, the term "multiple" means two or more. For example, multiple systems refer to two or more systems, and multiple terminals refer to two or more terminals.

[0066] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the indicated technical features. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature. The terms "comprising," "including," "having," and their variations all mean "including but not limited to," unless otherwise specifically emphasized.

[0067] In the description of the embodiments in this application, "some embodiments" are mentioned, which describe a subset of all possible embodiments. However, it is understood that "some embodiments" can be the same subset or different subsets of all possible embodiments, and can be combined with each other without conflict.

[0068] In the description, the terms “first, second, third, etc.” or module A, module B, module C, etc. are used only to distinguish similar objects and do not represent a specific ordering of objects. It is understood that a specific order or sequence may be interchanged where permitted so that the embodiments of this application described herein can be implemented in an order other than that illustrated or described herein.

[0069] The labels used to indicate steps in the description, such as S110, S120, etc., do not necessarily mean that the steps will be executed in this way. The order of the steps can be interchanged or executed simultaneously if permitted.

[0070] Technical and scientific terms related to this application:

[0071] Host-side memory refers to the memory stored on the host CPU.

[0072] Device memory is the high-bandwidth memory (HBM) on a computing accelerator processor.

[0073] The swap out instruction is used to swap some of the contents of the key-value cache to the host memory when the compute accelerator processor's memory space is insufficient.

[0074] The swap-in instruction is used when there is sufficient memory space in the accelerator processor to swap the requests to be processed into the accelerator processor's memory.

[0075] Synchronization means that the actions of both parties are coordinated and synchronized through waiting.

[0076] In asynchronous operations, neither party needs to coordinate; both can freely perform their respective operations.

[0077] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing embodiments of this application only and is not intended to limit this application.

[0078] The following sections introduce the solutions related to this application from different aspects.

[0079] One approach related to this application involves the scheduler issuing a synchronous swap instruction during large model inference, while batch processing is being performed. The executor executes the synchronous swap instruction, preparing the data required for the current batch in the memory of the accelerator processor before performing the inference computation for that batch. However, this approach suffers from limitations in bandwidth and speed. The time spent by the accelerator processor waiting for the synchronous swap to complete is significantly longer than the latency of the inference computation itself, delaying the processing time of batch requests and consequently negatively impacting the throughput performance of the inference server, thus reducing system throughput.

[0080] In view of this, embodiments of this application provide a large model inference cache management method that can overlap the data exchange time of the KV cache with the inference computation time of the computing acceleration processor during the large model inference process, reduce the latency consumption of KV cache data exchange, and thus improve end-to-end throughput.

[0081] The large model inference cache management method proposed in this application is applied to large model inference systems, such as the Ascend Inference Engine MindIE.

[0082] For example, Figure 1 This is a hardware system architecture diagram for large model inference cache management proposed in an embodiment of this application. Figure 1 As shown, the application system architecture includes a host and a device. The host includes a central processing unit (CPU) 10 and CPU memory 120, and the device includes a computing accelerator processor 11 and high bandwidth memory (HBM) 110.

[0083] The central processing unit (CPU) 10 executes the method proposed in the embodiments of this application to manage the scheduling and execution of large model inference cache.

[0084] The computing accelerator 11 includes one or more accelerator processing units, which may be neural processing units (NPUs) or graphics processing units (GPUs), to perform large model inference computations.

[0085] In a round of large model computation, if the HBM110 space on the device side can support the simultaneous execution of n requests for large model computation, this application executes nk requests for large model computation. The remaining HBM space, which could originally support k requests for large model computation, is now reserved as partial cache space for future large model computation. The cache space required for the current round of large model computation on the HBM110 is the first cache space, and the partial cache space reserved on the HBM110 for future large model computation is the second cache space.

[0086] To improve the throughput of large model systems, it is necessary to increase the parallelism of inference (i.e., batch size). However, the key-value cache increases linearly with the parallelism, and the space of the HBM110 is limited. Therefore, to improve throughput, it is necessary to allocate space in the host-side memory for storing the key-value cache. During inference, when the space in the device-side HBM110 is insufficient, part of the key-value cache content will be swapped to the host side, and swapped back when the space in the device-side HBM110 is not insufficient. The space on the CPU120 memory for storing the key-value cache is the third cache space.

[0087] The KV cache stored on HBM110 that provides the data required for the current round of large model calculation can be referred to as the first cache; the KV cache stored on HBM110 that provides the data required for the next round of large model calculation can be referred to as the second cache; and the KV cache stored on CPU120 memory that is waiting to be swapped can be referred to as the third cache.

[0088] Figure 2 This is a schematic diagram illustrating the logical framework of the large model inference cache management method proposed in this application. Figure 2 As shown, the logical framework for large model inference cache management includes a scheduler 21, a first executor 22, and a second executor 23.

[0089] First, scheduler 21 determines candidate inference request queue Q1; based on candidate inference request queue Q1, batch processing queue Q2 is formed, which is used for the current round of large model computation; based on the storage space occupancy on high-bandwidth memory HBM110, a first cache space and a second cache space are allocated; the first cache space is used to provide the cache space required for the current round of large model computation, and the second cache space is used to provide the cache space required for future large model computation; a decision queue Q3 is determined; the decision queue Q3 is used to acquire part of the first cache space and prefetch the second cache space when forming the batch processing queue; the decision queue includes synchronous exchange instructions and asynchronous exchange instructions; the decision queue is sent to the first executor 22.

[0090] Before the current round of large model calculation begins, the first executor 22 executes a synchronization exchange instruction to swap the inference requests in the candidate inference request queue into the allocated first cache space.

[0091] Scheduler 21 waits for the synchronization exchange to complete, obtains the assembled batch processing queue, and sends the assembled batch processing queue to the second executor 23 to call the kernel function for large model calculation.

[0092] The second executor 23 responds to the kernel function of the large model computation called by the scheduler, executes the current round of large model computation according to the assembled batch processing queue, and temporarily stores the current round of computation results in the first cache space.

[0093] The first executor 22 executes an asynchronous swap instruction during the current round of large model computation, swapping the inference requests in the candidate inference request queue into the allocated second cache space for prefetching the next round of batch processing queue.

[0094] After the current round of large model calculations is completed, scheduler 21 will process the next round of iterative batch processing queues.

[0095] Among them, the pending inference requests are inference requests waiting to execute the next round of large model calculations. The pending inference requests have completed pre-filled calculations, hold the first cache, and occupy part of the HBM space.

[0096] At the beginning of each iteration, scheduler 21 selects the token generated in the previous iteration as the inference request to be processed and performs continuous batch processing (CB) to obtain a batch processing queue. During each iteration of continuous batch processing, inference requests that have completed the current round of computation are deleted from HBM to free up space for future addition of new inference requests.

[0097] In this application, under the CB scenario, the granularity of batching is token rather than complete sequence, and the actual batch length is often less than the waiting queue length.

[0098] It is important to understand that the scheduler 21 is the main process running on the central processing unit (CPU) 10; the first executor 22 is the first child thread called by the main process; the execution hardware of the first executor 21 can be the central processing unit (CPU) or one or more accelerated computing units of the computing accelerator 11, without specific limitations here. The second executor 22 is the second child thread called by the main process, and the execution hardware of the second executor is one or more accelerated processing units in the computing accelerator 11.

[0099] Scheduler 21 sets up a decision queue and determines the cache space required for the next round of large model computation before the current round of large model computation begins. It performs an exchange operation in advance, that is, it prefetches the batch processing queue required for the next round of large model computation. The batch processing required for the next round of large model computation is executed in parallel with the current round of large model computation, saving the consumption of blocking and waiting in the next round of large model computation and reducing end-to-end latency.

[0100] The above is an introduction to the application system architecture and logical architecture of the large model inference cache scheduling provided in the embodiments of this application. Next, based on the above description and in conjunction with the accompanying drawings, the large model inference cache management method provided in the embodiments of this application will be introduced.

[0101] Figure 3 This document provides a scheduling flowchart for a large model inference cache management method according to an embodiment of this application. It is understood that the scheduling process of this method can be executed by a scheduler of any processor, device, equipment, or platform with computing power. Figure 3 As shown, the specific process includes the following steps: S31, determine the candidate inference request queue; S32, form a batch processing queue according to the candidate inference request queue, the batch processing queue is used for the current round of large model calculation; S33, allocate the first cache space and the second cache space according to the storage space occupancy on the high bandwidth memory HBM, the first cache space is used to provide the cache space required for the current round of large model calculation, and the second cache space is used to provide the cache space reserved for future large model calculations; S34, determine the decision queue; the decision queue is used to obtain part of the first cache space and prefetch the second cache space when forming the batch processing queue; the decision queue includes asynchronous exchange instructions and synchronous exchange instructions; S35, send the decision queue to the first executor; S36, send the formed batch processing queue to the second executor and call the kernel function of the large model calculation; S37, after the current round of large model calculation is completed, return to step S31 to perform the next round of iterative batch processing.

[0102] The above steps will be described in detail below with reference to specific implementation methods.

[0103] S31, determine the candidate inference request queue.

[0104] The candidate inference request queue includes multiple pending inference requests; among them, pending inference requests are inference requests waiting to execute the next round of large model calculation. Pending inference requests have completed pre-filled calculations, hold KV cache, and occupy part of HBM space.

[0105] The data structure for each pending inference request includes a request data token and a current state; in some possible implementations, the current state of each pending inference request can be identified by characters or numbers.

[0106] For example, "0" indicates "waiting for computation", "1" indicates "being swapped out", "2" indicates "already swapped out", and "3" indicates "being swapped in". The status of the inference request req to be processed can be determined based on whether the current status of the inference request req is "2"; if the determination result is "yes", the status of the inference request req is "already swapped out".

[0107] At the beginning of each iteration, the scheduler places multiple pending inference requests into the candidate inference request queue in order of priority; it then traverses the candidate inference request queue to determine the status of each pending inference request in the queue, which includes "waiting for computation", "being swapped out", "swapped out", "being swapped in", and "swapped in"; the status is used to indicate the current processing status of the pending inference request.

[0108] The inference request is in the "waiting to be computed" state before entering the Prefill phase for computation.

[0109] Inference requests during the Decoding phase include "swapping out", "swapping out", "swapping in", and "swapping in".

[0110] "Swapping out" indicates that the inference request is being moved out of HBM memory but has not yet been fully moved into CPU memory.

[0111] "Swapped out" means that the inference request has been completely moved into CPU memory.

[0112] "Swapping in" is a state where an inference request is being moved from the candidate inference request queue into the batch queue, but is not yet complete, or is being moved from CPU memory but is not yet fully moved into HBM memory.

[0113] "Swapped In" indicates that the inference request has been successfully moved into HBM memory and the batch queue has been set up.

[0114] For example, with a first-in-first-out (FIFO) priority, the scheduler can place multiple pending requests into the candidate inference request queue in a FIFO order at the start of each round of large model computation.

[0115] S32, group the candidate inference request queue into a batch processing queue, which is used for the current round of large model calculation.

[0116] In some possible implementations, the maximum allowable batch size can be calculated based on the storage space on the high-bandwidth memory and the large model, and the cache size of the batch processing queue can be determined. The cache size of the batch processing queue is less than or equal to the maximum batch size. Inference requests that occupy cache space that meets the cache size requirement are removed from the candidate inference request queue and added to the batch processing queue in a first-in-first-out order. The cache space occupied by the inference requests that meet the cache size requirement should be less than or equal to the cache size of the batch processing queue.

[0117] In some possible implementations, the scheduler employs continuous batching (CB) before the start of each round of iteration computation, performing batch processing queues at the granularity of word tokens; each token occupies a unit of cache space.

[0118] The cache space occupied by an inference request is related to the number of tokens in the inference request. In some possible implementations, the inference requests in the batch processing queue can be determined based on the number of tokens in the candidate inference request queue and the maximum batch processing volume. Inference requests that meet the cache space requirements are removed from the candidate inference request queue and added to the batch processing queue in order of priority. The cache space occupied by the qualified inference requests should be less than or equal to the cache size of the batch processing queue.

[0119] During each iteration of the batch queue, inference requests that have completed iterative computation will no longer exist in the batch queue. The scheduler releases the cache space it occupies in HBM so that new requests can join, saving time and resources.

[0120] In scenarios where HBM's storage space usage has not reached the saturation threshold, when the cache space occupied by inference requests in the candidate inference request queue Q1 is less than or equal to the cache space occupied by the maximum batch processing, the scheduler performs the following first batch processing method, adding the corresponding pending inference requests in the candidate inference request queue to the batch processing queue.

[0121] The first type of batch processing is as follows: Figure 4a As shown, each inference request req0, req1, ..., reqn in the candidate inference request queue Q1 is combined into the batch processing queue Q2. The actual number of tokens in the batch processing queue Q2 is equal to the number of tokens in the candidate inference request queue Q1.

[0122] In some possible implementations, the scheduler can determine whether the number of tokens n in the candidate inference request queue Q1 is less than the maximum batch size M before each round of large model computation begins. If the result is "yes", the inference requests in the candidate inference request queue Q1 are combined into the batch processing queue Q2 in a first-in-first-out order.

[0123] In some possible implementations, the saturation threshold for HBM's storage space can be a set maximum value for the proportion of HBM's storage space occupied. Saturation is determined based on whether the proportion of HBM's storage space occupied reaches the saturation threshold. For example, the maximum storage space occupancy rate is set to 98%. When the occupied space on HBM reaches 98% or more, it is determined that the HBM's storage space occupancy has reached the saturation threshold, and HBM is in a saturated state.

[0124] In some possible implementations, the saturation threshold of HBM's storage space can also be set based on the maximum number of tokens M that can be processed in a batch. When the unoccupied storage space of HBM does not meet the space requirement of the maximum number of tokens M that can be processed in a batch, it can be determined that the storage space occupancy of HBM has reached the saturation threshold.

[0125] In some possible implementations, the saturation threshold of HBM's storage space can also be the cache space required for the next round of large model computation. When the unused storage space of HBM cannot meet the cache space required for the next round of large model computation, it can be determined that the storage space occupancy of HBM has reached the saturation threshold. The state in which the storage space occupancy of HBM reaches the saturation threshold can be regarded as the storage space saturation state of HBM.

[0126] In some possible implementations, the candidate inference request queue is traversed to obtain the status of each pending inference request in the candidate inference request queue; it is determined whether the status of each pending inference request in the candidate inference request queue meets the status requirements, and the inference requests in the candidate inference request queue that meet the status requirements are added to the batch processing queue; the pending inference requests that meet the status requirements include inference requests with the status of "waiting for computation" or "already swapped in".

[0127] For pending inference requests with a status of "swapped out", they can be added to the batch processing queue when the next round of large model calculation is performed.

[0128] For example, in a scenario where the storage space occupancy of HBM has not reached the saturation threshold, if there is an inference request S in the candidate inference request queue Q1 that was swapped out due to insufficient cache space during the previous round of iterative inference calculation, the second batch processing method is executed.

[0129] The second batch processing method is as follows: Figure 4b As shown, the scheduler issues an asynchronous swap-in instruction to the inference request S with the status "swapped out", but does not group the content corresponding to the swapped-out request S into the current round's batch processing queue. Other requests are grouped into the current batch processing normally. When the next round of large model calculation begins, a synchronous swap-in instruction is issued to the swapped-out request queue S, grouping it into the corresponding batch processing queue Q2.

[0130] In some possible implementations, if there are inference requests with a "preemption successful" status in the candidate inference request queue Q1, the scheduler will add the inference request with the "preemption successful" status to the batch processing queue during the next round of computation.

[0131] In a scenario where HBM's storage space is saturated, if there is an inference request in the candidate inference request queue Q1 that successfully obtained cache space through preemption in the previous round of computation, the scheduler will perform the following third batch processing method.

[0132] The third batch processing method is as follows: Figure 4c As shown, due to the saturation of HBM's storage space, storage space is allocated to the pending inference requests in the candidate inference request queue Q1 through preemption. For preempted inference requests, such as reqn1, req3, ..., reqn, an asynchronous swap-out instruction is issued. However, when the current batch processing queue is formed, the space released by the swap-out is not used. In the next batch processing queue, a synchronous swap-out instruction is issued to use the space released by the swap-out.

[0133] Requests in an asynchronous exchange state occupy corresponding space on both the host and device sides because data transfer is not guaranteed to be completed in the current round. Although this space cannot be used for large-scale model computation in the current round, on the other hand, the prefetching implementation partially hides the time consumption of data transmission after the large-scale model computation, which helps to reduce latency and improve throughput.

[0134] In some possible implementations, a swap-out instruction is triggered when the number of pending requests n exceeds the maximum batch size M.

[0135] When HBM's storage space is saturated, there are still scenarios where batch processing queues cannot be formed without synchronous swapping out. In such scenarios, a synchronous swapping out command can be issued to the first executor.

[0136] S33, allocate a first cache space and a second cache space according to the storage space occupancy on the high-bandwidth memory HBM110. The first cache space is used to provide the cache space required for the current round of large model calculations, and the second cache space is used to provide the cache space required for future large model calculations.

[0137] In some possible implementations, step S33 includes the following steps S331-S332.

[0138] S331, allocate the first cache space required for the current round of large model computation.

[0139] In some possible implementations, it can be determined whether the space occupancy of the high-bandwidth memory has reached a saturation threshold; if the space occupancy of the high-bandwidth memory has not reached the saturation threshold, a first cache space for the current round of large model computation is allocated on the high-bandwidth memory, the size of which matches the cache space required for the current round of large model computation and the space occupied by inference requests in the corresponding batch queue.

[0140] When the high-bandwidth memory reaches a saturation threshold, destination space is allocated to the preemptible inference requests in the candidate inference request queue. The preemptible inference requests include the lowest priority inference requests in the candidate inference request queue; the destination space is CPU memory space. This swaps out some preemptible inference requests on HBM to free up some cache space, and allocates the cache space required for the current round of computation to the corresponding pending inference requests.

[0141] S332, determine the second cache space based on the first cache space.

[0142] In some possible implementations, the future large model computation includes the next round of large model computation. After allocating the first cache space, it is determined whether the space occupancy of HBM has reached the saturation threshold. If the space occupancy of HBM has not reached the saturation threshold, a second cache space is allocated on the high-bandwidth memory. The second cache space provides cache space that can be swapped in for inference requests of the batch processing queue required for the future large model computation.

[0143] If the space occupancy of the high-bandwidth memory has not reached the saturation threshold, and the queue of inference requests to be processed includes inference requests with the status of "swapped out", a second cache space is reserved for the "swapped out" inference requests during the current round of large model computation.

[0144] When the high-bandwidth memory reaches a saturation threshold, destination space is allocated to the preemptible inference requests in the candidate inference request queue. The preemptible inference requests include one or more of the lowest priority inference requests in the candidate inference request queue. The destination space is CPU memory space. This swaps out some low-priority inference requests on HBM to free up some HBM cache space, which is then reserved as a second cache space for the next round of large model computation.

[0145] When the space occupancy of high-bandwidth memory reaches the threshold of saturation, cache space can be obtained through preemption. The preemption method includes: first, allocating a destination space for the inference request to be preempted, which is the CPU memory space on the host side; then, the first executor moves the content of the inference request to be preempted to the destination space and releases the HBM cache space occupied by it.

[0146] If the space occupancy of high-bandwidth memory reaches the saturation threshold, and there are pending inference requests in the candidate inference request queue with the status of "being swapped out", the cache space can be released after the inference request in the "being swapped out" status is swapped out, and the released cache space can be used as a second cache space reserved for future large model calculations.

[0147] S34, determine the decision queue; the decision queue is used to obtain a portion of the first cache space and prefetch the second cache space when forming a batch processing queue; the decision queue includes asynchronous exchange instructions and synchronous exchange instructions.

[0148] Among them, asynchronous swapping instructions refer to instructions that asynchronously swap data into or out of the second buffer, and do not require waiting for the completion of the instruction during execution; synchronous swapping instructions refer to instructions that synchronously swap data into or out of the first buffer, and require waiting for the swapping-in or swapping-out instruction to complete during execution.

[0149] In some possible implementations, step S34 can be implemented in two ways, such as steps S341 and S342.

[0150] S341, when the space occupancy of the high-bandwidth memory has not reached the saturation threshold, the decision queue includes asynchronous swap-in instructions and synchronous swap-in instructions; the asynchronous swap-in instructions are used to asynchronously swap the inference requests required for the next round of computation into the second cache space, and do not need to wait for the instructions to complete during execution; the synchronous swap-in instructions are used to synchronously swap the inference requests required for the current round of computation into the first cache space, and need to wait for the instructions to complete during execution; the number of asynchronous swap-in instructions is one or more; the decision queue is used to obtain part or all of the first cache space when forming the batch processing queue through synchronous swap-in instructions, and to prefetch the second cache space through asynchronous swap-in instructions.

[0151] In some possible implementations, the decision queue Q3 further includes a first temporary queue for storing synchronous swap-in instructions. Before the current round of batch processing begins, the synchronous swap-in instructions in the first temporary queue are added to the decision queue, the asynchronous swap-in instructions to be executed in the current round are added to the decision queue Q3, and the synchronous swap-in instructions to be executed in the next round of batch processing are added to the first temporary queue. The number of asynchronous swap-in instructions can be one or more. Before the current round of large model computation begins, the decision queue Q3 obtains part of the first cache space required for the current round of large model computation in the batch processing queue through synchronous swap-in instructions. During the current round of large model computation, the second cache space is prefetched through one or more asynchronous swap-in instructions.

[0152] For example, if the storage space is not saturated, the scheduler can traverse the candidate inference request queue and issue several asynchronous swap-in instructions to the first executor to prefetch the second cache space required for the next round of large model computation earlier.

[0153] S342, when the space occupancy of the high-bandwidth memory reaches the saturation threshold, the freed storage space is obtained through a "preemption" mechanism. The decision queue Q3 used for "preemption" of cache space also includes asynchronous swap-out instructions and synchronous swap-out instructions. An asynchronous swap-out instruction is an instruction that asynchronously swaps data out of the cache space; execution does not require waiting for the instruction to complete. A synchronous swap-out instruction is an instruction that synchronously swaps data out of the cache space; execution requires waiting for the swap-out instruction to complete. The number of asynchronous swap-out instructions can be one or more. Before the current round of large-scale model computation begins, the decision queue uses synchronous swap-out instructions to swap out inference requests in the candidate inference request queue that are in the "swapping out" state from the high-bandwidth memory to the destination space. During the current round of large-scale model computation, one or more asynchronous swap-out instructions are used to pre-swap out inference requests in the candidate inference request queue that are to be preempted from the high-bandwidth memory, thereby releasing the space occupancy of the high-bandwidth memory to restore it to a state where the saturation threshold has not been reached.

[0154] In this embodiment, the decision queue Q3 further includes a second temporary queue, which is used to temporarily store the synchronous swap-out instructions. Before the current round of large model calculation begins, the synchronous swap-out instructions in the second temporary queue can be added to the decision queue; during the current round of large model calculation, one or more asynchronous swap-out instructions can be added to the decision queue; before the current round of large model calculation begins, the decision queue releases the cache space required for the current round of large model calculation by using the synchronous swap-out instructions when batch processing the queue; during the current round of large model calculation, the cache space required for the next round of large model calculation can be pre-released by using one or more asynchronous swap-out instructions.

[0155] When the first pending inference request encountered during batch processing of the queue requires space freed up by a "swap out" instruction, a synchronous swap out can be performed. In this case, due to space constraints, the scheduler can issue several additional asynchronous swap out instructions to free up more space for subsequent iterations.

[0156] It is important to understand that when the current round scheduler issues an asynchronous swap-out request, the first executor removes a portion of the cache space released by the request from HBM. This portion is not used in the current round of large model computation. When the next round of large model computation requires additional space, the scheduler issues the corresponding synchronous swap-out instruction, waits for the swap-out operation to complete, and then allocates the already released cache space to the large model computation.

[0157] S35 sends the decision queue to the first executor.

[0158] If the space occupancy of the high-bandwidth memory has not reached the saturation threshold, a synchronous swap-in instruction is issued to the first executor before the current round of large model calculation begins; during the current round of calculation, one or more asynchronous swap-in instructions are sent to the first executor.

[0159] In some possible implementations, the sufficiency of storage space can also be determined by the percentage of storage space occupied. If the sufficiency of HBM's storage space meets the requirements, the waiting queue can be traversed and several additional asynchronous swap-in decisions can be added to prefetch these future requests earlier.

[0160] When the space occupancy of the high-bandwidth memory reaches the saturation threshold, before the current round of large model calculation begins, the synchronous swap-out instruction is issued to the first executor; during the current round of large model calculation, one or more asynchronous swap-out instructions are issued to the first executor.

[0161] For example, during the current round of large model computation, the scheduler issues an asynchronous swap-out instruction to request 1. In the next round of large model iteration computation, when the scheduler encounters request 1 in the storage space that has already received an asynchronous swap-out instruction but has not yet completed the swap-out, it issues a synchronous swap-out instruction to it. After request 1 in the storage space has completed the swap-out, the HBM's storage space is released and restored to an unsaturated state. In the next round of batch processing, the decision queue can be determined using the implementation method described in S341, which will not be elaborated further here.

[0162] In some possible implementations, when the first inference request in the batch queue needs to use the cache space released by the swap request, the scheduler issues a synchronous swap instruction to the lower-priority inference request in the cache space to release the cache space it occupies. While waiting for the synchronous swap instruction to be executed by the first executor, the scheduler issues several asynchronous swap instructions to multiple lower-priority inference requests in the cache space to release more space for future large model calculations, thereby alleviating space shortage and shortening the waiting time.

[0163] S36, after the synchronization exchange is completed, the grouped batch queue is obtained and sent to the second executor.

[0164] S37. After the current round of large model calculation is completed, return to step S31 to perform the next round of iterative batch processing.

[0165] The above is a partial discussion of the scheduling phase of large model inference cache management. Next, we will continue to discuss the execution phase of large model inference cache management after the first executor receives the decision queue.

[0166] Figure 5 This is a flowchart illustrating the execution of the first executor in the large model inference cache management method provided in this application embodiment. It is understood that the execution flow of this method can be executed by any processor, device, equipment, or platform with computing power. Figure 5 As shown, the execution flow of the first executor includes the following steps:

[0167] S51, the first executor receives the decision queue; S52, before the current round of large model calculation begins, a synchronous exchange instruction is executed to swap the requests in the candidate inference request queue into the allocated first buffer space, so that the scheduler obtains the grouped batch processing queue; S53, during the current round of large model calculation, an asynchronous exchange instruction is executed to swap the requests in the candidate inference request queue into the allocated second buffer space.

[0168] The following describes steps S51-S53 in detail with reference to specific implementation methods.

[0169] S51, the first actuator receives the decision queue.

[0170] When the space occupancy of the high-bandwidth memory has not reached the saturation threshold, the decision queue received by the first actuator includes asynchronous swap-in instructions and synchronous swap-in instructions.

[0171] When HBM has ample storage space, the decision queue received by the first executor includes synchronous swap-in requests and several non-synchronous swap-out requests.

[0172] When the space occupancy of the high-bandwidth memory reaches a saturation threshold, the decision queue received by the first actuator includes asynchronous swap-out instructions and synchronous swap-out instructions.

[0173] When the first pending inference request encountered by the scheduler in the batch processing queue needs to free up space using the "swap out" instruction, the decision queue received by the first executor includes synchronous swap out instructions and several asynchronous swap out instructions.

[0174] S52, before the current round of large model calculation begins, the first executor executes the synchronization exchange instruction to swap the requests in the candidate inference request queue into the allocated cache space, so that the scheduler can obtain the grouped batch processing queue.

[0175] In some possible implementations, step S52 includes the following two implementation steps:

[0176] S521, if the space occupancy of the high-bandwidth memory has not reached the saturation threshold, before the current round of large model calculation begins, the first executor executes the synchronous swap-in instruction to swap the inference requests in the candidate inference request queue with the status of "swapping in" into the allocated first cache space. After the execution is completed, the status of the inference request is updated to "swapped in".

[0177] The scheduler executes step S36, waits for all swaps to be completed, obtains the grouped batch processing queue based on the "swapped" inference requests, sends the grouped batch processing queue to the second executor, and calls the kernel function for large model calculation.

[0178] S522, when the space occupancy of the high-bandwidth memory reaches the saturation threshold, before the current round of large model calculation begins, the first executor executes a synchronous swap-out instruction to swap out the inference request with the status "swapping out" from the high-bandwidth memory to the allocated destination space. After the swap-out is completed, the status of the inference request is updated to "swapped out".

[0179] The scheduler waits for the first executor to complete its swap-out operation before releasing the cache space occupied by inference requests with a status of "swapped out".

[0180] S53, during the current round of large model calculation, the first executor executes an asynchronous swap instruction to swap the requests in the candidate inference request queue into the allocated second cache space.

[0181] In some possible implementations, step S53 includes the following two implementation steps:

[0182] S531, when the space occupancy of the high-bandwidth memory has not reached the saturation threshold, during the current round of large model calculation, the first executor executes an asynchronous swap-in instruction, puts the inference request with the status of "swapped out" in the candidate inference request queue into the allocated second cache space, and updates the status of the inference request with the status of "swapped out" to "swapping in".

[0183] While the first executor executes the asynchronous swap-in instruction, the scheduler can call the kernel function for large model inference and start the large model computation for the current round. This does not require waiting for the first executor to complete the swap-in.

[0184] S532, when the space occupancy of the high-bandwidth memory reaches the saturation threshold, during the current round of large model calculation, the first executor executes an asynchronous swap instruction to swap out the preempted inference request from the high-bandwidth memory to release the occupied cache space, and updates the status of the preempted inference request to "swapping out".

[0185] When the first executor executes the asynchronous swap instruction, the scheduler 21 does not need to wait for the first executor to complete the swap, and can call the kernel function of the large model inference to start the large model computation of the current round.

[0186] When additional cache space is needed for the next round of large model computation, the inference request in the "swapping out" state has not yet been swapped out. Before the next round of large model computation begins, refer to step S522, the first executor executes the synchronous swapping out instruction to continue releasing HBM cache space. After the synchronous swapping out instruction is completed, the HBM storage space returns to the "unsaturated" state.

[0187] Although the first executor executing the asynchronous swap instruction may release some new HBM cache space, the large model computation does not use the released HBM cache space in this round. The large model computation will use the released cache space after the next round of synchronous swap instructions is completed.

[0188] Since asynchronous swapping instructions and synchronous swapping instructions are issued in pairs for the same preempted inference request, and asynchronous swapping instructions begin to execute before the next round of large model computation, the time for releasing cache space is brought forward, which greatly shortens the waiting time for the next round of large model computation.

[0189] The large model inference cache management method provided in this application embodiment also includes the execution process of a second executor.

[0190] When the second executor responds to the scheduler's call in step S531, it starts the large model calculation for the current round, executes the large model calculation for the current round according to the grouped batch processing queue, and stores the calculation result of the current round in the first cache space. After the large model calculation for the current round is completed, it notifies the scheduler to proceed to the next round of iteration and releases the space occupied by the batch processing queue that has completed the calculation for the current round.

[0191] When the second executor performs the current round of large model calculations, it does not need to wait for the asynchronous swap-in instruction in step S531 to be completed. Since the time required from when the first executor starts executing the asynchronous swap-in instruction to when the next round of large model calculations is blocked to wait for the swap-in to complete is greatly reduced, the time required is significantly reduced.

[0192] After the current round of large model computation is completed, the scheduler in the method provided in this application returns to step S31 to perform the next round of iterative scheduling, determines the candidate inference request queue required for the next round of large model computation, forms the batch processing queue required for the next round of large model computation based on the candidate inference request queue required for the next round of large model computation, uses the second cache space when forming the batch processing queue required for the next round of large model computation, and the decision queue also includes a synchronous waiting instruction; the synchronous waiting instruction is used to wait for the return information of the current round of large model computation, including completion information, preemption success information, etc.

[0193] Therefore, the large model inference cache management method provided in this application determines the decision queue of data required for the next round of inference at the beginning of the current round of large model calculation, performs data exchange operations in advance, pre-fetches the batch processing queue required for the next round of large model calculation, and executes the batch processing queue and the inference calculation of the computing acceleration processor in parallel, saving the consumption of blocking and waiting in the next round of calculation and reducing end-to-end latency.

[0194] Example 1

[0195] Figure 6a This is a scheduling flowchart for a large model inference cache management method provided in Embodiment 1 of this application. In scenarios where HBM is not saturated, the data exchange and inference computation parts in the large model inference process are overlapped to reduce the latency consumption of data exchange, thereby improving end-to-end throughput.

[0196] The large model inference cache management method provided in Embodiment 1 of this application may be partially or entirely referred to the foregoing. Figures 1-5 The relevant content in [the document / reference] clarifies that this method can be executed by any device, equipment, or platform with computational processing capabilities. For example... Figure 6a As shown, the scheduler begins processing the current round group batch, including the following steps S601-S605.

[0197] S601. Determine the candidate inference request queue Q1. The candidate inference request queue includes multiple inference requests to be processed.

[0198] At the beginning of each round of large model calculation, the scheduler adds multiple requests to be processed to the candidate inference request queue Q1 in the order of priority.

[0199] S602. Traverse the candidate inference request queue Q1, determine the status of the inference requests to be processed in the current round, and allocate cache space for the candidate inference request queue according to the status. The size of the cache space needs to meet the usage requirements of the next round of large model calculation. The specific process includes the following steps:

[0200] S6021. Traverse the candidate inference request queue Q1 to determine whether Q1 has not been traversed completely and whether the length n of the current batch processing queue Q2 is less than the maximum batch processing length M. If Q1 has not been traversed completely and n < M, execute step S6022; if Q1 has been traversed completely or n ≥ M, execute step S605.

[0201] S6022. Obtain the inference request req at the current head of the queue of Q1, judge the status of the inference request req, and allocate cache space for the candidate inference request queue according to the status of the inference request. The size of the cache space needs to meet the usage requirements of the next round of large model calculation.

[0202] As Figure 6a shown, judge whether the status of the inference request req is "swapped out" and whether the current group batch processing status is "not urgent". If the judgment result is "no", execute step S6023; if the judgment result is "yes", go to Figure 7 execute S701. The specific implementation can refer to Embodiment 2 of this application.

[0203] In some possible implementation manners, the status of "swapped out" can be determined according to whether the current status of the inference request req is the first identifier; if the determination result is "yes", the status of the inference request req is "swapped out". Exemplarily, the first identifier is "2", indicating that the current status is "swapped out".

[0204] The status of "urgent" means that if the inference request req is not swapped into the first cache space immediately, no requests can be grouped for the current batch processing; on the contrary, if requests can still be grouped for the current batch processing even if the inference request req is not swapped into the first cache space immediately, it is the status of "not urgent".

[0205] There can be different specific implementation manners for judging whether the current group batch processing is in the "urgent" status.

[0206] In some possible implementations, whether a process is "urgent" can be determined randomly. A threshold for the probability distribution of an "urgent" state can be specified. If the probability distribution of an "urgent" state reaches the threshold, the current batch is determined to be in an "urgent" state. If the probability distribution does not reach the threshold, the current batch is determined to be in a "non-urgent" state, thereby controlling the probability of asynchronous switching.

[0207] In some possible implementations, the "urgent" state can be determined by whether an immediate swap-in operation is required; if the determination result is "yes," the state of the request request (req) is inferred to be "urgent." If the determination result is "no," the state of the request request (req) is inferred to be "not urgent."

[0208] In some possible implementations, requests in the "urgent" state may not be processed urgently. In this case, the current round of large model calculation is skipped, and asynchronous swap requests in the next round of large model calculation that are in the "urgent" state will be processed and swapped in synchronously without affecting the normal execution of the program.

[0209] S6023, determine whether the status of the inference request req is "swapping in". If the result is "no", proceed to step S6024; if the result is "yes", proceed to step S603.

[0210] S6024, determine whether the storage space of HBM is sufficient for large model computation; if the result is "no", proceed to... Figure 8 Step S811 determines whether cache space for large model computation can be obtained through preemption. For specific implementation, please refer to Embodiment 3 of this application. If the determination result is "yes" or if cache space for large model computation is successfully obtained through preemption, step S6025 is executed.

[0211] S6025 allocates a second cache space for the inference request req for the next round of large model computation.

[0212] S603: Remove the synchronization swap-in instruction corresponding to the inference request req from the first temporary queue S1, add the synchronization swap-in instruction to the decision queue Q3, and logically release the source data space. "Logically" means that the instruction is made in the program, but it is not actually executed.

[0213] S604, dequeue req from candidate inference request queue Q1 and add it to batch processing queue Q2, then return to step S6021.

[0214] The length of the batch processing queue Q2 is n = n + req_size; req_size is the size of the cache space occupied by each inference request.

[0215] S605 sends the decision queue to the first executor and the grouped batch processing queue to the second executor.

[0216] S606, calling the kernel function for large model calculation, ends this round of batch processing scheduling.

[0217] It should be noted that if an asynchronous swap-in instruction swap_in_async(srcBlockId,dstBlockId) has already been executed, and then a corresponding synchronous swap-in instruction swap_in_sync(srcBlockId,dstBlockId) is issued and executed, the synchronous decision will not re-transfer the data, but will wait for the previous asynchronous task to complete; if the previous task has already completed, it will return directly.

[0218] Similarly, if an asynchronous swap out instruction swap_out_async(srcBlockId, dstBlockId) has been executed, and then a corresponding synchronous swap out instruction swap_out_sync(srcBlockId, dstBlockId) is issued and executed, the synchronous instruction will not perform data transfer again, but will wait for the previous asynchronous task to complete; if the previous task has already completed, it will return directly.

[0219] In some possible implementations, to synchronize previously executed asynchronous tasks, the scheduler can also issue a blocking and waiting instruction to perform the same function.

[0220] If a synchronous instruction is not paired with a previous asynchronous instruction, then the synchronous instruction needs to perform the entire data transfer and wait for the transfer task to complete.

[0221] Next, the cache management process after the first executor receives the decision queue in Embodiment 1 of this application will be further described.

[0222] Figure 6b This is a schematic diagram illustrating the execution flow of the large model inference cache management method provided in Embodiment 1 of this application. Figure 6b As shown, it includes the following steps S610-S612.

[0223] S610, the actuator receives the batch processing queue.

[0224] In scenarios where HBM is not saturated, the decision queue includes asynchronous swap-in instructions and synchronous swap-in instructions.

[0225] In scenarios where HBM has ample storage space, the decision queue includes synchronous swap-in instructions and several asynchronous swap-in requests.

[0226] S611, if the space occupancy of the high-bandwidth memory has not reached the saturation threshold, before the current round of large model calculation begins, the first executor executes the synchronous swap-in instruction to swap the inference request req1 in the candidate inference request queue with the status of "swapping in" into the allocated first cache space. After the execution is completed, the status of the inference request is updated to "swapped in".

[0227] In some possible implementations, the batch queue also includes a request req2 with a status of "swapped out". In the current round, an asynchronous swap-in instruction is executed to put the request req2 with a status of "swapped out" into the allocated second buffer space without waiting for the swap-in operation to complete.

[0228] When space is ample, the executor can traverse the waiting queue and execute multiple asynchronous swap-in instructions simultaneously to prefetch requests needed for computation of multiple future large models earlier.

[0229] S612, during the current round of large model calculation, the first executor executes an asynchronous swap instruction to swap the request req in the candidate inference request queue into the allocated second cache space.

[0230] For example, if the space occupancy of the high-bandwidth memory has not reached the saturation threshold, for requests with a status of "swapped out," an asynchronous swap-in instruction can be executed first, i.e., prefetching, while inference computation is performed simultaneously, without waiting for the data exchange to complete. In later iterations, if a request that has already undergone an asynchronous swap-in operation is encountered again, a synchronous swap-in instruction is executed, waiting for the previous asynchronous swap-in instruction to complete before the next round of inference computation. Since the asynchronous swap-in operation may have already started before the current round, and the req status is "swapping in," the waiting time required for this round of blocking is greatly reduced.

[0231] The second executor responds to the call in step S605 by retrieving the existing batch queue from the first cache and starting the large model calculation for the current round.

[0232] The second executor retrieves the batch processing queue from HBM to drive the large model to perform the current round of large model computation. This is done simultaneously with the first executor asynchronously swapping the data of the batch processing queue required for the next round of large model computation into HBM. After the current round of large model computation is completed, the results of the current round of computation are stored in the first cache space. The scheduler is notified to proceed to the next round of iteration, and the space occupied by the batch processing queue that has completed the current round of computation is released.

[0233] Therefore, the large model inference cache management method provided in Embodiment 1 of this application determines the decision queue of the data required for the next round of decoding when the current round of large model calculation begins in the scenario where HBM is not saturated, performs data exchange operation in advance, that is, pre-fetches the batch processing queue required for the next round of decoding settlement, and performs the exchange of batch processing queues in parallel with the large model inference calculation, thereby saving the consumption of inference blocking and waiting and reducing end-to-end latency.

[0234] Example 2

[0235] Figure 7 This is a schematic diagram illustrating an application scenario of a large model inference cache management method provided in Embodiment 2 of this application. It is applied to a scenario where, during the current batch processing of a round group, the head request (req) in the candidate inference request queue is in a "swapped out" state. For example... Figure 7 As shown, steps S601-S6023 can be referred to the implementation of steps S601-S6023 in the embodiment, and will not be repeated here. The process also includes the following steps S701-S706.

[0236] S701 allocates cache space to the candidate inference request queue based on the status.

[0237] Specifically, such as Figure 7 As shown, determine whether the HBM space is sufficient for the inference request req and the next round of large model calculation; if the determination result is "yes", execute step S703; if the determination result is "no", execute step S702.

[0238] S702, determine whether cache space can be preempted for the current inference request req. If the result is "yes", proceed to step S703; if the result is "no", return to step S6021.

[0239] In some possible implementations, the status return value can be used to determine whether the current inference request `req` can preempt space. If the status return value is "preemption successful," step S703 is executed; if the status return value is "preemption failed," the execution process returns to step S6021. The determination of the status return value can be transferred to... Figure 8 Step S811 is executed to obtain a status return value to determine whether the cache space can be preempted. For specific implementation details, please refer to Embodiment 3 of this application, which will not be repeated here.

[0240] S703 allocates cache space for swapping in the inference request req.

[0241] S704, determine the decision queue Q3 for obtaining the cache space for the next round of inference.

[0242] like Figure 7 As shown, asynchronous swap-in requests can be added to decision queue Q3, and corresponding synchronous swap-in requests can be added to the first temporary queue S1.

[0243] S705 allocates cache space for the current inference request `req` for the next round of large model computation. Both the cache space for asynchronous swapping and the cache space for the next round of large model computation are second cache spaces on HBM.

[0244] S706, update the current inference request req status to "swapping in", return to step S6021 for the next round of batch processing, refer to the implementation method of Example 1.

[0245] Continue traversing the candidate inference request queue Q1 until the traversal ends; proceed to step S605 to send the decision queue to the first executor; send the grouped batch processing queue to the second executor; then execute step S606 to call the kernel function for large model calculation, and this round of group batch processing scheduling ends.

[0246] Next, after receiving the decision queue, the first executor executes steps S610-S614, and the second executor begins processing the large model calculation for the current round. Specific implementation details can be found in some or all of the embodiments described in Example 1, and will not be repeated here.

[0247] In the scenario of Embodiment 2 of this application, there are inference requests S in the candidate inference request queue Q1 that were swapped out in the previous round of large model calculation due to insufficient space. The scheduler will not group the corresponding content of the inference requests marked as "swapped out" into the current batch processing for the time being, while other inference requests will be grouped into the current batch processing normally. In the next round of large model calculation, the queue of requests that have been swapped out will be grouped into the batch processing queue Q2.

[0248] Example 3

[0249] When allocating cache space for the candidate inference request queue, if the storage space of the first cache / KV cache becomes saturated, cache space can be acquired through preemption. Preemption methods include removing the inference request to be preempted from the candidate queue Q1 to release the cache space it occupies with its data; or waiting for the inference request being swapped out to release the cache space it occupies, until the size of the released cache space is sufficient to accommodate the swapping in of the currently pending inference request and the next round of large model computation.

[0250] Figure 8 This is a flowchart illustrating a preemption scenario in the large model inference cache management method provided in Embodiment 3 of this application.

[0251] The large model inference cache management method provided in Embodiment 3 of this application may be partially or entirely referred to the foregoing. Figures 1-7 The relevant description is provided. It is understood that this method can be executed by any device, apparatus, or platform with computational processing capabilities. For example... Figure 8 As shown, the process of acquiring cache space through preemption includes the following steps S811-S844.

[0252] S811, begin preemption, determine if there is a reasoning request req1 in the candidate reasoning request queue that is "being swapped out"; if the result is "no", proceed to step S821; if the result is "yes", proceed to step S841.

[0253] S821, retrieve the request req2 that will be preempted from the candidate inference request queue.

[0254] S822, determine whether the current batch processing status is "urgent"; if the determination result is "no", proceed to step S823; if the determination result is "yes", proceed to step S831.

[0255] S823 logically allocates cache space for swapping in.

[0256] S824 adds the asynchronous swap-out instruction to the decision queue Q3 and adds the synchronous swap-out instruction to the second temporary queue S2.

[0257] The second temporary queue S2 is used to temporarily store synchronization swap-out instructions.

[0258] S825, update the status of req2 to "Swapping out".

[0259] S826, set the preemption result to "failure" and return it to the scheduler, then execute step S845.

[0260] S831 logically allocates destination space for swapping out and releases source space. The destination space for swapping out is the CPU's memory space.

[0261] S832 adds the synchronous swap-out instruction to the decision queue Q3.

[0262] S833, set the status of req2 to "swapped out", and execute step S844.

[0263] S841, retrieve and remove the synchronization swap instruction corresponding to request req1 from the temporary storage queue S2, and add the synchronization swap instruction corresponding to req1 to the decision queue Q3.

[0264] S842 logically releases the source data space.

[0265] S843, set the status of request req1 to "swapped out", and execute step S844.

[0266] S844, set the preemption result to "success" and return it to the scheduler, then execute step S45.

[0267] S845, end the preemption and return the preemption result to step S6024 in Example 1 or step S702 in Example 2. Subsequent steps can be referred to Example 1 and Example 2, and will not be repeated here.

[0268] In the scenario of preemption in the large model inference cache management method provided in Embodiment 3 of this application, since the request is in an asynchronous swap-out state, the data exchange is not guaranteed to be completed in the current round. It will occupy the corresponding cache space of the host-side CPU memory and the computing acceleration processor HBM at the same time. The corresponding cache space cannot be used by the large model calculation in the current round. The large model calculation in the current round does not use the freed space. In the next round of large model calculation, the freed space is used to store the intermediate result data of the large model calculation.

[0269] In this way, the time consumed by data transmission can be partially hidden after the large model calculation, which helps to reduce latency and increase throughput.

[0270] The large model inference cache management method proposed in this application can balance computing power utilization and the additional latency consumption of data exchange, thereby optimizing overall performance.

[0271] The above is an introduction to the large model inference cache management method provided by the embodiments of this application. It should be understood that the sequence number of each step in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application. In addition, in some possible implementations, each step in the above embodiments may be selectively executed according to the actual situation, and may be partially or fully executed, without limitation here. Furthermore, all or part of any feature of any of the above embodiments may be freely and arbitrarily combined without contradiction; the combined technical solution is also within the scope of this application.

[0272] Next, based on the above, the apparatus for scheduling large model inference caches provided in the embodiments of this application will be described. For relevant descriptions of concepts, formulas, etc., involved in the following content, please refer to the above text.

[0273] Figure 9 This application provides a large model inference cache management device. For example... Figure 9 As shown, the large model inference cache management device 900 includes a scheduling module 91, a first execution module 92, and a second execution module 93.

[0274] The scheduling module 91 determines the candidate inference request queue; it assembles a batch processing queue based on the candidate inference request queue, the batch processing queue being used for the current round of large model computation; it allocates a first cache space and a second cache space based on the storage space occupancy on the high-bandwidth memory (HBM), the first cache space being the cache space required for the current round of large model computation, and the second cache space being used to provide the cache space required for future large model computations; it determines the decision queue; the decision queue is used to acquire a portion of the first cache space and prefetch the second cache space when assembling the batch processing queue; the decision queue includes synchronous exchange instructions and asynchronous exchange instructions; it sends the decision queue to the first execution module; it sends the assembled batch processing queue to the second execution module; while the first execution module executes the asynchronous exchange instructions, the scheduler module calls the kernel function for large model computation; and

[0275] The first execution module 92 receives the batch processing queue and the decision queue; according to the decision queue, it executes asynchronous exchange instructions and synchronous exchange instructions round by round, swapping the batch processing queue into the allocated cache space. After the swapping is completed, a second batch processing queue is obtained. The second batch processing queue is the request queue required for the current round of large model calculation; while executing the asynchronous exchange instructions, the kernel function of the large model inference is called.

[0276] The second execution module 93 responds to the kernel function of the large model inference called by the scheduler, executes the current round of large model calculation according to the assembled batch processing queue, and stores the calculation result of the current round in the first cache space.

[0277] Among them, the pending inference requests are inference requests waiting to execute the next round of large model computation. These pending inference requests have completed pre-filled computation, hold the first cache, and occupy part of the HBM space. The candidate inference request queue can be denoted as Q1.

[0278] In some possible implementations, the scheduling module 91 may select the token generated in the previous iteration as the reasoning request to be processed for continuous batch processing to obtain a batch processing queue.

[0279] For example, at the beginning of each round of iteration calculation, the scheduling module 91 selects the tokens generated by the previous round of large model calculation for batch processing; during each iteration of batch processing, the completed inference requests are deleted from the batch processing queue and new requests are added.

[0280] In some possible implementations, when the current round of large model calculation begins, the scheduling module 91 determines the data exchange instructions required for the next round of large model calculation and issues the decision queue Q3 and batch processing queue Q2 to the first execution module 92.

[0281] As an example of a software functional unit, the scheduling module 91 may include code running on a computing instance. The computing instance may include at least one of a physical host (computing device), a virtual machine, or a container. Further, the aforementioned computing instance may be one or more. For example, the scheduling module 91 may include code running on multiple hosts / virtual machines / containers. It should be noted that the multiple hosts / virtual machines / containers used to run the code may be distributed within the same region or in different regions. Further, the multiple hosts / virtual machines / containers used to run the code may be distributed within the same availability zone (AZ) or in different AZs, each AZ including one or more geographically proximate data centers. Typically, a region may include multiple AZs.

[0282] Similarly, multiple hosts / virtual machines / containers used to run this code can be distributed within the same Virtual Private Cloud (VPC) or across multiple VPCs. Typically, a VPC is set up within a region. Communication between two VPCs within the same region, as well as between VPCs in different regions, requires a communication gateway to be set up within each VPC to enable interconnection between VPCs.

[0283] As an example of a hardware functional unit, the scheduling module 91 may include at least one computing device, such as a computing device. Alternatively, the scheduling module 91 may also be a device implemented using an application-specific integrated circuit (ASIC) or a programmable logic device (PLD). The PLD may be implemented using a complex programmable logical device (CPLD), a field-programmable gate array (FPGA), generic array logic (GAL), or any combination thereof.

[0284] The scheduling module 91 includes multiple computing devices that can be distributed within the same region or in different regions. Similarly, the scheduling module 91 includes multiple computing devices that can be distributed within the same Availability Zone (AZ) or in different AZs. Likewise, the scheduling module 91 includes multiple computing devices that can be distributed within the same Virtual Private Cloud (VPC) or in multiple VPCs. These multiple computing devices can be any combination of computing devices such as computing devices, ASICs, PLDs, CPLDs, FPGAs, and GALs.

[0285] It should be noted that in other embodiments, Figure 9 The steps implemented by the scheduling module 91 and the first execution module 92 shown can be specified as needed. The scheduling module 91 and the first execution module 92 respectively implement different steps in the method provided in Figure 4 to realize all the functions of the device 110 for AI model performance analysis.

[0286] This application also provides a computing device 1000. For example... Figure 10 As shown, the computing device 1000 includes a bus 1002, a processor 1004, a memory 1006, and a communication interface 1008. The processor 1004, the memory 1006, and the communication interface 1008 communicate with each other via the bus 1002. The computing device 1000 can be a computing device or a terminal device. It should be understood that this application does not limit the number of processors and memories in the computing device 1000.

[0287] Bus 1002 can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. Buses can be categorized as address buses, data buses, control buses, etc. For ease of representation, Figure 10 The bus 1004 may be represented by a single line, but this does not mean that there is only one bus or one type of bus. The bus 1004 may include a path for transmitting information between various components of the computing device 1000 (e.g., memory 1006, processor 1004, communication interface 1008).

[0288] The processor 1004 may include any one or more processors such as a central processing unit (CPU), a graphics processing unit (GPU), a microprocessor (MP), or a digital signal processor (DSP).

[0289] The memory 1006 may include volatile memory, such as random access memory (RAM). The processor 104 may also include non-volatile memory, such as read-only memory (ROM), flash memory, hard disk drive (HDD), or solid state drive (SSD).

[0290] The memory 1006 stores executable program code, and the processor 1004 executes the executable program code to implement the functions of the scheduling module 91 shown in FIG4, thereby implementing all or part of the steps of the method in the above embodiments. That is, the memory 1006 stores instructions for executing all or part of the steps of the method in the above embodiments.

[0291] The communication interface 1008 uses transceiver modules such as, but not limited to, network interface cards and transceivers to enable communication between the computing device 1000 and other devices or communication networks.

[0292] This application also provides a computing device cluster. The computing device cluster includes at least one computing device 1000. This computing device can be a central computing device, an edge computing device, or a local computing device in a local data center. In some embodiments, the computing device can also be a terminal device such as a desktop computer, a laptop computer, or a smartphone.

[0293] like Figure 11 As shown, the computing device cluster includes at least one computing device 1000. The memory 1006 of one or more computing devices 1000 in the computing device cluster may store the same instructions for performing all or part of the steps in the methods of the above embodiments.

[0294] In some possible implementations, the memory 1006 of one or more computing devices 1000 in the computing device cluster may also store partial instructions for performing all or part of the steps in the methods of the above embodiments. In other words, a combination of one or more computing devices 1000 can jointly execute instructions for performing all or part of the steps in the methods of the above embodiments.

[0295] It should be noted that the memory 1006 in different computing devices 1000 within the computing device cluster can store different instructions, each used to execute a portion of the functions of the device 110 for AI model performance analysis. That is, the instructions stored in the memory 1006 of different computing devices 1000 can implement the aforementioned... Figure 10 The functions of one or more of the scheduling module 91, the first execution module 92, and the second execution module 93 shown are illustrated.

[0296] In some possible implementations, one or more computing devices in a computing device cluster can be connected via a network. This network can be a wide area network (WAN) or a local area network (LAN), etc. Figure 12 One possible connection method is shown. For example... Figure 12 As shown, two computing devices 1000A and 1000B are connected via a network. Specifically, they are connected to the network through communication interfaces in each computing device. In this possible implementation, the memory 1006 in computing device 1000A stores instructions for the functions of the scheduling module 91, the first execution module 92, and the second execution module 93. Simultaneously, the memory 1006 in computing device 1000B stores instructions for executing the functions of the second execution module 93.

[0297] It should be understood that Figure 12 The functions of computing device 1000A shown can also be performed by multiple computing devices 1000. Similarly, the functions of computing device 1000B can also be performed by multiple computing devices 1000.

[0298] Based on the methods in the above embodiments, this application provides a computer-readable storage medium storing a computer program that, when run on a processor, causes the processor to execute the methods in the above embodiments.

[0299] Based on the methods in the above embodiments, this application provides a computer program product that, when run on a processor, causes the processor to execute the methods in the above embodiments.

[0300] It is understood that the various numerical designations used in the embodiments of this application are merely for descriptive convenience and are not intended to limit the scope of the embodiments of this application.

Claims

1. A method for managing large model inference caches, characterized in that, The method includes a scheduler, a first executor, and a second executor, comprising: In the scheduler: a candidate inference request queue is determined; a batch processing queue is formed according to the candidate inference request queue, the batch processing queue being used for the current round of large model computation; The first cache space and the second cache space are allocated according to the storage space occupancy on the high bandwidth memory HBM. The first cache space is the cache space required for the current round of large model calculation, and the second cache space is used to provide the cache space required for future large model calculation. A decision queue is determined; the decision queue is used to acquire a portion of the first cache space and prefetch the second cache space when forming a batch processing queue; the decision queue includes synchronous exchange instructions and asynchronous exchange instructions. The decision queue is sent to the first executor; the grouped batch processing queue is sent to the second executor. In the first executor: before the current round of large model calculation begins, the synchronous exchange instruction is executed to swap the requests in the candidate inference request queue into the allocated first cache space, so that the scheduler obtains the grouped batch processing queue; during the current round of large model calculation, the asynchronous exchange instruction is executed to swap the requests in the candidate inference request queue into the allocated second cache space; While the first executor executes the asynchronous exchange instruction, the scheduler calls the kernel function for large model computation; In the second executor: in response to the kernel function of the large model inference called by the scheduler, the current round of large model calculation is performed according to the assembled batch queue, and the result of the current round of calculation is stored in the first cache space.

2. The method according to claim 1, characterized in that, The process of determining the candidate inference request queue includes: Multiple pending inference requests are placed into the candidate inference request queue in order of priority; Traverse the candidate inference request queue and determine the status of each inference request in the candidate inference request queue. The status includes "waiting for computation", "being swapped out", "swapped out", "being swapped in", and "swapped in". The status is used to indicate the current processing status of the inference request.

3. The method according to claim 1, characterized in that, The process of creating a batch processing queue based on the candidate inference request queue includes: The maximum allowed batch size is calculated based on the storage space on the high-bandwidth memory and the large model, and the buffer size of the batch processing queue is determined. The buffer size of the batch processing queue is less than or equal to the maximum batch size. Inference requests that occupy cache space that meets the cache size requirement are removed from the candidate inference request queue and added to the batch processing queue in a first-in-first-out order. The cache space occupied by the inference request that meets the cache size requirement should be less than or equal to the cache size of the batch processing queue.

4. The method according to claim 2 or 3, characterized in that, The batch processing queue based on the candidate inference request queue also includes: Add the inference requests that meet the status requirements from the candidate inference request queue to the batch processing queue; the inference requests that meet the status requirements include inference requests with the status of "waiting for computation" or "already swapped in".

5. The method according to any one of claims 1-4, characterized in that, The allocation of the first cache space and the second cache space based on the storage space occupancy on the high-bandwidth memory (HBM) includes: Determine whether the space occupancy of the high-bandwidth memory has reached a saturation threshold; If the space occupancy of the high-bandwidth memory has not reached the saturation threshold, a first cache space is allocated on the high-bandwidth memory. The size of the first cache space is matched with the cache space required for the current large model calculation and the space occupied by the inference requests in the corresponding batch processing queue.

6. The method according to claim 5, characterized in that, The method of allocating the first cache space and the second cache space according to the storage space occupancy on the high-bandwidth memory (HBM) further includes: When the space occupancy of the high-bandwidth memory reaches a saturation threshold, destination space is allocated to the preemptible inference requests in the candidate inference request queue; the preemptible inference requests include the lowest priority inference requests in the candidate inference request queue; the destination space is CPU memory space.

7. The method according to any one of claims 1-6, characterized in that, The future large model computation includes the next round of large model computation. The allocation of the first and second cache spaces based on the storage space occupancy on the high-bandwidth memory (HBM) includes: Determine whether the space occupancy of the high-bandwidth memory has reached the saturation threshold based on the first cache space; If the space occupancy of the high-bandwidth memory has not reached the saturation threshold, a second cache space is allocated on the high-bandwidth memory. The second cache space provides cache space that can be swapped in for inference requests of the batch processing queue required for future large model computations.

8. The method according to claim 7, characterized in that, The pending inference request queue includes inference requests with a status of "swapped out". A second cache space is allocated on the high-bandwidth memory, including: The second cache space is allocated for the "swapped out" inference request, which is the inference request that has been swapped out to CPU memory.

9. The method according to any one of claims 1-8, characterized in that, The asynchronous exchange instruction includes an asynchronous swap-in instruction, the synchronous exchange instruction includes a synchronous swap-in instruction, and the decision queue is used to obtain a portion of the first cache space and prefetch the second cache space when forming a batch processing queue, including: When the space occupancy of the high-bandwidth memory has not reached the saturation threshold, the decision queue includes asynchronous swap-in instructions and synchronous swap-in instructions; execution requires waiting for the instructions to complete; the number of asynchronous swap-in instructions is one or more. The decision queue is used to obtain part or all of the first cache space when forming a batch processing queue through the synchronous swap-in instruction, and to prefetch the second cache space through the asynchronous swap-in instruction.

10. The method according to claim 9, characterized in that, The decision queue further includes a first temporary queue for storing the synchronization swap-in instruction. The determination decision queue includes: Before the current round of large model calculation begins, the synchronization swap instruction in the first temporary queue is added to the decision queue; During the current round of large model calculation, the asynchronous swap-in instruction is added to the decision queue, and the number of the asynchronous swap-in instructions is one or more. Before the current round of large model calculation begins, the decision queue acquires a portion of the first cache space in the batch processing queue through the synchronous swap-in instruction; during the current round of large model calculation, the second cache space is prefetched through one or more asynchronous swap-in instructions.

11. The method according to any one of claims 1-8, characterized in that, The asynchronous exchange instruction includes an asynchronous swap-out instruction, the synchronous exchange instruction includes a synchronous swap-out instruction, and the decision queue is used to obtain a portion of the first cache space and prefetch the second cache space when forming a batch processing queue, including: When the space occupancy of the high-bandwidth memory reaches a saturation threshold, the decision queue is determined to include asynchronous swap-out instructions and synchronous swap-out instructions; the number of asynchronous swap-out instructions is one or more; before the current round of large model calculation begins, the decision queue uses the synchronous swap-out instructions to swap out the inference requests in the candidate inference request queue that are in the "swapping out" state from the high-bandwidth memory to the destination space; during the current round of large model calculation, the one or more asynchronous swap-out instructions are used to pre-swap out the inference requests that are to be preempted from the high-bandwidth memory in the candidate inference request queue; thus releasing the space occupancy of the high-bandwidth memory to restore the state that has not reached the saturation threshold.

12. The method according to claim 11, characterized in that, The decision queue further includes a second temporary queue for storing the synchronization switchout instruction. The determination decision queue includes: Before the current round of large model calculation begins, the synchronization swap instruction in the second temporary queue is added to the decision queue; During the current round of large model calculation, one or more of the asynchronous swap-out instructions are added to the decision queue; Before the current round of large model computation begins, the decision queue releases the cache space required for the current round of large model computation by means of the synchronous swap-out instruction when batch processing the queue; during the current round of large model computation, the cache space required for the next round of large model computation is pre-released by means of one or more asynchronous swap-out instructions.

13. The method according to any one of claims 1-12, characterized in that, Sending the decision queue to the first executor and sending the assembled batch processing queue to the second executor includes: Before the current round of large model calculation begins, the synchronous swap-in instruction is issued to the first executor; after the swap-in is completed, the assembled batch processing queue is sent to the second executor; During the current round of large model calculation, one or more asynchronous swap-in instructions are sent to the first executor.

14. The method according to any one of claims 1-12, characterized in that, Sending the decision queue to the first executor and sending the grouped batch processing queue to the second executor includes: Before the current round of large model calculation begins, the synchronization switchout instruction is sent to the first executor; During the current round of large model calculation, one or more asynchronous swap-out instructions are issued to the first executor.

15. The method according to any one of claims 1-14, characterized in that, The step of executing the synchronization exchange instruction before the current round of large model computation begins, swapping requests from the candidate inference request queue into the allocated first cache space, so that the scheduler obtains the assembled batch processing queue, includes: The first executor executes the synchronous swap-in instruction, swapping the inference request with the status "swapping in" into the allocated first cache space, and updates the status to "swapped in" after execution is completed; Execute the asynchronous swap-in instruction to put the inference request with the status "swapped out" into the allocated second cache space, and update the status of the inference request with the status "swapped out" to "swapping in".

16. The method according to claim 1 or 15, characterized in that, The method further includes: the scheduler determining the grouped batch processing queue based on the inference request with the status of "swapped in" or "waiting to be computed".

17. The method according to any one of claims 1-14, characterized in that, The step of executing the synchronization exchange instruction before the current round of large model computation begins, swapping requests from the candidate inference request queue into the allocated first cache space, so that the scheduler obtains the assembled batch processing queue, includes: Before the current round of large model calculation begins, the first executor executes the synchronous swap-out instruction to swap out all inference requests with the status "swapping out" from the high-bandwidth memory to the allocated destination space. After the execution is completed, the status is updated to "swapped out". During the current round of large model computation, an asynchronous swap-out instruction is executed to swap out the preempted inference request from the high-bandwidth memory and update the status of the preempted inference request to "swapping out".

18. The method according to any one of claims 1-17, characterized in that, The method further includes: After the current round of large model computation is completed, the scheduler iterates to determine the candidate inference request queues required for the next round of large model computation; and forms the batch processing queues required for the next round of large model computation based on the candidate inference request queues required for the next round of large model computation. The second cache space is used when assembling the batch queue required for the next round of large model computation; The decision queue also includes a synchronization wait instruction; the synchronization wait instruction is used to wait for the return information of the current round of large model calculation.

19. A large model inference cache management device, used to perform the method as described in any one of claims 1-18, characterized in that, The device includes at least: The scheduling module is used to determine candidate inference request queues; assemble batch processing queues based on the candidate inference request queues, the batch processing queues being used for the current round of large model computation; allocate a first cache space and a second cache space based on the storage space occupancy on the high-bandwidth memory (HBM), the first cache space being the cache space required for the current round of large model computation, and the second cache space being used to provide the cache space required for future large model computations; determine a decision queue; the decision queue is used to acquire a portion of the first cache space and prefetch the second cache space when assembling the batch processing queues; the decision queue includes synchronous exchange instructions and asynchronous exchange instructions; send the decision queue to a first execution module; send the assembled batch processing queues to a second execution module; while the first execution module executes the asynchronous exchange instructions, the scheduler module calls the kernel function for large model computation; and The first execution module is configured to receive the batch processing queue and the decision queue; execute the asynchronous exchange instruction and the synchronous exchange instruction round by round according to the decision queue, swap the batch processing queue into the allocated cache space, and obtain a second batch processing queue after the swap is completed. The second batch processing queue is the request queue required for the current round of large model computation; while executing the asynchronous exchange instruction, the kernel function of large model inference is called; the second execution module is configured to respond to the kernel function of large model inference called by the scheduler, execute the current round of large model computation according to the assembled batch processing queue, and store the computation result of the current round in the first cache space.

20. A computing device, characterized in that, include: At least one memory for storing programs; At least one processor is configured to execute a program stored in the memory, wherein when the program stored in the memory is executed, the processor is configured to perform the method as described in any one of claims 1-18.

21. A computing device cluster, characterized in that, It includes at least one computing device, each computing device including a processor and memory; The processor of the at least one computing device is configured to execute a request stored in the memory of the at least one computing device, such that the cluster of computing devices performs the method as described in any one of claims 1-18.

22. A computer-readable storage medium storing a computer program that, when run on a processor, causes the processor to perform the method as described in any one of claims 1-18.

23. A computer program product, characterized in that, When the computer program product is run on a processor, it causes the processor to perform the method as described in any one of claims 1-18.

Images