Large model inference service system and inter-stage resource adjustment method

Abstract

The application relates to the technical field of large models, in particular to a large model inference service system and a resource adjustment method between stages, wherein the system comprises: a pre-filling processor and a decoding processor; a computing resource controller that controls the computing resources of the pre-filling processor and the decoding processor based on a multi-process service, the multi-process service allowing a target number of streaming multiprocessors to be specified to a pre-filling process and a decoding process, wherein the pre-filling process represents an entity of the pre-filling processor, and the decoding process represents an entity of the decoding processor; and a unified storage manager that is used for managing the storage utilization of the system and the storage access of the pre-filling processor and the decoding processor to large model weights and KV caches. Thus, the problems of delay interference between pre-filling and decoding and storage inefficiency in the related art are solved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of large model, and particularly relates to a large model inference service system and a stage resource adjustment method. BACKGROUND

[0002] With the booming development of large model related applications, more and more users begin to use large models for content generation. The use mode is that the user sends a request to a large model service provider, and then waits for the return result. However, due to huge computing and memory access requirements, LLM (Large Language Model) inference service often introduces considerable delay. Unlike traditional requests, the delay of LLM requests consists of two different parts, because each LLM request involves two processing stages: a prefill stage and a decoding stage. The prefill stage processes the input text and generates the first output token (Token), and the decoding stage generates the output in a self-recursive manner, one token at a time. Therefore, the first part is the time from when the user sends the request to when the TTFT (Time-to-First-Token) is received, and the main part is the execution time of the prefill stage, and it also includes the time for the request to wait; the second part is the time from when the user receives the TPOT (Time-per-Output-Token) after the first output token, and the TPOT records the average delay of each token in the decoding stage. Therefore, under the constraints of TTFT and TPOT, how to maximize the system throughput is very important to reduce the cost of inference service.

[0003] Request batching is an effective method to improve system throughput, and the initial work chooses to place requests of different stages on the same GPU for calculation-“fusion design”. These works either prioritize the calculation of prefill requests or prioritize the calculation of decoding requests. The former is beneficial to TTFT but will cause the increase of TPOT, and vice versa. This trade-off is called “latency interference” between the two stages.

[0004] In order to reduce such interference, the related art divides the input of the prefill request into smaller prefill request segments, and then schedules the prefill request segments and the decoding request together into the same batch for calculation. The above work cannot completely avoid the interference between TTFT and TPOT, and it is very challenging to optimize the system throughput, especially under the condition of having SLO constraints on TTFT and TPOT.

[0005] To this end, the related art proposes a "separate design" to place the prefill request and the decoding request on different GPUs to eliminate the interference between the TTFT and the TPOT. In the separate system, one part of the GPU serves as a prefill processor to process the prefill request, and another GPU serves as a decoding processor to process the decoding request. However, although the storage separation design solves the problem of latency interference, it still has the following problems: storage imbalance, KV cache transmission overhead, difficulty in resource adjustment between the prefill and decoding processors, and weight copy. SUMMARY

[0006] The present application provides a large model inference service system and a stage resource adjustment method to solve the problems of latency interference between prefill and decoding and storage inefficiency in the related art.

[0007] The first aspect of the present application provides a large model inference service system, comprising: a prefill processor and a decoding processor; a computing resource controller for controlling the computing resources of the prefill processor and the decoding processor based on a multi-process service, wherein the multi-process service allows a target number of streaming multi-processors to be assigned to a prefill process and a decoding process, and the prefill process represents an entity of the prefill processor, and the decoding process represents an entity of the decoding processor; and a unified storage manager for managing the storage utilization of the system and the storage access of the prefill processor and the decoding processor to the large model weight and the KV cache.

[0008] Optionally, the computing resource controller saves the large model weight and the KV cache through the target process.

[0009] Optionally, the target process shares the corresponding pointer to the prefill process and the decoding process through inter-process communication.

[0010] Optionally, the computing resource controller asynchronously switches the processes.

[0011] Optionally, the asynchronous switching of the processes comprises: detecting whether the process of the current iteration is completed; if the process of the current iteration is not completed, waiting for the process of the current iteration to be completed; and if the process of the current iteration is completed, killing the process of the current iteration and switching to a new process.

[0012] Optionally, the storage utilization is locked before the update step is completed.

[0013] The second aspect embodiment of the application provides a method for adjusting resources between stages with service level objective awareness, which is used for resource adjustment of the large model inference service system in the above-mentioned embodiments, and includes the following steps: obtaining a first token time of a service level objective and an average interval time of each output token except the first output token; if the first token time and the average interval time of each output token except the first output token do not meet the service level objective, returning a percentage of stream multiprocessors visible to each of the current pre-padding process and the decoding process; if the first token time or the average interval time of each output token except the first output token meets the service level objective, determining a process that needs to be adjusted, and iteratively increasing a percentage of stream multiprocessors visible to the process that needs to be adjusted, and returning a percentage of stream multiprocessors visible to the pre-padding process or the decoding process after adjustment when the first token time and the average interval time of each output token except the first output token meet the service level objective.

[0014] Optionally, the iteratively increasing the percentage of stream multiprocessors visible to the process that needs to be adjusted includes: obtaining a maximum number of iterations and a step length of each iteration; and controlling the iterative increasing process according to the maximum number of iterations and the step length of each iteration.

[0015] Optionally, the determining the process that needs to be adjusted includes: if the first token time does not meet the service level objective, the process that needs to be adjusted is the pre-padding process; and if the average interval time of each output token except the first output token does not meet the service level objective, the process that needs to be adjusted is the decoding process.

[0016] Optionally, in the iterative increasing process, the method further includes: obtaining a first correspondence relationship between the pre-padding process and the first token time, and estimating the first token time in the iterative increasing process according to the first correspondence relationship; and obtaining a second correspondence relationship between the decoding process and the average interval time of each output token except the first output token, and estimating the average interval of each output token except the first output token in the iterative increasing process according to the second correspondence relationship.

[0017] Optionally, the first correspondence relationship is:

[0018]

[0019] The second correspondence relationship is:

[0020]

[0021] wherein, TTFT x is the first token time, TPOT yFor the average time interval of each output token except the first output token, a1, b1, λ, a2, b2 are parameters, and x, y are ratios of the stream multiprocessor.

[0022] Thus, the present application includes the following beneficial effects:

[0023] The embodiment of the present application constructs a large service model oriented inference service system, specifically including a pre-population processor, a decoding processor, a computing resource controller, and a unified storage manager. The computing resource controller is added to control the division of computing resources between the pre-population processor and the decoding processor. The unified storage manager is added to manage the storage utilization of the large service model oriented inference service system, and the storage access of the pre-population processor and the decoding processor to the large model weight and KV cache. Thus, the computing resource controller realizes computing separation, effectively reduces the delay interference between pre-population and decoding, and makes the system efficiently process system requests. At the same time, the unified storage manager realizes storage fusion, ensures the effective use of storage resources, and reduces resource waste. Thus, the technical problems of delay interference between pre-population and decoding, storage inefficiency, and the like in related technologies are solved.

[0024] Additional aspects and advantages of the present application will be in part apparent and in part pointed out hereinafter. BRIEF DESCRIPTION OF DRAWINGS

[0025] The above and / or additional aspects and advantages of the present application will become apparent and be readily appreciated from the following description, including the accompanying drawings, wherein:

[0026] Figure 1 A schematic diagram of computing fusion and storage fusion in related technologies;

[0027] Figure 2 A schematic diagram of computing separation and storage separation in related technologies;

[0028] Figure 3 A schematic diagram of computing separation and storage fusion provided by the embodiment of the present application;

[0029] Figure 4 A schematic diagram of a large model oriented inference service system provided according to the embodiment of the present application;

[0030] Figure 5 A schematic diagram of request processing when process switching in related technologies;

[0031] Figure 6 A schematic diagram of request processing when delayed switching after joining a target process according to the embodiment of the present application;

[0032] Figure 7A schematic diagram of allocating space atomization according to an embodiment of the present application is provided.

[0033] Figure 8 A specific structure diagram of a large model inference service system according to an embodiment of the present application is provided.

[0034] Figure 9 A flowchart of a service level objective perception inter-stage resource adjustment method according to an embodiment of the present application is provided. DETAILED DESCRIPTION

[0035] Embodiments of the present application are described in detail below, examples of which are shown in the accompanying drawings, wherein the same or similar reference numerals represent the same or similar elements or elements having the same or similar functions throughout. The embodiments described below by reference to the accompanying drawings are exemplary and are intended to explain the present application, and cannot be understood as limiting the present application.

[0036] Before describing the scheme of the present application, the method for improving the large model inference service system in the related art is introduced.

[0037] I. Computing fusion storage fusion design, such as Figure 1 shown.

[0038] Prioritize pre-populate requests or prioritize decode requests. The former is beneficial to TTFT but can cause TPOT to rise, and vice versa. This trade-off is called "latency interference" between the two stages

[0039] II. Computing classification storage separation design, such as Figure 2 shown.

[0040] Place pre-populate requests and decode requests on different GPUs to eliminate interference between TTFT and TPOT. In a separate system, a portion of the GPUs serve as pre-populate processors to process pre-populate requests, and the other GPUs serve as decode processors to process decode requests.

[0041] However, the storage separation has the following disadvantages: 1) storage imbalance. The pre-filling processor only generates a KV cache part corresponding to the input length. However, the decoding processor needs to store the entire KV cache corresponding to the complete sequence length including the output length. In the case of a large output length, the storage imbalance will become more serious. Therefore, the storage space required by the decoding processor is significantly larger than that of the pre-filling processor. 2) KV cache transmission overhead. Since the pre-filling stage and the decoding stage are deployed on different GPUs, the KV cache needs to be transmitted from the pre-filling processor to the decoding processor. There will be significant communication delay on low-end GPUs. Even for high-end GPUs equipped with NVLink, the transmission time will be counted into the TPOT, affecting the TPOT. 3) Difficulty in adjusting resources between the pre-filling and decoding processors. The workloads of pre-filling and decoding requests will change as the service process proceeds. However, the separated design can only make coarse-grained adjustments in GPU units. In addition, since the KV cache is mainly located on the decoding processor, special designs are often required to avoid the cost of KV cache transmission when adjusting, thereby introducing additional overhead. 4) Weight copy. Both the pre-filling processor and the decoding processor need to accommodate the entire model weight. This means that the separated design requires twice the number of GPUs to deploy the same model.

[0042] To this end, the present application provides a large model inference service system, as shown in Figure 3 , which decouples the features of calculation and storage, adopts a calculation separation and storage fusion design, retains the main advantages of storage fusion in the fusion type and calculation separation in the separated type, so that the large model can maintain lower latency under high request rate, thereby serving more requests in the same time.

[0043] Specifically, Figure 4 The large model inference service system provided by the embodiments of the present application is shown in the schematic diagram.

[0044] As shown in Figure 4 , the large model inference service system 10 includes a pre-filling processor 11, a decoding processor 12, a computing resource controller 13, and a unified storage manager 14.

[0045] The computing resource controller 13 controls the computing resources of the pre-filling processor 11 and the decoding processor 12 based on a multi-process service, which allows a target number of streaming multi-processors to be assigned to a pre-filling process and a decoding process. The pre-filling process represents an entity of the pre-filling processor, and the decoding process represents an entity of the decoding processor. The unified storage manager 14 is used to manage the storage utilization of the system and the storage access of the pre-filling processor and the decoding processor to the large model weight and the KV cache.

[0046] It can be understood that the embodiment of the application constructs a large service model reasoning service system 10, specifically including a pre-filling processor 11, a decoding processor 12, a computing resource controller 13 and a unified storage manager 14, by increasing the computing resource controller 13 to realize the control of the computing resource division between the pre-filling processor 11 and the decoding processor 12, and by increasing the unified storage manager 14, which is used for managing the storage utilization of the large service model reasoning service system 10 and the storage access of the pre-filling processor 11 and the decoding processor 12 to the large model weight and KV cache, so as to realize the computing separation through the computing resource controller 13, effectively reduce the delay interference between pre-filling and decoding, make the system can efficiently process system requests, and at the same time realize the storage fusion through the unified storage manager 14, ensure the effective utilization of storage resources, reduce the waste of resources, so that the system can maintain lower delay under high request rate, thereby serving more requests in the same time, and improving the throughput of the system.

[0047] Specifically, the embodiment of the application can realize the computing resource control of the pre-filling processor and the decoding processor based on the multi-process service MPS of the computing resource controller 13. The multi-process service allows to specify a target number of SMs (Streaming Multi-processor) to a certain pre-filling process and decoding, so as to realize the computing resource division at the SM level. The pre-filling process represents the entity of the pre-filling processor, and the decoder process represents the entity of the decoding processor.

[0048] In the embodiment of the application, the computing resource controller 13 saves the large model weight and KV cache through the target process.

[0049] The target process can also be understood as a resident process.

[0050] Since the entities of the pre-filling processor and the decoding processor in the system are a process, this pair of processes receives a computing resource partition parameter (x, y) when initializing, where x and y respectively represent the percentage of the number of SMs visible to the pre-filling and decoding processes after being scheduled by MPS. And since each processor is carried on a process, MPS does not support adjusting (x, y) of the existing process, so a pair of processes can only be assigned a set of (x, y). This means that adjusting (x, y) requires re-calling the MPS interface, thereby bringing the overhead of process switching. Specifically, switching needs to wait for the pre-filling processor and the decoding processor to complete the iteration they are running and store the generated tokens before terminating the two processes. After the original processes are terminated, the MPS interface is used to apply new (x, y) to two new processes, which are marked as new pre-filling processor and decoding processor. Based on the existing KV cache and saved tokens, the new processor continues to run.

[0051] In this process, the overhead comes from two aspects. First, as shown in Figure 5 Figure 2, the initialization of two MPS processes includes the following steps: 1) load the weights to the new pre-filling process and share the weights with the new decoding process through IPC (and vice versa); 2) copy the KV cache to the new decoding process; 3) initialize the inference engine for both new processes. The above switching steps cause the newly arrived requests to be blocked during the switching process until the new pair of processes are ready. When serving Llama3-8B at 10 requests per second, the typical switching overhead is 22.8 seconds, causing 228 requests to be blocked. Second, the switching timing is also important. Since the pre-filling process and the decoding process have different processing progress, a synchronization operation needs to be inserted to block the process that finishes the current iteration earlier.

[0052] In short, to solve the first overhead, i.e. the time and resource overhead required to restart the pre-filling process and the decoding process when adjusting the allocation of computing resources, the embodiments of the present application introduce a target process in the system, as shown in Figure 6 The computing resource controller saves the large model weights and the KV cache through the target process, avoiding the overhead of repeatedly loading the weights and copying the KV cache.

[0053] In the embodiments of the present application, the target process shares the corresponding pointers to the pre-filling process and the decoding process through inter-process communication.

[0054] It can be understood that the target process of the embodiments of the present application shares the corresponding pointers to the pre-filling process and the decoding process through inter-process communication, so that the pre-filling and decoding processes can access the weights and KV cache through the pointers, reducing the delay of data transmission.

[0055] In the embodiments of the present application, the computing resource controller 13 switches the processes asynchronously.

[0056] To solve the second overhead, i.e. when adjusting the allocation of computing resources, due to the different processing progress of the pre-filling process and the decoding process, a synchronization operation needs to be inserted to block the process that finishes the current iteration earlier, the computing resource controller 13 of the embodiments of the present application uses asynchronous process switching to reduce the synchronization overhead caused by different processing progress. The specific asynchronous switching process is as follows.

[0057] In the embodiments of the present application, the asynchronous switching process includes: detecting whether the process of the current iteration is completed; if the process of the current iteration is not completed, waiting for the process of the current iteration to be completed; if the process of the current iteration is completed, killing the process that completes the current iteration and switching to a new process.

[0058] It can be understood that the asynchronous switching schedule of the embodiment of the present application includes: detecting whether the process of the current iteration (the pre-filling process and the decoding process) is completed, if the process of the current iteration is not completed, waiting for the process of the current iteration to be completed, if the process of the current iteration is completed, killing the process of the current iteration and switching to a new process, thereby reducing the blocking time of the system, and the MPS allows the resource percentage of all running processes to be greater than 100%, which indicates that during the switching process, although some processes are terminated, other processes can still continue to compete for resources, ensuring efficient use of resources. Through asynchronous switching, resource adjustment can be completed in a short time, avoiding resource idling and improving resource utilization.

[0059] In the embodiment of the present application, the storage utilization is locked before the update step is completed.

[0060] It should be noted that the storage manager mainly focuses on two types of storage access, namely reading model weights and accessing KV cache. The unified storage manager manages model weights in a relatively direct manner because model weights are read-only. However, the management of KV cache access is relatively complex. The system uses the same KV cache paging storage mechanism as vLLM to access KV cache through a block table index. Therefore, once the block table index is determined, the KV cache can be accessed without conflict. However, when the pre-filling processor and the decoding processor open up KV cache space in an asynchronous manner, conflicts occur.

[0061] The pre-filling processor opens up space to store the generated KV cache when single-layer computation is completed, and the decoding processor opens up KV cache in PagedAttention computation of each layer. The asynchronous working manner of the two processors may cause a read-after-write (WAR) conflict in storage management. As shown in Figure 7 The storage space allocation of the KV cache contains three steps: 1) query the storage utilization to see if there is a free block; 2) get a block for KV cache storage; 3) update the storage utilization. When one processor updates the utilization immediately after another processor queries the utilization, a WAR conflict occurs. As a result, the process that performs the query operation updates the wrong storage utilization according to the queried value.

[0062] Therefore, the unified storage manager of the embodiment of the present application uses an atomic KV cache block allocation manner, that is, the storage utilization is locked before the update step is completed. This atomic allocation manner can ensure the correct storage utilization of the two asynchronous processes.

[0063] The large model inference service system of the present application is described below through a specific embodiment, that is, a large model inference service system with the characteristics of computing separation and storage fusion, so that the large model can maintain lower latency under high request rate, thereby serving more requests in the same time, and the service request rate can be increased by 1.55-1.72 times, and the specific system structure is as shown in Figure 8

[0064] Among them, the pre-filling and decoding processors process the pre-filling and decoding requests respectively, and each processor maintains its own waiting request and running request queue. The system realizes computing separation through a computing resource controller, which determines the division of computing resources between the two processors and makes real-time adjustments as needed. For the storage part, the system manages the storage utilization of the system through a unified storage manager, which is also responsible for the storage access of the two processors to the weight and KV cache.

[0065] I. Computing resource controller

[0066] The computing resource controller realizes computing separation based on the CUDA application interface of MPS. MPS allows a certain number of SMs to be designated to a process, thereby realizing computing resource partitioning at the SM level. Specifically, the entities of the pre-filling processor and the decoding processor in the system are both a process, and this pair of processes receives the computing resource partitioning parameters (x, y) at initialization, where x and y respectively represent the percentage of the number of SMs visible to the pre-filling and decoding processes after scheduling by MPS.

[0067] Since each processor is carried on a process, MPS does not support adjusting (x, y) of an existing process, so a pair of processes can only be assigned a set of (x, y). This means that adjusting (x, y) requires re-invoking the MPS interface, thereby incurring the overhead of process switching. Specifically, switching needs to wait for the pre-filling processor and the decoding processor to complete their running iterations and store the generated tokens before terminating the two processes. After the original processes are terminated, a new (x, y) is applied to the two new processes using the MPS interface, which are marked as the new pre-filling processor and the decoding processor. Based on the existing KV cache and saved tokens, the new processors continue to run.

[0068] In this process, the overhead comes from two aspects. First, as shown in Figure 5 ​As shown, initializing two MPS processes includes the following steps: 1) loading weights into the new pre-filling process and sharing weights with the new decoding process via IPC (and vice versa); 2) copying the KV cache to the new decoding process; and 3) initializing the inference engine for the two new processes. These switching steps cause newly arriving requests to be blocked during the switching process until the new pair of processes is ready. When serving Llama3-8B at 10 requests per second, the typical switching overhead is 22.8 seconds, resulting in 228 blocked requests. Secondly, the timing of the switching is also important. Because the pre-filling and decoding processes have different processing speeds, synchronization operations need to be inserted to block the process that completed the current iteration earlier. An alternative solution is process refresh: that is, the slower process can be terminated, and computation can be re-executed on the new process.

[0069] like Figure 6 As shown, to address the first overhead, this application introduces a persistent process that continuously stores weights and key-value (KV) caches during service, preventing other processes from repeatedly loading weights and copying KV caches. Specifically, this persistent process shares corresponding pointers with the pre-filling and decoding processes via IPC, allowing them to access the weights and KV caches through these pointers. The storage space for the weights and KV caches is not released when the process stops, and new processes can quickly access the storage via IPC. Furthermore, the overhead can be further reduced by delaying the switchover point, i.e., switching only occurs when the new process is ready, masking the pause period during the switchover. Before the process switchover, arriving requests are still calculated on the old process using the old (x, y) parameters.

[0070] To address the second overhead, this application employs an asynchronous switching method. This involves first killing the process that has completed the current iteration and switching to a new process, then waiting for any processes that haven't yet completed their current iteration before performing the same operation. Because MPS allows all running processes to have a resource percentage greater than 100%, this method can handle the overhead of asynchronous switching, as each process will compete for resources for a short period.

[0071] II. Unified Storage Manager

[0072] The storage manager primarily focuses on two types of storage access: reading model weights and accessing the key-value (KV) cache. The unified storage manager manages model weights relatively directly because they are read-only. However, managing KV cache access is more complex. The system uses the same KV cache paging storage mechanism as vLLM, accessing the KV cache through block table indexes. Therefore, once the block table index is determined, the KV cache can be accessed without conflict. However, conflicts can occur when the pre-fill processor and the decoding processor allocate KV cache space asynchronously.

[0073] The pre-filling processor opens up space to store the generated KV cache when the single-layer computation is completed, and the decoding processor opens up the KV cache in the PagedAttention computation of each layer. The asynchronous working mode of the two processors can cause a read-after-write (WAR) conflict in storage management. As shown in Figure 7 the storage space allocation of the KV cache includes three steps: 1) query the storage utilization to see if there is a free block; 2) obtain a block for KV cache storage; and 3) update the storage utilization. A WAR conflict occurs when one processor updates the utilization rate immediately after the other processor queries the utilization rate. As a result, the process that performs the query operation updates the wrong storage utilization according to the queried value. To solve this challenge, the unified storage manager uses an atomic KV cache block allocation method, that is, the storage utilization is locked before the update step is completed. This atomic allocation method can ensure the correct storage utilization of the two asynchronous processes.

[0074] According to the large model inference service system proposed in the embodiments of the present application, a computing resource controller is added to control the division of computing resources between the pre-filling processor and the decoding processor, and a unified storage manager is added to manage the storage utilization of the large service model inference service system and the storage access of the pre-filling processor and the decoding processor to the large model weight and the KV cache. Thus, the computing resource controller realizes computing separation, effectively reduces the delay interference between pre-filling and decoding, enables the system to efficiently process system requests, and simultaneously realizes storage fusion through the unified storage manager, ensures the effective use of storage resources, reduces resource waste, and enables the system to maintain lower latency under a high request rate, thereby serving more requests in the same time and improving the throughput of the system.

[0075] Next, the service level target aware inter-phase resource adjustment method proposed in the embodiments of the present application is described with reference to the accompanying drawings.

[0076] Figure 9 is a flowchart of the service level target aware inter-phase resource adjustment method of the embodiments of the present application.

[0077] As shown in Figure 9 , the service level target aware inter-phase resource adjustment method is used for resource adjustment of the large model inference service system described above and includes the following steps:

[0078] In step S101, the first token time of the service level target and the average interval time of each output token except the first output token are obtained.

[0079] It can be understood that the embodiment of the application can obtain the TTFT (Time-to-First-Token) and TPOT (Time-per-Output-Token) of the SLO (Service Level Objective), and the TPOT records the average delay of each token in the decoding stage, so as to subsequently adjust the resources.

[0080] In step S102, if neither the first token time nor the average interval time of each output token except the first output token meets the service level objective, the percentage of the stream multiprocessors visible to the current pre-filling process and decoding process is returned.

[0081] It can be understood that the embodiment of the application can return the percentage of the stream multiprocessors visible to the current pre-filling process and process when neither the TTFT nor the TPOT meets the SLO, because there is no adjustment space at this time.

[0082] In step S103, if the first token time or the average interval time of each output token except the first output token meets the service level objective, the process that needs to be adjusted is determined, and the percentage of the stream multiprocessors visible to the process that needs to be adjusted is iteratively increased, and the percentage of the stream multiprocessors visible to the pre-filling process or decoding process after adjustment is returned when the first token time and the average interval time of each output token except the first output token meet the service level objective.

[0083] It can be understood that the embodiment of the application can determine the process that needs to be adjusted when either the TTFT or the TPOT does not meet its SLO, iteratively increase the percentage of the stream multiprocessors visible to the process that needs to be adjusted, and return the percentage of the stream multiprocessors visible to the pre-filling process or decoding process after adjustment when the first token time and the average interval time of each output token except the first output token meet the service level objective, so that the resource allocation of the pre-filling and decoding processors can be dynamically adjusted according to the actual performance of the large model inference service system, and the TTFT and TPOT can meet the SLO.

[0084] In the embodiment of the application, determining the process that needs to be adjusted includes: if the first token time does not meet the service level objective, the process that needs to be adjusted is the pre-filling process; and if the average interval time of each output token except the first output token does not meet the service level objective, the process that needs to be adjusted is the decoding process.

[0085] It can be understood that the embodiments of the present application can accurately determine the process that needs to be adjusted according to the specific performance bottleneck. If the TTFT does not meet the SLO, the pre-filling process is adjusted. If the TPOT does not meet the SLO, the decoding process is adjusted.

[0086] In the embodiments of the present application, the iteration increases the percentage of the number of stream multiprocessors that the process that needs to be adjusted can be seen, comprising: obtaining the maximum number of iterations and the step length of each iteration increase; and controlling the iteration increase process according to the maximum number of iterations and the step length of each iteration increase.

[0087] The maximum number of iterations and the step length of each iteration increase can be pre-set according to specific conditions.

[0088] It can be understood that the embodiments of the present application can control the iteration increase process according to the maximum number of iterations and the step length of each iteration increase, ensuring the controllability and stability of resource adjustment, and avoiding system instability caused by excessive adjustment.

[0089] In the embodiments of the present application, in the iteration increase process, it further comprises: obtaining a first correspondence relationship between the pre-filling process and the first token time, and estimating the first token time in the iteration increase process according to the first correspondence relationship; and obtaining a second correspondence relationship between the decoding process and the average interval time of each output token except the first output token, and estimating the average interval of each output token except the first output token in the iteration increase process according to the second correspondence relationship.

[0090] It can be understood that the embodiments of the present application can more accurately predict and adjust resource allocation by establishing the first correspondence relationship between the pre-filling process and the TTFT, and the second correspondence relationship between the decoding process and the TPOT, further optimizing the system performance.

[0091] In the embodiments of the present application, the first correspondence relationship is:

[0092]

[0093] The second correspondence relationship is:

[0094]

[0095] The TTFT x is the first token time, and the TPOT y is the average interval time of each output token except the first output token, and a1, b1, λ, a2, b2 are parameters, and x, y are the ratio of stream multiprocessors.

[0096] The service level objective aware inter-stage resource adjustment method (also known as adjustment algorithm) of the present application will be described below through a specific embodiment, which is specifically:

[0097] The allocation of computing resources, i.e., (x, y), is adjusted periodically according to winndow_size. The system collects TTFT and TPOT recorded in the period window to update the TTFT / TPOT estimation model. When neither TTFT nor TPOT meets the SLO, the algorithm directly returns the current (x, y) because there is no adjustment space in this case. Otherwise, the algorithm first determines which one of TTFT or TPOT does not meet its SLO and iteratively increases the visible SM ratio of the corresponding processor. After each increase, the algorithm evaluates whether the updated ratio meets the SLO, and if not, it continues to iteratively increase until it meets. This process has two parameters, i.e., the maximum number of iterations max_step and the step size step_size of each iteration.

[0098] The algorithm needs to model the relationship between TTFT and TPOT according to the accessible SM ratios x and y, respectively, which is used in the corresponding TTFT and TPOT estimation model. Since the two processors process requests asynchronously, TTFT and TPOT can be modeled separately. The TTFT of a single request consists of two parts of latency, i.e., the waiting latency and the pre-filling latency. For TPOT, since the single iteration time is short and the number of concurrent requests is high, it can be assumed that the request is executed immediately after arriving at the decoding processor, because the waiting latency can not be considered when calculating TPOT. The more SM the process can use, the more computing resources it can utilize in parallel, and the processing rate will grow proportionally. Therefore, the processing latency can be modeled as:

[0099]

[0100] where l x represents the processing latency of the pre-filling and decoding stages under the SM ratio x. For TTFT, the M / M / 1 queuing model is applied to model the sum of the waiting and pre-filling latency as follows:

[0101]

[0102] where r and μ x represent the given input request rate and the pre-filling processing rate under the SM ratio x, respectively. w is the sum of the waiting latency and the processing latency. Bringing μ x = 1 / l x ∝ x into the above formula, it is further written as:

[0103]

[0104] Therefore, the relationship between TTFT and x, and TPOT and y is represented as:

[0105]

[0106] The parameters a1, b1, l, a2, b2 are learnable and can be computed from real-time observations of TTFT and TPOT. The constant terms b1, b2 serve to characterize the constant overhead in the latency. Given (x, y), TTFT and TPOT can be estimated using the above equations to quantify the impact of adjusting (x, y) during service.

[0107] It should be noted that the foregoing explanation and description of the embodiment of the large model inference service system also applies to the service level objective aware inter-phase resource adjustment method of the embodiment, which will not be described here.

[0108] The service level objective aware inter-phase resource adjustment method according to the embodiments of the present application can determine whether the service level objective is met based on the first token output time and the average time interval of each output token except the first output token, dynamically adjust the resource allocation of the pre-padding process and the decoding process, improve the resource utilization efficiency, and improve the system performance of the subsequent large model inference service system.

[0109] In the description of the present specification, the description of the terms "one embodiment", "some embodiments", "an example", "a specific example", or "some examples" and the like means that the specific features, structures, materials or characteristics described in connection with the embodiment or example are included in at least one embodiment or example of the present application. In the present specification, the illustrative description of the above terms is not necessarily directed to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described can be combined in any one or N embodiments or examples in a suitable manner. In addition, the person skilled in the art can combine and combine the different embodiments or examples described in the present specification and the features of the different embodiments or examples without contradiction.

[0110] In addition, the terms "first", "second" are only for descriptive purposes and cannot be understood as indicating or implying relative importance or implicitly indicating the number of indicated technical features. Therefore, the features defined with "first", "second" can explicitly or implicitly include at least one of the features. In the description of the present application, the meaning of "N" is at least two, for example, two, three, etc., unless otherwise specifically limited.

[0111] Any process or method described in a flowchart or otherwise described herein can be understood as representing code modules, segments, or portions of code that include one or more executable instructions for implementing the specified logical functions or steps, and the various embodiments of the application include alternative implementations of the described processes or methods, in which the order of steps can be changed, including the use of simultaneous steps or reverse order of steps, where necessary and / or desirable.

[0112] It should be understood that portions of the application can be implemented in hardware, software, firmware, or combinations thereof. In the above embodiments, steps or methods can be implemented in software or firmware that is stored in memory and executed by a suitable instruction execution system. As with the hardware, the software or firmware can be implemented using any of a number of programming languages, such as C, C++, Java, or the like, or combinations thereof, and as such, is not limited to any particular programming language. As such, the software or firmware can be implemented in a plurality of formats, including but not limited to source code, object code, bytecode, interpreted code, or the like.

[0113] Those of ordinary skill in the art will appreciate that the steps carried out by the methods described above can be carried out in whole or in part by a program instructing relevant hardware, and the program described above can be stored in a computer-readable storage medium. When the program is executed, it includes one or a combination of the steps of the method embodiments.

[0114] Although the embodiments of the application have been shown and described above, it should be understood that the above embodiments are exemplary, and are not to be interpreted as limiting the application, and those of ordinary skill in the art can make changes, modifications, replacements and variations to the above embodiments within the scope of the application.

Claims

1. A service system for large-scale model inference, characterized in that, include: Pre-fill processor and decoding processor; A computing resource controller implements computing resource control for the pre-filled processor and the decoding processor based on a multi-process service. This multi-process service allows a target number of streaming multiprocessors to be assigned to the pre-filled process and the decoding process, where the pre-filled process represents an entity of the pre-filled processor, and the decoding process represents an entity of the decoding processor. The computing resource controller stores large model weights and key-value caches through a target process, and the target process shares corresponding pointers with the pre-filled process and the decoding process through inter-process communication. The computing resource controller asynchronously switches processes. A unified storage manager is used to manage the system's storage utilization and the storage access of the pre-filled processor and the decoder to the large model weights and KV cache.

2. The large-model inference service system according to claim 1, characterized in that, The asynchronous switching process includes: Check whether the current iteration process has completed; If the current iteration process has not been completed, then wait for the current iteration process to complete; If the process of the current iteration is completed, the process that has completed the current iteration will be killed and the process will be switched to a new process.

3. The large-model inference service system according to claim 1, characterized in that, The storage utilization rate is locked before the update step is completed.

4. A method for inter-stage resource adjustment based on service level target awareness, characterized in that, The method is used for resource adjustment in the large-model inference service system according to any one of claims 1-3, wherein the method includes the following steps: The time to acquire the first token of the service level target and the average interval between each output token except the first output token; If neither the first token time nor the average interval time of each output token other than the first output token meets the service level target, then return the percentage of streaming multiprocessors visible to the current pre-filling process and decoding process respectively. If the first token time or the average interval time of each output token other than the first output token does not meet the service level target, then the process that needs adjustment is determined, and the percentage of the number of streaming multiprocessors visible to the process that needs adjustment is iteratively increased. When the first token time and the average interval time of each output token other than the first output token meet the service level target, the percentage of the number of streaming multiprocessors visible to the pre-filled process or decoding process after adjustment is returned.

5. The inter-stage resource adjustment method for service level target awareness according to claim 4, characterized in that, The iteration increases the percentage of the number of streaming multiprocessors visible to the processes that need adjustment, including: Obtain the maximum number of iterations and the step size for each iteration; The iteration increment process is controlled based on the maximum number of iterations and the step size for each iteration.

6. The inter-stage resource adjustment method for service level target awareness according to claim 4, characterized in that, The process for determining which adjustments are needed includes: If the first token time does not meet the service level target, then the process that needs to be adjusted is a pre-filled process; If the average interval between each output token (excluding the first output token) does not meet the service level target, then the process that needs adjustment is the decoding process.

7. The inter-stage resource adjustment method for service level target awareness according to claim 6, characterized in that, The iterative increase process also includes: Obtain the first correspondence between the pre-filling process and the first token time, and estimate the first token time in the iterative increment process based on the first correspondence. Obtain a second correspondence between the decoding process and the average interval time of each output token except the first output token, and estimate the average interval of each output token except the first output token during the iterative increase process based on the second correspondence.

8. The inter-stage resource adjustment method for service level target awareness according to claim 7, characterized in that, The first correspondence is: The second correspondence is: ， in, For the first token time, The average time interval for each output token except the first output token. , , , , For parameters, , This represents the ratio of streaming multiprocessors.

Images