Large model-oriented reasoning service system and inter-stage resource adjustment method

Abstract

The invention relates to the technical field of large models, in particular to a large-model-oriented reasoning service system and an inter-stage resource adjustment method, and the system comprises a pre-filling processor and a decoding processor; a computing resource controller implementing computing resource control of the prefill processor and the decode processor based on a multi-process service, the multi-process service allowing a target number of streaming multiprocessors to be designated to a prefill process representing an entity of the prefill processor and a decode process representing an entity of the decode processor; and the unified storage manager is used for managing the storage utilization rate of the system and the storage access of the pre-filling processor and the decoding processor to the large model weight and the KV cache. Therefore, the problems of delay interference, low storage efficiency and the like between pre-filling and decoding in related technologies are solved.

Description

Technical Field

[0001] The present application relates to the field of large model technology, and in particular to a large model reasoning service system and a method for adjusting resources between stages. Background Art

[0002] With the booming development of large model-related applications, more and more users are starting to use large models for content generation. The usage method is that users send requests to large model service providers and then wait for the results to be returned. However, due to the huge computing and memory access requirements, LLM (Large Language Model) reasoning services often introduce considerable latency. Unlike traditional requests, the latency of LLM requests consists of two different parts, because each LLM request involves two processing stages: the pre-filling stage and the decoding stage. The pre-filling stage processes the input text and generates the first output token (Token), and the decoding stage generates outputs in an autoregressive manner, one token per iteration. Therefore, the first part is the time from when the user sends the request to when it receives TTFT (Time-to-First-Token), the main part of which is the execution time of the pre-filling stage, and it also includes the time for request waiting; the second part is when the user receives TPOT (Time-per-Output-Token, the average time between each output token except the first output token), which records the average latency of each token in the decoding stage. Therefore, how to maximize the system throughput while complying with the TTFT and TPOT constraints is very important to reduce the cost of reasoning services.

[0003] Request batching is an effective way to improve system throughput. Initial work chose to put requests from different stages on the same GPU for calculation - "fusion design". These works either prioritize pre-filled requests or decoded requests. The former is beneficial to TTFT but will cause an increase in TPOT, and vice versa. This trade-off is called "delay interference" between the two stages.

[0004] In order to mitigate such interference, the related technology divides the input of the pre-fill request into smaller pre-fill request segments, and then schedules the pre-fill request segments and the decoding requests together into the same batch for calculation. The above work cannot completely avoid the interference between TTFT and TPOT, especially under the condition that SLO constrains TTFT and TPOT, it is very challenging to optimize the system throughput.

[0005] To this end, the related art proposes a "separate design" to place prefill requests and decoding requests on different GPUs to eliminate interference between TTFT and TPOT. In a separate system, some GPUs act as prefill processors to process prefill requests, while other GPUs act as decoding processors to process decoding requests. However, although the storage separation design solves the problem of latency interference, it still has the following problems: storage imbalance, KV cache transmission overhead, difficulty in adjusting resources between prefill and decoding processors, and weight copies. Summary of the invention

[0006] The present application provides a large-model inference service system and a method for adjusting resources between stages to solve the problems of delay interference, storage inefficiency, etc. between pre-filling and decoding in related technologies.

[0007] The first aspect of the present application provides a large model inference service system, including: a pre-filling processor and a decoding processor; a computing resource controller, which implements computing resource control of the pre-filling processor and the decoding processor based on a multi-process service, and the multi-process service allows a target number of streaming multiprocessors to be assigned to the pre-filling process and the decoding process, wherein the pre-filling process represents an entity of the pre-filling processor and the decoding process represents an entity of the decoding processor; a unified storage manager, which is used to manage the storage utilization of the system and the storage access of the pre-filling processor and the decoding processor to the large model weights and KV cache.

[0008] Optionally, the computing resource controller saves large model weights and KV cache through the target process.

[0009] Optionally, the target process shares the corresponding pointer with the pre-filling process and the decoding process through inter-process communication.

[0010] Optionally, the computing resource controller switches processes asynchronously.

[0011] Optionally, the asynchronous switching process includes: detecting whether the process of the current iteration is completed; if the process of the current iteration is not completed, waiting for the process of the current iteration to be completed; if the process of the current iteration is completed, killing the process that has completed the current iteration and switching to the new process.

[0012] Optionally, storage utilization is locked until the update step is completed.

[0013] A second aspect of the present application provides a service level target-aware inter-stage resource adjustment method, which is used for resource adjustment of the large model inference service system in the above-mentioned embodiment, and includes the following steps: obtaining the first token time of the service level target and the average interval time of each output token except the first output token; if the first token time and the average interval time of each output token except the first output token do not meet the service level target, returning the percentage of the number of streaming multiprocessors visible to the current pre-filling process and decoding process respectively; if the first token time or the average interval time of each output token except the first output token meets the service level target, determining the process that needs to be adjusted, and iteratively increasing the percentage of the number of streaming multiprocessors visible to the process that needs to be adjusted; when the first token time and the average interval time of each output token except the first output token meet the service level target, returning the adjusted percentage of the number of streaming multiprocessors visible to the pre-filling process or decoding process.

[0014] Optionally, iteratively increasing the percentage of the number of streaming multiprocessors visible to the process that needs to be adjusted includes: obtaining a maximum number of iterations and a step size for each iteration increase; and controlling the iterative increase process according to the maximum number of iterations and the step size for each iteration increase.

[0015] Optionally, determining the process that needs to be adjusted includes: if the first token time does not meet the service level target, the process that needs to be adjusted is the pre-filling process; if the average interval time of each output token except the first output token does not meet the service level target, the process that needs to be adjusted is the decoding process.

[0016] Optionally, during the iterative increase process, it also includes: obtaining a first correspondence between the pre-filling process and the first token time, and estimating the first token time in the iterative increase process based on the first correspondence; obtaining a second correspondence between the decoding process and the average interval time of each output token except the first output token, and estimating the average interval of each output token except the first output token in the iterative increase process based on the second correspondence.

[0017] Optionally, the first corresponding relationship is:

[0018]

[0019] The second corresponding relationship is:

[0020]

[0021] Among them, TTFT x is the first token time, TPOT yis the average time interval between each output token except the first output token, a1, b1, λ, a2, b2 are parameters, and x, y are the ratios of the streaming multiprocessor.

[0022] Therefore, this application includes the following beneficial effects:

[0023] The embodiment of the present application constructs a large service model inference service system, which specifically includes a pre-filling processor, a decoding processor, a computing resource controller and a unified storage manager. By adding a computing resource controller, the computing resource division between the pre-filling processor and the decoding processor can be controlled. By adding a unified storage manager, it is used to manage the storage utilization of the large service model inference service system, as well as the storage access of the pre-filling processor and the decoding processor to the large model weight and KV cache, so that the computing separation is realized through the computing resource controller, and the delay interference between pre-filling and decoding is effectively reduced, so that the system can efficiently process system requests, and at the same time, storage fusion is realized through the unified storage manager, ensuring the effective use of storage resources and reducing resource waste. As a result, technical problems such as delay interference between pre-filling and decoding and storage inefficiency in related technologies are solved.

[0024] Additional aspects and advantages of the present application will be given in part in the description below, and in part will become apparent from the description below, or will be learned through the practice of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] The above and / or additional aspects and advantages of the present application will become apparent and easily understood from the following description of the embodiments in conjunction with the accompanying drawings, in which:

[0026] Figure 1 A schematic diagram of computing fusion and storage fusion in related technologies;

[0027] Figure 2 A schematic diagram of computing separation and storage separation in the related art;

[0028] Figure 3 A schematic diagram of the computation separation and storage fusion provided in an embodiment of the present application;

[0029] Figure 4 A schematic diagram of a large model reasoning service system provided according to an embodiment of the present application;

[0030] Figure 5 A schematic diagram of request processing during process switching in the related art;

[0031] Figure 6 A schematic diagram of request processing when delaying switching after joining a target process according to an embodiment of the present application;

[0032] Figure 7A schematic diagram of the atomic allocation space provided according to an embodiment of the present application;

[0033] Figure 8 A specific structural diagram of a large model reasoning service system provided according to an embodiment of the present application;

[0034] Fig. 9 The present invention is a flowchart of a service level objective-aware inter-stage resource adjustment method provided according to an embodiment of the present application. DETAILED DESCRIPTION

[0035] The embodiments of the present application are described in detail below, and examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are intended to be used to explain the present application, and should not be construed as limiting the present application.

[0036] Before describing the solution of the present application, the method of improving the large model reasoning service system in the related art is introduced.

[0037] 1. Computing and storage fusion design, such as Figure 1 shown.

[0038] Prioritize the calculation of pre-fill requests or prioritize the calculation of decode requests. The former is beneficial to TTFT but will cause an increase in TPOT, and vice versa. This trade-off is called "delay interference" between the two stages.

[0039] 2. Design of separation of computing, classification and storage, such as Figure 2 shown.

[0040] Prefill requests and decode requests are placed on different GPUs to eliminate interference between TTFT and TPOT. In a split system, some GPUs are used as prefill processors to process prefill requests, while other GPUs are used as decode processors to process decode requests.

[0041] However, storage separation has the following disadvantages: 1) Storage imbalance. The prefilling processor only generates the portion of the KV cache corresponding to the input length. The decoding processor needs to store the entire KV cache relative to the full sequence length including the output length. When the output length is large, the storage imbalance will become more serious. Therefore, the storage space required by the decoding processor is significantly larger than that of the prefilling processor. 2) KV cache transmission overhead. Since the prefilling stage and the decoding stage are deployed on different GPUs, the KV cache needs to be transferred from the prefilling processor to the decoding processor. There will be significant communication delays on low-end GPUs. Even for high-end GPUs equipped with NVLink, the transmission time will be calculated into TPOT, which will affect TPOT. 3) Difficulty in adjusting resources between the prefilling and decoding processors. The workload of prefilling and decoding requests will change as the service process proceeds. However, the separated design can only perform coarse-grained adjustments on a GPU basis. In addition, since the KV cache is mainly located on the decoding processor, specific designs are often required to avoid the cost of KV cache transmission when adjusting, thereby introducing additional overhead. 4) Weight replicas. Both the prefilling processor and the decoding processor need to accommodate the entire model weight. This means that to deploy the same model, the split design requires twice the number of GPUs.

[0042] To this end, this application provides a large model reasoning service system, such as Figure 3 As shown in the figure, the features of computing and storage are decoupled, and a computing separation and storage fusion design is adopted, which retains the main advantages of storage fusion in the fusion type and computing separation in the separation type. This enables large models to maintain lower latency under high request rates, thereby serving more requests in the same time.

[0043] Specifically, Figure 4 A schematic diagram of a large model reasoning service system provided in an embodiment of the present application.

[0044] like Figure 4 As shown, the large model-oriented reasoning service system 10 includes: a pre-filling processor 11, a decoding processor 12, a computing resource controller 13 and a unified storage manager 14.

[0045] Among them, the computing resource controller 13 implements the computing resource control of the pre-filling processor 11 and the decoding processor 12 based on the multi-process service. The multi-process service allows the target number of streaming multiprocessors to be assigned to the pre-filling process and the decoding process, wherein the pre-filling process represents the entity of the pre-filling processor and the decoding process represents the entity of the decoding processor; the unified storage manager 14 is used to manage the storage utilization of the system and the storage access of the pre-filling processor and the decoding processor to the large model weights and KV cache.

[0046] It can be understood that the embodiment of the present application constructs a large service model reasoning service system 10, which specifically includes a pre-filling processor 11, a decoding processor 12, a computing resource controller 13 and a unified storage manager 14. By adding a computing resource controller 13, it is possible to control the computing resource division between the pre-filling processor 11 and the decoding processor 12. By adding a unified storage manager 14, it is used to manage the storage utilization of the large service model reasoning service system 10, as well as the storage access of the pre-filling processor 11 and the decoding processor 12 to the large model weights and KV cache, so that computing separation is achieved through the computing resource controller 13, effectively reducing the delay interference between pre-filling and decoding, so that the system can efficiently process system requests, and at the same time, storage fusion is achieved through the unified storage manager 14, ensuring the effective use of storage resources and reducing resource waste, so that the system can maintain lower latency under high request rates, thereby serving more requests in the same time and improving the system throughput.

[0047] Specifically, the embodiment of the present application can use the computing resource controller 13 to implement computing resource control of the pre-filling processor and the decoding processor based on the multi-process service MPS. The multi-process service allows the target number of SMs (Streaming Multi-processor) to be assigned to a certain pre-filling process and decoding process, thereby realizing computing resource differentiation at the SM level. The pre-filling process represents the entity of the pre-filling processor, and the decoder process represents the entity of the decoding processor.

[0048] In an embodiment of the present application, the computing resource controller 13 saves the large model weights and KV cache through the target process.

[0049] Among them, the target process can also be understood as a resident process.

[0050] Since the entities of the pre-filling processor and the decoding processor in the system are both processes, the pair of processes receive the computing resource partition parameters (x, y) at initialization, where x and y represent the percentage of the number of SMs visible to the pre-filling and decoding processes after MPS scheduling. And since each processor is mounted on a process, MPS does not support adjusting the (x, y) of an existing process, so a pair of processes can only be assigned a set of (x, y). This means that adjusting (x, y) requires re-calling the MPS interface, which incurs the overhead of process switching. Specifically, the switch needs to wait for the pre-filling processor and the decoding processor to complete their running iterations and store the generated tokens before terminating the two processes. After the original process is terminated, the MPS interface is used to apply the new (x, y) to the two new processes, marking them as new pre-filling processors and decoding processors. Based on the existing KV cache and saved tokens, the new processor continues to run.

[0051] In this process, the cost comes from two aspects. First, Figure 5 As shown, initializing two MPS processes includes the following steps: 1) loading weights into the new pre-filling process and sharing weights with the new decoding process via IPC (and vice versa); 2) copying the KV cache to the new decoding process; 3) initializing the inference engine for the two new processes. The above switching steps will cause newly arriving requests to be blocked during the switching process until the new pair of processes are ready. When serving Llama3-8B at 10 requests per second, the typical switching overhead is 22.8 seconds, resulting in 228 requests being blocked. Secondly, the timing of the switch is also important. Since the pre-filling process and the decoding process have different processing progress, synchronization operations need to be inserted to block processes that have completed earlier than the current iteration.

[0052] In short, in order to solve the first overhead, that is, the time and resource overhead required to restart the pre-filling process and the decoding process when adjusting the computing resource allocation, the embodiment of the present application introduces a target process in the system, such as Figure 6 As shown, the computing resource controller saves the large model weights and KV cache through the target process, avoiding the overhead of repeatedly loading weights and copying KV cache.

[0053] In an embodiment of the present application, the target process shares the corresponding pointer with the pre-filling process and the decoding process through inter-process communication.

[0054] It can be understood that the target process of the embodiment of the present application shares the corresponding pointer to the pre-filling process and the decoding process through inter-process communication, so that the pre-filling and decoding processes can access the weight and KV cache through the pointer, reducing the delay of data transmission.

[0055] In the embodiment of the present application, the computing resource controller 13 switches processes asynchronously.

[0056] In order to solve the second overhead, that is, when adjusting the computing resource allocation, since the pre-filling process and the decoding process have different processing progress, it is necessary to insert a synchronization operation to block the process that completes the current iteration earlier, and the overhead generated, the computing resource controller 13 of the embodiment of the present application uses an asynchronous switching process to reduce the synchronization overhead caused by different processing progress. The specific asynchronous switching process is as follows.

[0057] In an embodiment of the present application, the asynchronous switching process includes: detecting whether the process of the current iteration is completed; if the process of the current iteration is not completed, waiting for the process of the current iteration to be completed; if the process of the current iteration is completed, killing the process that has completed the current iteration and switching to the new process.

[0058] It can be understood that the asynchronous switching agenda of the embodiment of the present application includes: detecting whether the process of the current iteration (pre-filling process and decoding process) is completed; if the process of the current iteration is not completed, waiting for the process of the current iteration to be completed; if the process of the current iteration is completed, killing the process of the current iteration and switching to the new process, thereby reducing the blocking time of the system, and MPS allows the resource percentage of all running processes to be greater than 100%, which indicates that during the switching process, although some processes are terminated, other processes can still continue to compete for resources, ensuring efficient use of resources. Through asynchronous switching, resource adjustment can be completed in a shorter time, avoiding resource idleness and improving resource utilization.

[0059] In the embodiment of the present application, the storage utilization is locked before the updating step is completed.

[0060] It should be noted that the storage manager mainly focuses on two types of storage access, namely reading model weights and accessing the KV cache. The unified storage manager manages model weights in a relatively straightforward way because model weights are read-only. However, the management of KV cache access is relatively complex. The system uses the same KV cache paging storage mechanism as vLLM, accessing the KV cache through the block table index. Therefore, once the block table index is determined, the KV cache can be accessed without conflict. However, conflicts occur when the pre-filling processor and the decoding processor open up KV cache space in an asynchronous manner.

[0061] The pre-filling processor allocates space to store the generated KV cache when a single layer of calculation is completed, and the decoding processor allocates KV cache in the PagedAttention calculation of each layer. The asynchronous working mode of the two processors may cause read-after-write (WAR) conflicts in storage management. Figure 7 As shown in Figure 1, the storage space allocation of the KV cache consists of three steps: 1) querying the storage utilization to see if there are free blocks; 2) obtaining blocks for KV cache storage; 3) updating the storage utilization. A WAR conflict occurs when one processor queries the utilization and another processor updates the utilization immediately. The result is that the process performing the query operation updates the wrong storage utilization based on the queried value.

[0062] For this reason, the unified storage manager of the embodiment of the present application uses an atomic KV cache block allocation method, that is, the storage utilization is locked before the update step is completed. This atomic allocation method can ensure that the storage utilization of the two asynchronous processes is correct.

[0063] The following describes the large model reasoning service system of the present application through a specific embodiment, that is, a large model reasoning service system with computing separation and storage fusion features, so that the large model can maintain lower latency under high request rate, thereby serving more requests in the same time, and the service request rate can be increased by 1.55-1.72 times. The specific system structure is as follows: Figure 8 shown.

[0064] The pre-fill and decode processors handle pre-fill and decode requests respectively, and each processor maintains its own waiting request and running request queues. The system implements computational separation through a computational resource controller, which determines the division of computational resources between the two processors and performs real-time adjustments as needed. For the storage part, the system manages the storage utilization of the system through a unified storage manager, which is also responsible for the storage access of the two processors to the weight and KV cache.

[0065] 1. Computing Resource Controller

[0066] The computing resource controller implements computing separation based on the CUDA application interface of MPS. MPS allows a certain number of SMs to be assigned to a process, thereby realizing computing resource partitioning at the SM level. Specifically, the entities of the pre-filling processor and the decoding processor in the system are both processes. This pair of processes receives the computing resource partitioning parameters (x, y) at initialization, where x and y represent the percentage of SMs visible to the pre-filling and decoding processes after MPS scheduling.

[0067] Since each processor is mounted on a process, MPS does not support adjusting the (x, y) of an existing process, so a pair of processes can only be assigned one set of (x, y). This means that adjusting (x, y) requires re-calling the MPS interface, which incurs the overhead of process switching. Specifically, the switch needs to wait for the pre-filling processor and the decoding processor to complete their running iterations and store the generated tokens before terminating the two processes. After the original process is terminated, the MPS interface is used to apply the new (x, y) to the two new processes, marking them as new pre-filling processors and decoding processors. Based on the existing KV cache and saved tokens, the new processor continues to run.

[0068] In this process, the cost comes from two aspects. First, Figure 5As shown, initializing two MPS processes includes the following steps: 1) loading weights to the new pre-filling process and sharing weights with the new decoding process via IPC (and vice versa), 2) copying the KV cache to the new decoding process, and 3) initializing the inference engine for the two new processes. The above switching steps will cause newly arriving requests to be blocked during the switching process until the new pair of processes are ready. When serving Llama3-8B at 10 requests per second, the typical switching overhead is 22.8 seconds, resulting in 228 requests being blocked. Secondly, the timing of the switch is also important. Since the pre-filling process and the decoding process have different processing progress, synchronization operations need to be inserted to block the process that has completed the current iteration earlier. An alternative solution is process refresh: that is, the slower process can be terminated and the calculation can be re-executed on the new process.

[0069] like Figure 6 As shown, in order to solve the first overhead, the present application introduces a resident process, which always saves the weights and KV cache during the service, avoiding the need for other processes to repeatedly load weights and copy the KV cache. Specifically, this resident process shares the corresponding pointers to the pre-filling process and the decoding process through IPC, so that the pre-filling and decoding processes can access the weights and KV cache through pointers. The storage space of the weights and KV cache will not be released when the process stops, and the new process can quickly access the storage through IPC. In addition, the overhead can be further reduced by delaying the switching point, that is, switching only when the new process is ready, covering up the stagnation period during the switching. Before the process switches, the arriving request is still calculated on the old process using the old (x, y) parameters.

[0070] To solve the second overhead, this application uses an asynchronous switching method, that is, first kill the process that has completed the current iteration and switch to the new process, and wait for the unfinished process to complete the current iteration before performing the same operation. Because MPS allows the resource percentage of all running processes to be greater than 100%, this method can be used to handle the overhead of asynchronous switching, and each process will compete for resources in a short period of time.

[0071] Unified Storage Manager

[0072] The storage manager focuses on two types of storage access, reading model weights and accessing the KV cache. The unified storage manager manages model weights in a straightforward way because model weights are read-only. However, the management of KV cache access is relatively complex. The system uses the same KV cache paging storage mechanism as vLLM to access the KV cache through the block table index. Therefore, once the block table index is determined, the KV cache can be accessed without conflict. However, conflicts occur when the pre-fill processor and the decoding processor open up KV cache space in an asynchronous manner.

[0073] The pre-filling processor allocates space to store the generated KV cache when a single layer of calculation is completed, and the decoding processor allocates KV cache in the PagedAttention calculation of each layer. The asynchronous working mode of the two processors may cause read-after-write (WAR) conflicts in storage management. Figure 7 As shown in Figure 1, the storage space allocation of the KV cache consists of three steps: 1) querying the storage utilization to see if there are free blocks; 2) obtaining blocks for KV cache storage; and 3) updating the storage utilization. A WAR conflict occurs when one processor queries the utilization and another processor updates the utilization immediately. The result is that the process performing the query operation updates the incorrect storage utilization based on the queried value. To address this challenge, the unified storage manager uses an atomic KV cache block allocation method, that is, the storage utilization is locked before the update step is completed. This atomic allocation method ensures that the storage utilization of the two asynchronous processes is correct.

[0074] According to the large-model inference service system proposed in the embodiment of the present application, it specifically includes a pre-filling processor, a decoding processor, a computing resource controller and a unified storage manager. By adding a computing resource controller, it is possible to control the division of computing resources between the pre-filling processor and the decoding processor. By adding a unified storage manager, it is used to manage the storage utilization of the large-model inference service system, as well as the storage access of the pre-filling processor and the decoding processor to the large model weights and KV cache, so that computing separation is achieved through the computing resource controller, effectively reducing the delay interference between pre-filling and decoding, so that the system can efficiently process system requests, and at the same time, storage fusion is achieved through the unified storage manager, ensuring the effective use of storage resources and reducing resource waste, so that the system can maintain lower latency under high request rates, thereby serving more requests in the same time and improving the system throughput.

[0075] Next, the service level target-aware inter-stage resource adjustment method proposed in an embodiment of the present application will be described with reference to the accompanying drawings.

[0076] Fig. 9 It is a flow chart of the service level target-aware inter-stage resource adjustment method of an embodiment of the present application.

[0077] like Fig. 9 As shown, the service level target-aware inter-stage resource adjustment method is used for resource adjustment of the above-mentioned large model reasoning service system, and includes the following steps:

[0078] In step S101, the first token time of the service level target and the average interval time of each output token except the first output token are obtained.

[0079] It can be understood that the embodiments of the present application can obtain the TTFT (Time-to-First-Token, first token time) and TPOT (Time-per-Output-Token, the average time interval for each output token except the first output token) of the service level objective SLO. TPOT records the average delay of each token in the decoding stage to facilitate subsequent resource adjustment.

[0080] In step S102 , if the first token time and the average interval time of each output token except the first output token do not meet the service level target, the percentage of the number of streaming multiprocessors visible to the current pre-filling process and the decoding process is returned.

[0081] It is understandable that the embodiment of the present application may return the current pre-filled process and the percentage of the number of streaming multiprocessors visible to each process when both TTFT and TPOT fail to meet the SLO because there is no adjustment space at this time.

[0082] In step S103, if the first token time or the average interval time of each output token except the first output token meets the service level target, the process that needs to be adjusted is determined, and the percentage of the number of streaming multiprocessors visible to the process that needs to be adjusted is iteratively increased. When the first token time and the average interval time of each output token except the first output token meet the service level target, the adjusted percentage of the number of streaming multiprocessors visible to the pre-filling process or the decoding process is returned.

[0083] It can be understood that the embodiment of the present application can determine the process that needs to be adjusted when either TTFT or TPOT does not meet its SLO, iteratively increase the percentage of the number of streaming multiprocessors visible to the process that needs to be adjusted, and return the adjusted percentage of the number of streaming multiprocessors visible to the pre-filling process or the decoding process when the first token time and the average interval time of each output token except the first output token meet the service level target, so as to dynamically adjust the resource allocation of the pre-filling and decoding processors according to the actual performance of the large model inference service system to ensure that TTFT and TPOT meet the SLO.

[0084] In an embodiment of the present application, determining the process that needs to be adjusted includes: if the first token time does not meet the service level target, the process that needs to be adjusted is the pre-filling process; if the average interval time of each output token except the first output token does not meet the service level target, the process that needs to be adjusted is the decoding process.

[0085] It is understandable that the embodiments of the present application can accurately determine the process that needs to be adjusted according to the specific performance bottleneck. If TTFT does not meet the SLO, the pre-filling process is adjusted; if TPOT does not meet the SLO, the decoding process is adjusted.

[0086] In an embodiment of the present application, iteratively increasing the percentage of the number of streaming multiprocessors visible to the process that needs to be adjusted includes: obtaining the maximum number of iterations and the step size of each iteration increase; and controlling the iterative increase process according to the maximum number of iterations and the step size of each iteration increase.

[0087] Among them, the maximum number of iterations and the step size increased in each iteration can be set in advance according to specific circumstances.

[0088] It can be understood that the embodiments of the present application can control the iteration increase process according to the maximum number of iterations and the step size of each iteration increase, thereby ensuring the controllability and stability of resource adjustment and avoiding system instability caused by excessive adjustment.

[0089] In an embodiment of the present application, during the iterative increase process, it also includes: obtaining a first correspondence between the pre-filling process and the first token time, and estimating the first token time in the iterative increase process based on the first correspondence; obtaining a second correspondence between the decoding process and the average interval time of each output token except the first output token, and estimating the average interval of each output token except the first output token in the iterative increase process based on the second correspondence.

[0090] It can be understood that the embodiment of the present application can more accurately predict and adjust resource allocation and further optimize system performance by establishing a first corresponding relationship between the pre-filling process and TTFT, and a second corresponding relationship between the decoding process and TPOT.

[0091] In the embodiment of the present application, the first corresponding relationship is:

[0092]

[0093] The second corresponding relationship is:

[0094]

[0095] Among them, TTFT x is the first token time, TPOT y is the average time interval between each output token except the first output token, a1, b1, λ, a2, b2 are parameters, and x, y are the ratios of the streaming multiprocessor.

[0096] The following describes the service level target-aware inter-stage resource adjustment method (also referred to as an adjustment algorithm) of the present application through a specific embodiment, which is as follows:

[0097] The computing resource allocation, i.e. (x, y), is adjusted periodically according to winndow_size. The system collects the TTFT and TPOT recorded in the periodic window to update the TTFT / TPOT estimation model. When both TTFT and TPOT do not meet the SLO, the algorithm directly returns the current (x, y) because there is no room for adjustment in this case. Otherwise, the algorithm first determines which of TTFT or TPOT does not meet its SLO, and iteratively increases the visible SM ratio of the corresponding processor. After each increase, the algorithm evaluates whether the updated ratio meets the SLO. If not, it continues to iterate until it meets the SLO. This process has two parameters, the maximum number of iterations max_step and the step size step_size increased in each iteration.

[0098] The algorithm needs to model the relationship between TTFT and TPOT based on the accessible SM ratios x and y, respectively, for use in the corresponding TTFT and TPOT estimation models. Since the two processors process requests asynchronously, TTFT and TPOT can be modeled separately. The TTFT of a single request consists of two delays, namely the waiting delay and the pre-filling delay. For TPOT, due to the short single iteration time and the high number of concurrency, it can be assumed that the request is executed immediately after arriving at the decoding processor, because the waiting delay can be ignored in calculating TPOT. The more SMs a process can use, the more computing resources it can utilize in parallel, and the processing rate will increase proportionally. Therefore, the processing delay can be modeled as:

[0099]

[0100] Among them, l x represents the processing delay of the pre-filling and decoding stages under the SM ratio x. For TTFT, the M / M / 1 queue model is applied to model the sum of waiting and pre-filling delays as follows:

[0101]

[0102] where r and μ x They represent the pre-fill processing rate under a given input request rate and SM ratio x. w is the sum of the waiting delay and the processing delay. x =1 / l x Substituting ∝x into the equation, the above formula can be further written as:

[0103]

[0104] Therefore, the relationship between TTFT and x, and TPOT and y is expressed as:

[0105]

[0106] The parameters a1, b1, λ, a2, and b2 are learnable and can be calculated by real-time observation of TTFT and TPOT. The constant terms b1 and b2 are used to characterize the constant overhead in the delay. Given (x, y), the above formulas can be used to estimate TTFT and TPOT in order to quantify the impact of adjusting (x, y) during service.

[0107] It should be noted that the aforementioned explanation of the embodiment of the large-model reasoning service system is also applicable to the service level target-aware inter-stage resource adjustment method of this embodiment, and will not be repeated here.

[0108] According to the service level target-aware inter-stage resource adjustment method proposed in the embodiment of the present application, it is possible to determine whether the service level target is met based on the first token output time and the average time interval of each output token except the first output token, dynamically adjust the resource allocation of the pre-filling process and the decoding process, improve resource utilization efficiency, and thus improve the system performance of subsequent large-model inference service systems.

[0109] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present application. In this specification, the schematic representations of the above terms are not necessarily directed to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or N embodiments or examples in a suitable manner. In addition, those skilled in the art may combine and combine the different embodiments or examples described in this specification and the features of the different embodiments or examples, without contradiction.

[0110] In addition, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Therefore, a feature defined as "first" or "second" may explicitly or implicitly include at least one of the features. In the description of this application, "N" means at least two, such as two, three, etc., unless otherwise clearly and specifically defined.

[0111] Any process or method description in a flowchart or otherwise described herein may be understood to represent a module, fragment or portion of code comprising one or N executable instructions for implementing the steps of a custom logical function or process, and the scope of the preferred embodiments of the present application includes alternative implementations in which functions may not be performed in the order shown or discussed, including performing functions in a substantially simultaneous manner or in reverse order depending on the functions involved, which should be understood by technicians in the technical field to which the embodiments of the present application belong.

[0112] It should be understood that the various parts of the present application can be implemented in hardware, software, firmware or a combination thereof. In the above-mentioned embodiments, the steps or methods can be implemented in software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented in hardware, as in another embodiment, it can be implemented by any one of the following technologies known in the art or their combination: a discrete logic circuit having a logic gate circuit for implementing a logic function on a data signal, a dedicated integrated circuit having a suitable combination of logic gate circuits, a programmable gate array, a field programmable gate array, etc.

[0113] A person of ordinary skill in the art may understand that all or part of the steps carried by the method for implementing the above-mentioned embodiment may be completed by instructing related hardware through a program, and the above-mentioned program may be stored in a computer-readable storage medium, which, when executed, includes one of the steps of the method embodiment or a combination thereof.

[0114] Although the embodiments of the present application have been shown and described above, it can be understood that the above embodiments are exemplary and cannot be understood as limitations on the present application. Ordinary technicians in this field can change, modify, replace and modify the above embodiments within the scope of the present application.

Claims

1. A large model-oriented reasoning service system, characterized in that: include: a pre-fill processor and a decoding processor; a computing resource controller that implements computing resource control of the pre-filling processor and the decoding processor based on a multi-process service, wherein the multi-process service allows a target number of streaming multiprocessors to be assigned to a pre-filling process and a decoding process, wherein the pre-filling process represents an entity of the pre-filling processor and the decoding process represents an entity of the decoding processor; A unified storage manager is provided for managing storage utilization of the system and storage accesses of the pre-fill processor and the decode processor to large model weights and KV cache.

2. The large model-oriented reasoning service system according to claim 1, characterized in that: The computing resource controller saves large model weights and KV cache through the target process.

3. The large model-oriented reasoning service system according to claim 2, characterized in that: The target process shares the corresponding pointer with the pre-filling process and the decoding process through inter-process communication.

4. The large model-oriented reasoning service system according to claim 1, characterized in that: The computing resource controller switches processes asynchronously.

5. The large model-oriented reasoning service system according to claim 4, characterized in that: The asynchronous switching process includes: Check whether the current iteration process is completed; If the process of the current iteration is not completed, wait for the process of the current iteration to be completed; If the process of the current iteration is completed, the process that has completed the current iteration is killed and switched to a new process.

6. The large model-oriented reasoning service system according to claim 1, characterized in that: The storage utilization is locked until the updating step is completed.

7. A service level objective-aware inter-stage resource adjustment method, characterized in that: The method is used for resource adjustment of a large model reasoning service system according to any one of claims 1 to 6, wherein the method comprises the following steps: Get the service level target's first token time and the average time between each output token except the first output token; If both the first token time and the average interval time of each output token except the first output token do not meet the service level target, then returning the percentage of the number of streaming multiprocessors visible to the current pre-filling process and the decoding process respectively; If the first token time or the average interval time of each output token except the first output token meets the service level target, the process that needs to be adjusted is determined, and the percentage of the number of streaming multiprocessors visible to the process that needs to be adjusted is iteratively increased. When the first token time and the average interval time of each output token except the first output token meet the service level target, the adjusted percentage of the number of streaming multiprocessors visible to the pre-filling process or the decoding process is returned.

8. The service level objective-aware inter-stage resource adjustment method according to claim 7, characterized in that: The iterative increase of the percentage of the number of streaming multiprocessors visible to the process that needs to be adjusted includes: Get the maximum number of iterations and the step size increased by each iteration; The iterative increase process is controlled according to the maximum number of iterations and the step size of each iteration increase.

9. The service level objective-aware inter-stage resource adjustment method according to claim 7, characterized in that: The process of determining the process that needs to be adjusted includes: If the first token time does not meet the service level target, the process that needs to be adjusted is a pre-filling process; If the average interval time of each output token except the first output token does not meet the service level target, the process that needs to be adjusted is the decoding process.

10. The service level objective-aware inter-stage resource adjustment method according to claim 9, characterized in that: In the iterative increase process, it also includes: Acquire a first corresponding relationship between the pre-filling process and the first token time, and estimate a first token time in an iterative increase process according to the first corresponding relationship; A second corresponding relationship between the decoding process and the average interval time of each output token except the first output token is obtained, and an average interval of each output token except the first output token in the iterative increase process is estimated according to the second corresponding relationship.

11. The service level objective-aware inter-stage resource adjustment method according to claim 10, characterized in that: The first corresponding relationship is: The second corresponding relationship is: Among them, TTFT x is the first token time, TPOT y is the average time interval between each output token except the first output token, a1, b1, λ, a2, b2 are parameters, and x, y are the ratios of the streaming multiprocessor.

Images