Request processing method, and electronic device, storage medium and program product

Abstract

The present application belongs to the technical field of artificial intelligence. Provided are a request processing method, and an electronic device, a storage medium and a program product. The method includes: receiving an inference request, wherein the inference request is used for requesting the use of a large model to perform model inference; adding the inference request to a request queue, wherein the request queue is used for storing a plurality of inference requests; on the basis of inference information of the plurality of inference requests, determining priorities of the plurality of inference requests, wherein the inference information comprises task difficulty, waiting time and a corresponding user level of each of the plurality of inference requests; and on the basis of the priorities of the plurality of inference requests, selecting at least one inference request from among the plurality of inference requests to perform model inference in an inference stage.

Description

Request processing methods, electronic devices, storage media and program products

[0001] Cross-references

[0002] This application claims priority to Chinese Patent Application No. 202411439303.3, filed on October 15, 2024, entitled "Request Processing Method, Electronic Device, Storage Medium and Program Product", the entire contents of which are incorporated herein by reference. Technical Field

[0003] This application relates to the field of artificial intelligence technology, and in particular to a request processing method, electronic device, storage medium, and program product. Background Technology

[0004] With the rapid development of Artificial Intelligence (AI) and Machine Learning technologies, large-scale pre-trained models (referred to as Large Language Models (LLMs), such as GPT-4 and PaLM, have become core components of AI research and applications due to their powerful multimodal processing capabilities and broad application prospects. A typical application of large models is large model inference, which utilizes pre-trained large models for practical task processing. It is a crucial step in transforming the potential of large models into real-world value, playing a vital role in areas such as intelligent question answering, automatic code generation, medical diagnosis, and autonomous driving, significantly improving the performance and user experience of various intelligent applications.

[0005] Large-scale model inference typically involves massive computational loads and complex resource management, especially when handling high-concurrency inference requests. Efficiently allocating computing resources, reducing response time, and improving overall service quality are pressing technical challenges. Currently, most related technologies schedule inference requests based on task queuing and first-come, first-served strategies. However, this approach lacks flexibility and cannot balance user needs with resource utilization efficiency, ultimately impacting system performance and user experience. Summary of the Invention

[0006] This application provides a request processing method, an electronic device, a storage medium, and a program product.

[0007] In a first aspect, a request processing method is provided, comprising: receiving an inference request, the inference request being used to request model inference using a large model; adding the inference request to a request queue, the request queue being used to store multiple inference requests; determining the priority of the multiple inference requests based on inference information of the multiple inference requests, the inference information including the task difficulty, waiting time, and corresponding user level of the multiple inference requests; and selecting at least one inference request from the multiple inference requests to perform model inference in one inference phase based on the priority of the multiple inference requests.

[0008] In a second aspect, an electronic device is provided, comprising: a processor; and a memory for storing processor-executable instructions; wherein the processor is configured to execute the instructions to implement the method as described in the first aspect.

[0009] Thirdly, a computer-readable storage medium is provided, wherein when instructions in the storage medium are executed by a processor of an electronic device, the electronic device is enabled to perform the method described in the first aspect.

[0010] Fourthly, a computer program product is provided, the computer program product including a non-transitory computer-readable storage medium storing a computer program operable to cause a computer to perform some or all of the steps of the method described in the first aspect. Attached Figure Description

[0011] To more clearly illustrate the technical solutions in this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0012] Figure 1 is a schematic diagram of the reasoning process of a large model according to an embodiment of this application.

[0013] Figure 2 is a schematic diagram of the pre-filling stage and the decoding stage in a large model inference according to an embodiment of this application.

[0014] Figure 3 is a flowchart illustrating a request processing method according to an embodiment of this application.

[0015] Figure 4 is a schematic diagram of the request processing flow when the preemption mode is recompute mode according to an embodiment of this application.

[0016] Figure 5 is a schematic diagram of the request processing flow when the preemption mode is the replacement mode in an embodiment of this application.

[0017] Figure 6 is a flowchart illustrating the scheduling of inference requests in a request queue according to an embodiment of this application.

[0018] Figure 7 is a schematic diagram of an embodiment of this application performing a reasoning request.

[0019] Figure 8 is a schematic diagram of a system architecture for implementing large model inference according to an embodiment of this application.

[0020] Figure 9 is a schematic diagram of the structure of a request processing system according to an embodiment of this application.

[0021] Figure 10 is a schematic diagram of the structure of a scheduler according to an embodiment of this application.

[0022] Figure 11 is a schematic diagram of the structure of an electronic device according to an embodiment of this application.

[0023] Figure 12 is a schematic diagram of the structure of a request processing device according to an embodiment of this application. Detailed Implementation

[0024] To enable those skilled in the art to better understand the technical solutions in this application, the technical solutions in this application will be clearly and completely described below with reference to the accompanying drawings of one or more embodiments. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by those skilled in the art without creative effort should fall within the protection scope of this application.

[0025] The terms "first," "second," etc., used in this application and the claims are used to distinguish similar objects and not to describe a specific order or sequence. It should be understood that such use of data can be interchanged where appropriate so that this application can be implemented in orders other than those illustrated or described herein. Furthermore, in this application and the claims, "and / or" indicates at least one of the connected objects, and the character " / " generally indicates that the preceding and following objects are in an "or" relationship.

[0026] The technical solutions provided in this application can be applied to scenarios where large models are used for reasoning. These large models can be, for example, GPT-4, PaLM, etc., and are not specifically limited here. Scenarios where large models are used for reasoning can be, for example, intelligent question answering, automatic code generation, medical diagnosis, autonomous driving, etc., and are not specifically limited here either.

[0027] To facilitate understanding of the technical solutions provided in the embodiments of this application, the reasoning process of large models (i.e., the reasoning principle of large models) in related technologies will be explained below.

[0028] The large model is an autoregressive model, and its inference process can be illustrated in Figure 1. In Figure 1, when using the large model for inference, the user can input a prompt (as shown in Figure 1, "What is the square root of 4?"). After receiving the prompt and performing inference based on it, the large model can output an answer (i.e., the inference result, as shown in Figure 1, "The answer is 2", with a length of N). As can be seen from Figure 1, the large model performs N inference iterations throughout the entire process. Each time the large model performs an inference, it outputs only one token (a basic unit in natural language processing, which can be a word, a segment of words, a character, or other form of text fragment). The output token is concatenated with the input token and used as the input for the next inference iteration. This process is repeated until the end marker is encountered.

[0029] In the inference process shown in Figure 1, the input for the previous inference differs from the input for the subsequent inference by only one word. Therefore, there is a problem of redundant computation between the two inferences. To avoid this problem, related techniques can cache the intermediate results during the inference process of a large model. This allows these intermediate results to be reused in subsequent inferences, thereby reducing redundant computation and improving the overall inference speed. The cached intermediate results are typically two variables, Key and Value (e.g., the key and value parameters in a Transformer, used to accelerate attention computation); therefore, this cache is also called a KV cache.

[0030] The inference phase of a large model can be divided into a prefill phase and a decoding phase. The complete inference process of a single inference request may include one prefill phase and n decoding phases, where n is 0 or a positive integer. For a better understanding of the inference phases of a large model, please refer to Figure 2.

[0031] Figure 2 is a schematic diagram of the pre-filling and decoding stages corresponding to the large model inference process shown in Figure 1. The large model inference process shown in Figure 2 includes one pre-filling stage (corresponding to inference round 1) and two decoding stages (corresponding to inference rounds 2 and 3). In the pre-filling stage, the model input is "What is the square root of 4?". After performing one pre-filling based on the model input, the model executor can output "Answer". Simultaneously, the model executor can cache the intermediate results generated in the pre-filling stage into the KV cache. In the first decoding stage, the model input includes the output of the pre-filling stage and the intermediate results cached in the KV cache. After performing the first decoding based on the model input, the model executor can output "Yes". Simultaneously, the model executor can cache the intermediate results generated in the first decoding stage into the KV cache. In the second decoding stage, the model input includes the output of the first decoding stage and the intermediate results cached in the KV cache. After performing the second decoding based on the model input, the model executor can output "2". At this point, the model inference process ends, and the model output (i.e., the inference result) is "The answer is 2".

[0032] It should be noted that the technical solution provided in this application embodiment can realize iterative priority scheduling of inference requests. That is, when scheduling inference requests in this application embodiment, the granularity of scheduling can be accurate to each generated word. That is, each scheduling will obtain a set of inference requests (one or more inference requests) according to the priority and perform a model inference in the inference stage. Each inference request involved in the model inference generates only a new word. One pre-filling and one decoding can be regarded as a model inference in the inference stage.

[0033] The technical solutions provided by the various embodiments of this application are described in detail below with reference to the accompanying drawings.

[0034] Figure 3 is a flowchart illustrating a request processing method according to an embodiment of this application. The request processing method shown in Figure 3 can be executed by an inference server (such as a large model inference engine) used for model inference, and includes the following steps.

[0035] Step S302: Receive an inference request. The inference request is used to request the use of a large model for model inference.

[0036] Inference requests can be sent by users to the inference server. For example, users can send inference requests to the inference server through the user web interface provided by the inference server. The number of inference requests can be one or more (from one or more users), and there is no specific limit here.

[0037] Step S304: Add the inference request to the request queue, which is used to store multiple inference requests.

[0038] After receiving an inference request, the inference server can add it to a request queue. The request queue stores multiple inference requests, which can be multiple inference requests received simultaneously by the inference server (i.e., concurrent requests) or inference requests received at different times; there are no specific limitations here. Furthermore, these multiple inference requests can be inference requests that have not yet been pre-filled, or inference requests that have been pre-filled and are in the decoding stage; there are no specific limitations here either. In other words, any inference request that has not yet completed the inference process can be stored in the request queue.

[0039] In some implementations, the request queue may include multiple sub-queues. When storing inference requests, the request queue can store inference requests in different sub-queues according to the status of the inference request. The status of the inference request can be the inference stage it is in, which includes a pre-filling stage and a decoding stage. When storing inference requests according to their status, for example, inference requests that have not been pre-filled can be stored in one sub-queue, while inference requests that have been pre-filled and are in the decoding stage can be stored in other sub-queues. Further, as mentioned above, this embodiment can implement preemptive scheduling of inference requests (the preempted resource can be GPU resource; when preemption occurs, a higher-priority inference request will preempt the GPU resource of a lower-priority inference request). Therefore, during the inference process of multiple inference requests, there may be inference requests with preempted resources. In this embodiment, for inference requests with preempted resources, their corresponding KV cache will be offloaded from the GPU to the CPU (for inference requests without preempted resources, their corresponding KV cache is stored in the GPU). In this way, when storing inference requests, the request queue can also store inference requests in different sub-queues based on the location of the corresponding KV cache (or whether the inference request has been preempted). For example, inference requests with KV cache located on the CPU (i.e., inference requests with preempted resources) can be stored in one sub-queue, while inference requests with KV cache located on the GPU (i.e., inference requests without preempted resources) can be stored in other sub-queues.

[0040] In some implementations, the request queue may include three sub-queues, referred to as the first queue, the second queue, and the third queue. The first queue stores inference requests that have not been prefilled. Optionally, the first queue also stores inference requests that require recomputation after resource preemption (i.e., inference requests that need to be re-prefilled). Inference requests in the first queue do not occupy the key-value cache. The second queue stores inference requests that have been prefilled and are in the decoding stage. The inference requests in the second queue are inference requests that have not been preempted, and these requests have corresponding key-value caches stored in the GPU. The third queue stores inference requests that are in the decoding stage and have been preempted. Optionally, the inference requests in the third queue may be inference requests that do not require recomputation after resource preemption, and these requests have corresponding key-value caches stored in the CPU.

[0041] Step S306: Determine the priority of multiple inference requests based on the inference information of the multiple inference requests. The inference information includes the task difficulty, waiting time, and corresponding user level of the multiple inference requests.

[0042] After storing inference requests in a request queue, multiple inference requests in the queue can be scheduled for processing. In this embodiment, when scheduling multiple inference requests in the request queue, the priority of each request can be determined first based on its inference information, and then the requests can be scheduled for processing based on their priorities. For each inference request, its inference information includes at least the task difficulty, waiting time, and corresponding user level. This information can be carried in the inference request or obtained by the inference server through other means; no specific limitation is made here.

[0043] The task difficulty of an inference request characterizes the ease or difficulty of performing inference on that request. This task difficulty involves factors such as the prompt length (i.e., the length of the model input, such as the prompt "What is the square root of 4?" shown in Figure 1), the user-specified maximum output length (i.e., the length of the final output of the large model), the upper limit of the request length that the large model can infer, the remaining decoding attempts of the large model for the inference request, the GPU model and number used by the large model and inference service, and the size of the CPU cores and main memory. Among these, the GPU model and number used by the large model and inference service, and the size of the CPU cores and main memory are common environmental factors that do not affect the priority of the inference request and can be excluded. Therefore, in this embodiment, the task difficulty of an inference request mainly considers four characteristics: the prompt length, the user-specified maximum output length, the upper limit of the request length that the large model can infer, and the remaining decoding attempts of the large model for the inference request. In other words, the task difficulty of an inference request can be related to at least one of these four characteristics. Among these features, the length of the prompt word in the inference request and the maximum output length specified by the user are negatively correlated with task difficulty (because the longer the prompt word length and the maximum output length, the longer the inference request takes, and correspondingly, the higher the task difficulty). The upper limit of the request length that a large model can infer is determined by the properties of the large model, and is generally a fixed value. The remaining decoding attempts of the large model for the inference request are equal to the maximum output length specified by the user minus the number of generated tokens.

[0044] The inference request waiting time can be the time a inference request waits to be inferred, and can characterize the starvation level of the inference request. In this embodiment, the inference request waiting time can be expressed as:

[0045] T1 = now - arrival_time. Where now represents the current time and arrival_time represents the time when the inference request reaches the inference server.

[0046] The user level corresponding to the inference request is the level of the user who sent the inference request. Generally, user attributes (such as gender, occupation, etc.) can reflect some characteristics of user requests. In this embodiment, users can be divided into different user levels according to different attributes. The higher the user level, the higher the priority of the corresponding inference request. That is to say, inference requests from high-level users can be given higher priority. The number of user levels can be determined according to actual needs and is not specifically limited here.

[0047] When determining the priority of multiple inference requests based on inference information from multiple inference requests, in some implementations, for each inference request, the following steps S3061 to S3064 may be included:

[0048] Step S3061: Determine the first priority based on the task difficulty of the reasoning request.

[0049] As mentioned earlier, the task difficulty of an inference request is related to at least one of four characteristics: the length of the prompt words in the inference request, the maximum output length specified by the user, the upper limit of the request length that the large model can infer, and the remaining decoding attempts of the large model for the inference request. Therefore, when determining the first priority, it can be determined based on at least one of these four characteristics. Furthermore, since the scheduling granularity of the inference request in this embodiment is iterative, and the task difficulty of the inference request varies at different inference stages, the inference stage of the inference request also needs to be considered when determining the first priority.

[0050] In some implementations, determining the first priority based on the task difficulty of the inference request may include: determining the state of the inference request, which includes whether the inference request has not been pre-filled or has been pre-filled and is in the decoding stage; when the inference request is in the state of not being pre-filled, determining the first priority based on the length of the prompt words of the inference request, the maximum output length specified by the user, and the upper limit of the request length that the large model can infer; when the inference request is in the state of being pre-filled and is in the decoding stage, determining the first priority based on the remaining decoding attempts of the large model for the inference request and the upper limit of the request length that the large model can infer.

[0051] In some implementations, for inference requests that have not been pre-filled, since no lexical model inference has been performed yet, determining the first priority requires considering the length of the prompt word, the user-specified maximum output length, and the upper limit of the request length that the large model can infer. For inference requests that have been pre-filled and are in the decoding stage, since these requests have already been pre-filled, determining the first priority only requires considering the remaining decoding attempts of the large model for the inference request and the upper limit of the request length that the large model can infer.

[0052] In some implementations, determining the first priority based on the length of the inference request prompt, the maximum output length specified by the user, and the upper limit of the request length that the large model can infer may include: determining a first ratio of the difference between the upper limit of the request length that the large model can infer and the length of the inference request prompt to the upper limit of the request length; determining a second ratio of the difference between the upper limit of the request length that the large model can infer and the maximum output length specified by the user to the upper limit of the request length; summing the first ratio and the second ratio according to the specified first coefficient and the second coefficient, and determining the first priority based on the summing result.

[0053] The first and second coefficients can be obtained from the fitting function of the inference task time versus the prompt word length and the maximum output length, and then their specific values ​​can be determined through experimental search. No specific limits are placed on the magnitude of the first and second coefficients here. When determining the first priority based on the prompt word length of the inference request, the user-specified maximum output length, and the upper limit of the request length that the large model can infer, the following formula can be used:

[0054] In the above formula, `prompt_len` represents the length of the prompt word in the inference request, `output_len` represents the maximum output length specified by the user, and `model_len` represents the upper limit of the request length that the large model can infer. Here, `alpha` is the first coefficient, and `beta` is the second coefficient. `alpha` and `beta` are hyperparameters related to the service deployment environment, and they are determined as follows: the range of values ​​for `alpha` and `beta` is obtained based on the fitting function of the inference task time with the prompt word length and the maximum output length, and then specific values ​​are determined through experimental search. Experiments show that `alpha` and `beta` have a certain generalization ability, that is, the same set of `alpha` and `beta` can be used in different inference environments without significantly affecting performance. Optionally, the value of `alpha` can be fixed at 1, and the value of `beta` can be between 3 and 5 or between 2 and 10.

[0055] In some implementations, determining the first priority based on the remaining number of decoding attempts for the inference request by the large model and the upper limit of the request length that the large model can infer may include: determining a third ratio of the difference between the upper limit of the request length that the large model can infer and the remaining number of decoding attempts for the inference request by the large model to the upper limit of the request length; and determining the first priority based on the third ratio and a specified third coefficient.

[0056] The third coefficient can be obtained by fitting a function to the reasoning task time, prompt word length, and maximum output length, and then determining its specific value through experimental search. The specific size of the third coefficient is not limited here. Optionally, in some implementations, the third coefficient can be the same as the second coefficient described above.

[0057] When determining the first priority based on the remaining decoding attempts for the inference request of the large model and the upper limit of the request length that the large model can infer, the following formula can be used:

[0058] In the above formula, `model_len` represents the upper limit of the request length that the large model can infer, and `remain_iter_num` represents the remaining decoding attempts of the large model for the inference request, which is equal to the maximum output length specified by the user minus the number of tokens already generated. `beta` is the third coefficient, the same as the second coefficient, and is a hyperparameter related to the service deployment environment. Optionally, the value of `beta` can be between 3 and 5 or between 2 and 10.

[0059] Based on the above-described method for determining the first priority, it can be determined using the following formula when determining the first priority according to the difficulty of the reasoning request:

[0060] It should be noted that the function used to determine the first priority is a linear function. In other possible implementations, the function used to determine the first priority can also be other functions, such as inverse functions, trigonometric functions, exponential functions, logarithmic functions, etc., without specific limitations here.

[0061] Step S3062: Determine the second priority based on the waiting time of the inference request and the first parameter.

[0062] The first parameter is a hyperparameter. When determining the second priority based on the inference request's wait time and the first parameter, it can be determined using the following formula:

[0063] priority_2 = b * (now - arrival_time), where b is the first parameter and now - arrival_time is the waiting time for the inference request.

[0064] Step S3063: Determine the third priority based on the user level and the second parameter corresponding to the inference request.

[0065] The second parameter is a hyperparameter. When determining the third priority based on the user level corresponding to the inference request and the second parameter, it can be determined using the following formula:

[0066] priority_3 = c * n, where c is the second parameter and n is the user level corresponding to the inference request. Different values ​​of n can correspond to different user levels. For example, the range of n can be {1, 2, 3, 4, 5}, indicating that the user level is divided into five levels. The larger the value of n, the higher the user level.

[0067] Step S3064: Determine the priority of the inference request based on the first priority, the second priority, and the third priority.

[0068] When determining the priority of inference requests, considering only the task difficulty of the inference requests can achieve the optimal overall performance of the inference service. However, under heavy service pressure, some long-sequence requests may exceed the user's maximum waiting time, resulting in a poor user experience for these requests. If only the waiting time is considered, the scheduling of inference requests will revert to the original first-come, first-served scheduling strategy. Therefore, this embodiment, when determining the priority of inference requests, comprehensively considers both performance and fairness. The priority of inference requests can be determined based on three factors: task difficulty, waiting time, and corresponding user level. Specifically, the priority of inference requests is determined according to a first priority, a second priority, and a third priority.

[0069] In some implementations, when determining the priority of a reasoning request based on a first priority, a second priority, and a third priority, the sum of the first priority, the second priority, and the third priority can be used to determine the priority of the reasoning request, as shown in the following formula:

[0070] priority=priority_1+priority_2+priority_3=priority_1+b*(now-arrival)+c*n.

[0071] As can be seen from the above formula, by adjusting the two hyperparameters b and c (i.e., the first and second parameters mentioned above), the influence of task difficulty, waiting time, and user level on priority can be changed. To flexibly adjust system performance and fairness, this embodiment sets a time tolerance threshold (starved_threshold). Parameters b and c are related to the time tolerance threshold, which is adjustable. Adjusting the time tolerance threshold adjusts parameters b and c. To ensure system performance and fairness, when adjusting parameters b and c by adjusting the time tolerance threshold, the priority of multiple inference requests determined by the adjusted parameters b and c must satisfy at least one of the following two rules: When waiting times are the same, the higher the user level corresponding to the inference request, the higher the priority; when user levels are the same, if the waiting time of an inference request exceeds the time tolerance threshold, its priority is higher than that of a newly arrived inference request.

[0072] In other words, when adjusting parameters b and c using the time tolerance threshold, regardless of the adjustment, it must be ensured that: for inference requests of the same user level, those whose waiting time exceeds the time tolerance threshold have a higher priority than newly arriving inference requests, and / or, for inference requests with the same waiting time, inference requests from users with higher user levels have a higher priority. From this, the values ​​of parameters b and c can be derived. For ease of understanding, the derivation process of parameter b will be explained below using parameter b as an example.

[0073] As mentioned earlier, for inference requests of the same user level, once the waiting time exceeds the time tolerance threshold `starved_threshold`, the request has a higher priority than a newly arrived inference request of the same user level; that is, `p_starved` must be greater than `p_new`. The formulas for `p_starved` and `p_new` are as follows:

[0074] p_starved=priority_1+priority_2+priority_3=x+b*starved_threshold+c*n.

[0075] Where x is the value obtained by substituting prompt_len and output_len from the inference request into priority_1. Since it is assumed here that it can be any request, the value range of x is (0, alpha + beta).

[0076] p_new=priority_1+priority_2+priority_3=y+c*n.

[0077] Similarly, y is the value obtained by substituting prompt_len and output_len from the inference request into priority_1. Since the waiting time of the inference request is 0, priority_2 = 0. Since it is assumed to be an arbitrary request, the range of y is (0, alpha + beta).

[0078] Since p_starved needs to be greater than p_new, i.e., x + b*starved_threshold + c*n needs to be greater than y + c*n, we can obtain the formula b*starved_threshold > yx. Since the formula b*starved_threshold > yx holds for any x and y, we can deduce that b*starved_threshold > max(yx). Since max(yx) equals alpha + beta, we can further deduce that b*starved_threshold >= alpha + beta. Given starved_threshold, simply setting b = (alpha + beta) / starved_threshold will ensure that inference requests of the same user level, after waiting for more than starved_threshold, have a higher priority than newly arriving inference requests. Therefore, we can deduce the value of parameter b, i.e., b = (alpha + beta) / starved_threshold.

[0079] Based on the same principle, the value of parameter c can be derived, and the derivation process of parameter c will not be explained in detail here. The derived value of parameter c is (alpha+beta)(1+1 / starved_threshold).

[0080] After obtaining the values ​​of parameters b and c, the complete form of the priority function for the inference request can be obtained:

[0081] Among them, b=(alpha+beta) / starved_threshold, c=(alpha+beta)(1+1 / starved_threshold).

[0082] The aforementioned priority function comprehensively considers three factors: performance (determined by task difficulty), waiting time, and user level. It uses a time tolerance threshold (starved_threshold) to control the system's preference between performance and fairness. Generally, the higher the time tolerance threshold, the lower the proportion of waiting time in the priority function. When the time tolerance threshold is low, the scheduling strategy favors first-come, first-served, emphasizing fairness. When the time tolerance threshold is high, scheduling prioritizes efficiency. Furthermore, this priority function also ensures that requests from higher-level users have higher priority. In practical applications, if fairness is preferred, the time tolerance threshold can be set lower, such as 120ms. If the number of inference requests is large and system performance is prioritized, the time tolerance threshold can be set higher, such as 1200ms. Of course, 120ms and 1200ms are merely illustrative examples. In implementation, the time tolerance threshold can be flexibly set according to actual business needs; no specific limit is placed on the size of the time tolerance threshold here.

[0083] Step S308: Based on the priority of multiple inference requests, select at least one inference request from the multiple inference requests to perform model inference in one inference phase.

[0084] After determining the priorities of multiple inference requests in the request queue using the method described in S306 above, at least one inference request can be selected from the multiple inference requests for a single inference phase model inference. For example, one or more inference requests can be selected sequentially in descending order of priority for a single inference phase model inference. Since the priority comprehensively considers the task difficulty, waiting time, and corresponding user level of the inference request, selecting at least one inference request for inference based on priority not only enables preemptive scheduling of inference requests but also balances user needs and resource utilization, thereby improving the overall performance and stability of the system. Furthermore, when processing the selected at least one inference request, specifically performing a single inference phase model inference, such as a pre-filling or decoding, iterative priority scheduling of inference requests can be achieved, further optimizing resource allocation, reducing response time, and improving overall service quality.

[0085] In some implementations, as described above, the request queue may include a first queue, a second queue, and a third queue. The first queue stores inference requests that have not been pre-filled, the second queue stores inference requests that have been pre-filled and are in the decoding stage, and the third queue stores inference requests that are in the decoding stage and have had their resources preempted. Thus, when selecting at least one inference request from multiple inference requests for a single inference stage based on their priorities, the process may include the following steps: determining whether the third queue is empty; if the third queue is empty, selecting at least one inference request from the first queue for a single pre-filling stage model inference based on their priorities; if the third queue is not empty, selecting at least one inference request from the second queue for a single decoding stage model inference based on their priorities.

[0086] This application embodiment allows setting corresponding priorities for a first queue, a second queue, and a third queue. When setting queue priorities, to improve the processing efficiency of the first word unit (the word unit output during the pre-filling stage), shorten latency, and optimize resource allocation, the first queue can be set to have the highest priority when no resource preemption occurs. Furthermore, to ensure that inference requests in the second and third queues can be processed, and to further optimize resource allocation, the second and third queues can be set to have a higher priority than the first queue when resource preemption occurs. Further, considering that the inference requests stored in the second queue are those without preempted resources, while the inference requests stored in the third queue are those with preempted resources, and that during resource preemption, high-priority inference requests typically preempt the resources of low-priority inference requests (ensuring that high-priority inference requests can be processed first), the second queue can also be set to have a higher priority than the third queue when resource preemption occurs. Thus, when scheduling inference requests in these three queues, the overall scheduling logic can be as follows: If no resource preemption occurs, inference requests in the first queue are processed first. If the first queue is empty (i.e., the number of inference requests in the first queue is 0), then inference requests in the second queue are processed. If resource preemption occurs, inference requests in the second queue are processed first, then preempted requests in the third queue are processed, and finally requests in the first queue are processed.

[0087] Based on the overall scheduling logic for the first, second, and third queues described above, when selecting at least one inference request for a model inference phase based on priority, it can first be determined whether the third queue is empty. If the third queue is empty, meaning the number of inference requests stored in the third queue is 0, it indicates that no resource preemption has occurred. In this case, at least one inference request can be selected from the first queue based on its priority for a model inference phase pre-filling. If the third queue is not empty, meaning the number of inference requests stored in the third queue is not 0, it indicates that resource preemption has occurred, and the inference request for the preempted resource has been stored in the third queue. In this case, at least one inference request can be selected from the second queue based on its priority for a model inference phase decoding.

[0088] When selecting at least one inference request from the first queue for a pre-filling phase model inference based on the priority of the inference requests in the first queue, some implementations may include the following steps: prioritizing the inference requests in the first queue according to their priority; starting with the inference request with the highest priority, sequentially determining whether the inference request can be inferred in this round; and performing a pre-filling phase model inference on at least one inference request that can be inferred in this round.

[0089] When determining whether a certain inference request can be inferred in the current round, starting from the highest priority inference request, taking the highest priority inference request in the first queue as an example, the following steps may be included: determining whether the inference request meets the first condition; if the inference request meets the first condition, determining that the inference request can be inferred in the current round.

[0090] The first condition is the condition that a reasoning request must meet when the current round is able to reason about it. In some implementations, the first condition may include at least one of the following: the number of requests already accommodated in the current round is less than the maximum number of requests that the current round can accommodate; the number of tokens already accommodated in the current round is less than the maximum number of tokens that the current round can accommodate; and there are sufficient reasoning resources to reason about the reasoning request, wherein reasoning resources may include, but are not limited to, GPU resources.

[0091] It should be noted that, in some implementations, in order to ensure that the inference request can be executed smoothly, the inference request can only be considered to be able to infer the inference request in this round if the inference request meets the three conditions in the first condition mentioned above; otherwise, the inference request can be considered to be unable to infer the inference request in this round.

[0092] For the highest priority inference request in the first queue, if it is determined that the inference request can be inferred in this round according to the above method, the second highest priority inference request can be judged according to the same method. If the second highest priority inference request can be inferred in this round, the third highest priority inference request can be judged according to the same method, and so on, until a certain inference request cannot be inferred in this round (after each judgment on whether inference is possible, if a certain inference request can be inferred, the corresponding inference resources will be deducted. Therefore, in the process of cyclic judgment, the inference resources are constantly decreasing).

[0093] After performing the above cyclical judgment on the inference requests in the first queue, for one or more inference requests that can be inferred, a pre-filling phase of model inference can be performed in this round; for one or more inference requests that cannot be inferred, they can wait for the next inference phase of model inference.

[0094] It should be noted that for the highest priority inference request in the first queue, after the above judgment, the result may be that the inference request cannot be inferred in this round. In this case, the above loop judgment operation can be terminated, and it can be determined that there are no inference requests that can be inferred in the first queue, that is, the number of inference requests that can be inferred in this round in the first queue is 0. When the number of inference requests that can be inferred in this round in the first queue is 0, in some embodiments, the inference requests in the second queue can be processed, that is, according to the priority of the inference requests in the second queue, at least one inference request can be selected from the second queue for model inference in the decoding stage. This can avoid resource waste and improve resource utilization and processing efficiency.

[0095] The above examples illustrate how to select at least one inference request from the first queue for a pre-filling phase model processing when the third queue is empty, i.e., no resource preemption has occurred. It should be noted that in some implementations, there may be certain preconditions when selecting at least one inference request from the first queue for a pre-filling phase model inference. If the third queue is determined to be empty, this precondition can be checked first, and then the decision on whether to select at least one inference request from the first queue for a pre-filling phase model inference can be based on the check result. In one implementation, if the third queue is determined to be empty, before selecting at least one inference request from the first queue for a pre-filling phase model inference based on the priority of the inference requests in the first queue, it can be checked whether the first queue meets a second condition. The second condition may include the first queue not being empty and / or the current scheduling interval threshold of the first queue being reached. If the first queue meets the second condition, such as the first queue not being empty, or the first queue not being empty and the current scheduling interval threshold of the first queue being reached, then at least one inference request can be selected from the first queue for a pre-filling phase model inference based on the priority of the inference requests in the first queue. If the first queue does not meet the second condition, such as the first queue being empty, or the first queue not being empty but not yet reaching the scheduling interval threshold of the first queue, then no scheduling processing will be performed on the first queue.

[0096] Optionally, if the first queue does not meet the second condition, the inference requests in the second queue can be scheduled, that is, according to the priority of the inference requests in the second queue, at least one inference request can be selected from the second queue for model inference in the decoding stage.

[0097] It's important to note that the reason the first queue is not empty is used as the second condition is that if the first queue is empty, there are no inference requests, and scheduling cannot be performed on it. The reason the scheduling interval threshold for the first queue is used as the second condition is that, without resource preemption, the overall scheduling logic for the first, second, and third queues prioritizes inference requests in the first queue. This could lead to inference requests in the second queue remaining unprocessed for a period, resulting in longer processing delays and impacting user experience. To avoid this issue, before selecting at least one inference request from the first queue for a pre-filling phase model inference based on its priority, it's necessary to determine if the scheduling interval threshold for the first queue has been reached. If it has, the inference requests in the first queue can be scheduled; otherwise, they can be ignored, and instead, the inference requests in the second queue can be scheduled. That is, at least one inference request from the second queue can be selected for a decoding phase model inference based on its priority. This avoids the problem of inference requests in the second queue remaining unprocessed when no resource preemption occurs. The scheduling interval threshold for the first queue can be determined based on actual needs and is not specifically limited here.

[0098] In this embodiment, if the third queue is not empty, or the third queue is empty and the first queue is also empty, or the third queue is empty and the number of inference requests that can be inferred in this round in the first queue is 0, or the third queue is empty and the first queue does not meet the second condition, the inference requests in the second queue can be scheduled. That is, according to the priority of the inference requests in the second queue, at least one inference request is selected from the second queue for a decoding phase model inference. In some implementations, when selecting at least one inference request from the second queue for a decoding phase model inference based on the priority of the inference requests in the second queue, the following steps may be included: prioritizing the inference requests in the second queue according to their priority; starting with the highest priority inference request, sequentially determining whether the inference resource can infer the inference request; if the inference resource cannot infer the inference request, performing resource preemption starting with the lowest priority inference request in the second queue until the inference resource can infer the inference request; and performing a decoding phase model inference on at least one inference request that the inference resource can infer.

[0099] Inference resources may include, but are not limited to, GPU resources. Since the inference requests in the second queue have been pre-filled and are in the decoding stage, and the main factor affecting whether an inference request can be inferred in this round is whether there are enough inference resources, when determining whether the inference requests in the second queue can be inferred in this round, we specifically determine whether the inference resources are sufficient to infer the inference requests in the second queue, that is, whether the inference resources can meet the inference resource requirements of the inference requests in the second queue.

[0100] When determining whether an inference resource can infer inference requests in the second queue, the process begins with the highest-priority inference request in the second queue. If the inference resource can infer the highest-priority request, the process continues to determine if it can infer the second-highest-priority request. If the inference resource cannot infer the highest-priority request, resource preemption occurs. This involves preempting resources from the lowest-priority request in the second queue, in ascending order of priority, for the remaining inference requests (the preempted resources will be used to infer the highest-priority request). If, after preemption, the inference resource can infer the highest-priority request, the process continues to determine if it can infer the second-highest-priority request. If, after preemption, the inference resource still cannot infer the highest-priority request, the process ends, and the determination of whether the inference resource can infer other inference requests in the second queue is stopped. In this case, the number of requests that can be inferred in the second queue in this round is determined to be 0.

[0101] When determining whether the inference resource can infer the second highest priority inference request, if the inference resource can infer the second highest priority inference request, then the determination continues to determine whether the inference resource can infer the third highest priority inference request. If the inference resource cannot infer the second highest priority inference request, then resource preemption is performed according to the method described above until the inference resource can infer the second highest priority inference request.

[0102] The above example uses the highest and second highest priority inference requests in the second queue to illustrate how to determine whether an inference resource can infer the inference requests in the second queue. Other inference requests in the second queue can be judged using the same method, and will not be repeated here.

[0103] Starting with the highest priority inference request, after sequentially determining whether the inference resource can infer the inference requests in the second queue, the determination results may include the following:

[0104] In the first scenario: no resource preemption occurs, and the inference resource can infer all inference requests in the second queue.

[0105] The second scenario: resource preemption occurs, and the inference resource is able to infer some of the inference requests in the second queue.

[0106] The third scenario: resource preemption occurs, and the inference resources are still unable to infer the inference requests in the second queue, meaning that the number of requests in the second queue that can be inferred in this round is 0.

[0107] Regarding the first scenario described above, since the inference resources are sufficient to infer all inference requests in the second queue (i.e., the inference resources can meet the resource requirements of the inference requests in the second queue), all inference requests in the second queue can undergo a decoding phase model inference in this round. Furthermore, since no resource preemption occurs, there may be remaining inference resources. To avoid resource waste, in some implementations, the inference requests in the third queue can be scheduled. Specifically, based on the priority of the inference requests in the third queue, at least one inference request is selected from the third queue for a decoding phase model inference. This avoids resource waste and improves resource utilization.

[0108] Regarding the second scenario, since the inference resources can only infer some of the inference requests in the second queue, meaning the inference resources can only meet the inference resource requirements of some of the inference requests in the second queue, for the inference requests in the second queue that can be inferred, a model inference in the decoding phase can be performed in this round. For the inference requests that cannot be inferred, they can wait for the next round of model inference.

[0109] Regarding the third scenario, since the number of requests in the second queue that can be inferred in this round is 0, model inference can be skipped for these requests, and they can wait for the next round of model inference. Optionally, considering that different inference requests may require different inference resources at different inference stages, if inference resources cannot infer the requests in the second queue, they may be able to infer the requests in the third queue. Therefore, to avoid resource waste, when the number of requests in the second queue that can be inferred in this round is 0, the inference requests in the third queue can be scheduled. That is, based on the priority of the inference requests in the third queue, at least one inference request is selected from the third queue for one decoding stage model inference. This avoids resource waste and improves resource utilization.

[0110] In response to the second and third scenarios mentioned above, in the event of resource preemption, for any inference request whose resource has been preempted, in some implementations, any of the following operations can be performed: store the inference request in a third queue; unload the KV cache occupied by the inference request from the GPU to the CPU; or store the inference request in a first queue; release the KV cache occupied by the inference request; save the tokens generated by the inference request to the CPU.

[0111] In one embodiment, for inference requests with preempted resources, this application provides two preemption modes: a recomputation mode and a replacement mode. The recomputation mode involves moving the preempted inference request from the second queue back to the first queue and releasing the KV cache occupied by the inference request, while retaining the generated tokens (which can be stored in the CPU). These tokens will then wait for scheduling in the first queue, regenerate the KV cache, and be added back to the second queue for the decoding stage. The process is illustrated in Figure 4. The replacement mode involves moving the preempted inference request from the second queue to the third queue. The occupied KV cache is unloaded from the GPU to the CPU. This clears some GPU resources for inference requests that preempt resources, while the generated tokens and KV cache of the preempted inference request are retained in the CPU. Subsequently, as some inference requests in the second queue are completed, the corresponding KV cache can be released, leaving some GPU resources available. Inference requests in the third queue can then be moved back to the second queue for decoding, and the occupied KV cache will be loaded from the CPU to the GPU. The process is illustrated in Figure 5.

[0112] Based on the two preemption modes provided in this application embodiment, when resource preemption occurs, for the inference request of the preempted resource, one of the preemption modes can be selected. If the replacement mode is selected, the inference request can be stored in the third queue, and then the KV cache occupied by the inference request can be unloaded from the GPU to the CPU. If the recomputation mode is selected, the inference request can be stored in the first queue, and then the KV cache occupied by the inference request can be released, while the tokens generated by the inference request can be saved to the CPU. The selection of the preemption mode can be based on the actual situation, and no specific limitation is made here on how to select the preemption mode. Optionally, in some embodiments, the preemption mode can be selected based on the current hardware resources. For example, if the hardware's computing power is more advantageous than its bandwidth (e.g., high computing power, low bandwidth graphics card), making recomputation more efficient than replacement, then the recomputation mode is used. If the hardware's bandwidth is more advantageous than its computing power (e.g., low computing power, high bandwidth graphics card), then the replacement mode is used.

[0113] The above examples illustrate how to select at least one inference request from the second queue for a single decoding phase model processing operation when the third queue is not empty, i.e., resource preemption is currently occurring. It should be noted that in some implementations, there may be certain preconditions when selecting at least one inference request from the second queue for a single decoding phase model processing operation. If the third queue is determined to be not empty, this precondition can be checked first, and then the decision on whether to select at least one inference request from the second queue for a single decoding phase model inference operation can be based on the check result. In one implementation, if the third queue is determined to be not empty, before selecting at least one inference request from the second queue for a single decoding phase model inference operation based on the priority of the inference requests in the second queue, it can be checked whether the second queue meets a third condition. Optionally, the third condition may include the second queue being not empty. If the second queue meets the third condition, for example, if the second queue is not empty, then at least one inference request can be selected from the second queue for a single decoding phase model inference operation based on the priority of the inference requests in the second queue. If the second queue does not meet the third condition, for example, if the second queue is empty, then scheduling of the second queue is not required.

[0114] If the third condition is not met in the second queue, the inference requests in the third queue can be processed, that is, at least one inference request can be selected from the third queue for model inference in the decoding stage according to the priority of the inference requests in the third queue, so as to avoid wasting resources.

[0115] In this embodiment, if the third queue is not empty, and if no resource preemption occurs in the inference requests in the second queue, or the number of requests that can be inferred in the second queue is 0, or the second queue does not meet the third condition, then the third queue can be scheduled. That is, based on the priority of the inference requests in the third queue, at least one inference request is selected from the third queue for one decoding phase model inference. In some implementations, selecting at least one inference request from the third queue for one decoding phase model inference based on the priority of the inference requests in the third queue may include the following steps:

[0116] Based on the priority of the inference requests in the third queue, the inference requests in the third queue are prioritized; starting with the highest priority inference request, it is determined in turn whether the inference resource can infer the inference request; if the inference resource can infer the inference request, the KV cache corresponding to the inference request is loaded from the CPU to the GPU; and a model inference in the decoding stage is performed for at least one inference request that the inference resource can infer.

[0117] Inference resources may include, but are not limited to, GPU resources. Since the inference requests in the third queue have been pre-filled and are in the decoding stage, and the main factor affecting whether an inference request can be inferred in this round is whether there are enough inference resources, when determining whether the inference requests in the third queue can be inferred in this round, we specifically determine whether the inference resources are sufficient to infer the inference requests in the third queue, that is, whether the inference resources can meet the inference resource requirements of the inference requests in the third queue.

[0118] When determining whether an inference resource can infer a request in the third queue, the process starts with the highest-priority inference request in the third queue. It checks if the inference resource can infer that request. If it can, it checks if the resource can infer the second-highest priority request, and so on, until the resource can no longer infer a request. (Each time a check is performed, if a request can be inferred, the corresponding inference resource is deducted; therefore, the inference resource decreases continuously during the iterative process.)

[0119] After performing the aforementioned iterative checks on the inference requests in the third queue, for one or more inference requests that can be inferred, a model inference during the decoding phase can be performed in this round. For one or more inference requests that cannot be inferred, they can wait for the next round of model inference. Specifically, during the model inference during the decoding phase for inference requests that can be inferred, based on the replacement mode for resource preemption described above, the KV cache corresponding to the inference request can be loaded from the CPU to the GPU for continued decoding.

[0120] The above details how to select at least one inference request from multiple inference requests in the request queue (including the first queue, the second queue, and the third queue) for model inference in one inference phase. To facilitate understanding of the scheduling logic for inference requests in the request queue in this application's embodiments, an example implementation will be described below, please refer to Figure 6. Figure 6 is a flowchart illustrating the scheduling of inference requests in the request queue according to an embodiment of this application. The implementation of the embodiment shown in Figure 6 is as follows.

[0121] Step S1: Determine if the third queue is empty.

[0122] When scheduling inference requests in the first, second, and third queues, based on the overall scheduling logic of the three queues (in the absence of resource preemption, inference requests in the first queue are processed first, followed by those in the second and third queues; in the presence of resource preemption, inference requests in the second queue are processed first, followed by preempted inference requests in the third queue, and finally inference requests in the first queue), it can be determined first whether the third queue is empty.

[0123] If the third queue is empty, it means that no resource preemption has occurred, and step S2 can be executed. If the third queue is not empty, it means that resource preemption has occurred, and step S3 can be executed.

[0124] Step S2: Determine whether the current time has reached the scheduling interval threshold of the first queue and whether the first queue is not empty.

[0125] If the scheduling interval threshold of the first queue has been reached and the first queue is not empty, then the inference requests in the first queue can be scheduled. If the scheduling interval threshold of the first queue has not been reached or the first queue is empty, then step S3 can be executed.

[0126] When scheduling inference requests in the first queue, the priorities of these requests can be calculated and sorted in descending order of priority. The priority of each request in the first queue can be calculated using the following priority function:

[0127] Among them, b=(alpha+beta) / starved_threshold, c=(alpha+beta)(1+1 / starved_threshold).

[0128] After calculating the priority of the inference requests in the first queue and sorting them in descending order according to priority, the inference requests in the first queue can be iterated through to determine whether the inference resource is sufficient to infer the current request. If yes, the current request is added to the pending inference queue and resources are allocated to it, and the loop continues; otherwise, the loop exits. The pending inference queue is used to store requests that need to be inferred in this round.

[0129] After performing the aforementioned iterative checks on the inference requests in the first queue, we can further determine whether the inference queue is not empty. If the inference queue is not empty, it means that the number of requests that can be inferred in this round in the first queue is not 0. In this case, the inference queue can be output, and the inference requests in the inference queue will be subjected to a pre-filling phase of model inference by the inference server. If the inference queue is empty, it means that the number of requests that can be inferred in this round in the first queue is 0. In this case, we can execute step S3 to avoid wasting resources.

[0130] Step S3: Determine whether the second queue is not empty.

[0131] If the second queue is empty, meaning the number of inference requests in the second queue is 0, then step S4 can be executed to avoid resource waste. If the second queue is not empty, meaning the number of inference requests in the second queue is not 0, then the priorities of the inference requests in the second queue can be calculated and sorted in descending order of priority. The priority of the inference requests in the second queue can be calculated using the following priority function:

[0132] Among them, b=(alpha+beta) / starved_threshold, c=(alpha+beta)(1+1 / starved_threshold).

[0133] After calculating the priority of inference requests in the second queue and sorting them in descending order according to priority, the inference requests in the second queue can be iterated to determine whether the inference resource is available for the current request. If so, the current request is added to the pending inference queue and resources are allocated to it, and the loop continues. If not, the lowest priority request in the second queue is preempted in turn, and the cache of the preempted request is released until the current request can be inferred. For inference requests with preempted resources, they can be stored in the third queue, and the KV cache occupied by the inference request is unloaded from the GPU to the CPU; or they can be stored in the first queue, and the KV cache occupied by the inference request is released, and the tokens generated by the inference request are saved to the CPU (the corresponding processing flow is not shown in Figure 6).

[0134] After performing the aforementioned iterative checks on the inference requests in the second queue, it can be further determined whether resource preemption has occurred in this round. If resource preemption has occurred, it means that there are no remaining inference resources to infer the inference requests in the third queue. In this case, a pending inference queue can be output, and the inference requests in the pending inference queue will be subjected to model inference during the decoding phase by the inference server. If no resource preemption has occurred, it means that there may still be inference resources remaining. To avoid resource waste, step S4 can be executed.

[0135] Step S4: Calculate the priority of inference requests in the third queue and sort them in descending order of priority.

[0136] When calculating the priority of inference requests in the third queue, the following priority function can be used:

[0137] Among them, b=(alpha+beta) / starved_threshold, c=(alpha+beta)(1+1 / starved_threshold).

[0138] After calculating the priority of inference requests in the third queue and sorting them in descending order according to priority, the inference requests in the third queue can be iterated through to determine whether the inference resources are sufficient to infer the current request. If so, the key-value cache of the current request is loaded from the CPU to the GPU, and the current request is added to the inference queue and resources are allocated to it, continuing the loop. If not, the loop is exited and the inference queue is output. The inference requests in the inference queue will be subjected to model inference during the decoding phase by the inference server.

[0139] After performing one round of model inference based on the process shown in Figure 6, the next round of model inference can be performed based on the process shown in Figure 6. This enables iterative priority scheduling of inference requests, thereby optimizing resource allocation, reducing response time, and improving overall service quality.

[0140] In some implementations, after selecting at least one inference request from multiple inference requests to perform model inference in one inference phase, the following operations may be performed: determine whether at least one inference request has completed model inference; for any inference request, perform the following operations: if the inference request has completed model inference, release the KV cache corresponding to the inference request; if the inference request has not completed model inference, put the inference request into the request queue for model inference in the next inference phase.

[0141] When determining whether at least one inference request has completed model inference, taking a single inference request as an example, one could check if an end marker was encountered during model inference for that request. If so, the model inference can be considered complete. Alternatively, one could check if the output obtained in this inference phase has reached the user-specified output length. If so, the model inference can be considered complete. No specific limitations are made here on how to determine whether an inference request has completed model inference.

[0142] After determining whether at least one inference request has completed model inference, for any one of these inference requests, if model inference has been completed, the KV cache corresponding to that inference request can be released, i.e., the GPU resources occupied by that inference request can be released. If model inference has not been completed, the inference request can be placed in the request queue and model inference for the next inference stage can be performed. For example, after pre-filling an inference request, if the inference request has not completed model inference, the inference request can be placed in the second queue of the request queue (the second queue is used to store inference requests that have completed pre-filling and are in the decoding stage) for decoding. Alternatively, after decoding an inference request, if the inference request has not completed model inference, the inference request can be returned to the second queue of the request queue for the next decoding. When performing model inference for the next inference stage, the priorities of multiple inference requests in the request queue can be recalculated, and at least one inference request can be selected from the multiple inference requests according to the priority for model inference in the next inference stage. The implementation method can be found in the relevant content described in S306 and S308 above, and will not be described in detail here. This allows for iterative priority scheduling of inference requests, which can further optimize resource allocation, reduce response time, and improve overall service quality.

[0143] Based on the content described in S302 to S308 above, it can be seen that the technical solution provided by the embodiments of this application is as follows:

[0144] (1) A comprehensive priority scheduling algorithm is proposed. This algorithm takes into account factors such as the waiting time of inference requests, the corresponding user level, and the difficulty of tasks. The algorithm is optimized through a large number of experiments to make it more flexible and generalizable to various actual production environments.

[0145] (2) A scheduling algorithm that considers a time tolerance threshold (starvation tolerance threshold) is proposed. In one embodiment, the present application introduces a time tolerance threshold into the priority function to avoid user starvation, while allowing the service provider to adjust its preference for efficiency and fairness according to demand.

[0146] (3) A multi-scenario generalized adaptive scheduling algorithm is proposed. In one embodiment, the priority function of this application includes multiple hyperparameters, which can be derived with matching values ​​based on different scenarios and experimental data. By adjusting these hyperparameters, a flexible balance between efficiency and fairness can be achieved to adapt to the needs of various scenarios.

[0147] (4) An iterative scheduling method is proposed. In one embodiment, the present application focuses on the autoregressive characteristics of large language models and realizes iterative scheduling. In each round, requests are calculated according to priority, and then requests for inference in this round are selected according to priority. The inference results of each round are returned in a streaming manner, thereby optimizing resource allocation, improving user experience, and significantly reducing overall response latency.

[0148] Based on the above four points, the embodiments of this application can significantly improve the system's response speed and resource utilization during large model inference, while taking into account both performance and fairness, and adapting to various real-world production environments.

[0149] To facilitate understanding of the technical solutions provided in the embodiments of this application, specific scenarios will be illustrated below, with corresponding scenario data shown in Table 1. Given the large size of the complete request data, the specific data for all inference requests is not shown here; only key data is listed: the unique identifier request_id for each request (which can be generated by the user interface of the inference server), user level, the length of the request input prompt_len, and the length of the request output output_len. A complete example of a user-sent request is also shown.

[0150] Table 1

[0151] An example of an inference request is as follows:

[0152] The specific process for processing inference requests is as follows:

[0153] Step 1000: Create a large model inference service on the artificial intelligence platform.

[0154] Step 1001: The user sends an inference request to the large model inference service from a web page on the user's computer.

[0155] Step 1002: The large model inference service receives and parses the inference request.

[0156] Step 1003: Tokenize the prompt of the reasoning request (segmentation or compilation), that is, convert human-understandable characters into digital tokens that the large model can reason about.

[0157] Example: "prompt":"What should I do if I have trouble sleeping?" ---------tokenizer----->prompt_token_ids:[1,29871,234,160,164,234,159,163,30413,31076,233,131,145,31882,232,141,161,30882].

[0158] Step 1004: Place the inference request into the first queue to await scheduling and inference.

[0159] Step 1005: Scheduling and Reasoning.

[0160] First round of scheduling:

[0161] Step 100501: Calculate the priority of each inference request in the first queue. The calculation results are shown in Table 2. Here, starved_threshold = 120ms. For ease of representation, the priority function is scaled by a value of 4096.

[0162] Table 2

[0163] Step 100502: Sort the multiple inference requests in the first queue in descending order of priority. The sorting results are shown in Table 3.

[0164] Table 3

[0165] Step 100503: Loop through the inference requests in the first queue, determine whether the inference cache (inference resource) can infer the current inference request, if yes, add the current inference request to the queue to be inferred, and continue to loop through and determine other inference requests in the first queue; if no, exit the loop.

[0166] In this embodiment, nine inference requests can be selected from the first queue, as shown in Table 4.

[0167] Table 4

[0168] Step 100504: Call the model executor to perform inference. The inference process is shown in Figure 7. Each inference generates a token.

[0169] The states of the first, second, and third queues before and after prefilling the nine inference requests in the first queue are shown in Table 5.

[0170] Table 5

[0171] Step 100505: Determine if any inference requests have been completed in this round of inference. If so, remove this request from the first queue, globally update the request status, and reclaim resources. Place any incomplete requests from this round of inference into the second queue.

[0172] Second round of scheduling:

[0173] Step 100506: Determine whether the remaining inference resources can continue to be pre-filled. In this embodiment, we take the case where pre-filling cannot continue as an example. In this case, the second queue can be scheduled.

[0174] Step 100507: Calculate the priority of each inference request in the second queue.

[0175] Step 100508: Sort the inference requests in the second queue from highest to lowest priority.

[0176] Step 100509: Loop through the inference requests in the second queue and determine if the inference cache (inference resources) is sufficient to infer the current inference request. If yes, add the current inference request to the pending inference queue and continue looping; if no, preempt the lowest priority inference request in the second queue in turn, releasing the cache of the preempted inference request, until the current inference request can be inferred. If all inference requests with lower priority than the current inference request have been preempted and the resource requirements for inferring the current inference request are still not met, then the current inference request is also treated as preempted. In this embodiment, no preemption occurred in the second queue, and all 9 inference requests could be inferred.

[0177] Step 100510: Call the model executor to perform inference.

[0178] Continuing with the scheduling process described above, Table 6 lists the states of the first, second, and third queues during the scheduling process. The states of the first, second, and third queues in Table 6 are the states before this inference phase; after inference, the states change to the states in the next row.

[0179] Table 6

[0180] As can be seen from Table 6, no resource preemption occurred during the entire inference process. In fact, a careful analysis of the scheduling strategy reveals that the probability of preemption is not high (because during the decoding phase of model inference, the inference request generally requires fewer inference resources, so the probability of sending resource preemption is low).

[0181] Step 1006: Detokenize the reasoning request that has completed the reasoning process, that is, convert the digital token into characters that humans can understand.

[0182] Step 1007: Post-processing, packaging the inference request and the result of the inference request.

[0183] Step 1008: Return the reasoning results to the user.

[0184] Example of reasoning result:

[0185] There are many reasons for poor sleep, including lifestyle, diet, stress, and mental state. Here are some suggestions that may be helpful:

[0186] 1. Maintain a consistent sleep schedule: Go to bed at the same time every night and avoid changing it arbitrarily. This helps your body and mind establish a consistent sleep pattern.

[0187] 2. Create a comfortable sleep environment: Your bed and bedroom should be comfortable, quiet, and warm. You can use a soft mattress, soft sweater, and soft blankets to create a comfortable sleep environment.

[0188] 3. Avoid eating dinner and drinking alcohol: Eating dinner and drinking alcohol may make it difficult for you to sleep at night because they may stimulate your body and make it hard to fall asleep.

[0189] 4. Avoid using electronic devices: Using electronic devices, such as mobile phones, computers and televisions, may make it difficult for you to fall asleep because their screens and sounds may interfere with your sleep.

[0190] In summary, upon receiving an inference request, this embodiment adds the request to a request queue to store multiple inference requests. Then, based on the task difficulty, waiting time, and corresponding user level of each request, the priority of the multiple inference requests is determined. Based on the priority, at least one inference request is selected for a single inference phase. Since the priority of inference requests is determined by comprehensively considering the task difficulty, waiting time, and corresponding user level, and at least one inference request is selected for inference based on priority, preemptive scheduling of inference requests can be achieved, while also balancing user needs and resource utilization, thereby improving the overall performance and stability of the system. Furthermore, after selecting at least one inference request based on priority, a single inference phase can be performed during the inference process, such as pre-filling or decoding. This allows for iterative priority scheduling of inference requests, further optimizing resource allocation, reducing response time, and improving overall service quality. By optimizing the inference service scheduling mechanism, this embodiment significantly enhances the adaptability and reliability of the system, thereby promoting the widespread application and popularization of large models in various fields.

[0191] The request processing method provided in this application embodiment can be provided in the form of an inference service, which can be deployed in an artificial intelligence platform. In some implementations, when a user performs model inference, they can use a web application interface to upload the inference request to the inference service via the network. The inference service can use the method provided in this application embodiment to schedule the inference request and feed the inference result back to the user via the network. A corresponding system architecture (application environment) example can be shown in Figure 8.

[0192] This application also provides a request processing system. In some embodiments, the structure of the request processing system can be as shown in Figure 9. The request processing system shown in Figure 9 includes a user interface, a parameter module, and a Large Model Inference Engine (LLM Engine), which will be described in detail below.

[0193] The large model inference engine is a critical and core component of the system, utilizing iterative scheduling and efficient memory management to maximize service throughput. The large model inference engine is responsible for receiving inference requests and generating inference results, involving the scheduling and resource allocation of the entire inference process. Its specific functions include:

[0194] (1) Initialization: Configuration rationalization check, tokenizer model initialization, loading large model, request scheduling initialization, distributed parallel inference environment initialization, worker process initialization and cache management initialization, etc.

[0195] (2) The reasoning process of managing reasoning requests: add reasoning requests to the scheduler's request pool, call the scheduler to schedule reasoning requests, call the model executor to perform model reasoning, encapsulate the reasoning results, and return the reasoning results, etc.

[0196] (3) Resource management: Allocate and release CPU main memory and GPU video memory, and monitor the remaining status of CPU main memory and GPU video memory.

[0197] The large model inference engine manages the entire inference process and resource allocation. The two core functions of inference and resource management can be implemented by a distributed system and a central controller.

[0198] The distributed system is primarily responsible for functions such as distributed environment initialization, distributed parallel inference, model initialization, model loading, physical cache management, and model execution. Functions such as physical cache management and model execution are implemented through each worker process. A worker process is an abstraction of the hardware GPU; at the software level, it refers to a worker instance (each process on the GPU maintains its own worker instance). Each worker instance manages two important instances:

[0199] Model runner: Responsible for loading the model and performing inference.

[0200] The Cache Engine is responsible for managing the physical blocks of the key-value cache on the GPU and CPU. The cache manager in the scheduler determines the allocation and release of physical blocks, and the Cache Engine performs hardware data management based on the allocation results.

[0201] The central controller is primarily responsible for request scheduling, logical management and allocation of memory resources. It has two main components: a cache manager and a scheduler. The cache manager is responsible for allocating key-value cache physical blocks for inference requests, and logically manages these blocks virtually. The scheduler's main function is to determine which inference requests are passed to the model executor for model inference during an inference phase. The scheduler is the core component of this embodiment, and its scheduling logic can be found in the corresponding content of the embodiment shown in Figure 3, which will not be described in detail here.

[0202] In some implementations, the scheduler structure can be as shown in Figure 10. The scheduler shown in Figure 10 includes a scheduling budget, preemption mode, scheduling policy, and request queue. The scheduler needs to select requests from the request queue for the next inference round based on the scheduling budget, preemption mode, and scheduling policy. The scheduling budget records the maximum number of requests and terms that the current inference round can accommodate, as well as the number of requests and terms already accommodated in this round. The scheduling budget is responsible for determining whether a request can be executed in the current round based on these attribute values. It should be noted that whether the current request has sufficient resources to execute also requires calling functions in the cache manager to determine this.

[0203] Based on the request processing system shown in Figure 9, when the large model inference engine receives a new inference request, it places the request in the first queue to wait for pre-filling. After pre-filling, it enters the second queue for decoding in a loop until the process ends. The implementation process is as follows:

[0204] (1) The user calls the user interface to pass the request to the Large Model Inference Engine (LLM Engine).

[0205] (2) The large model inference engine receives the request, uses the scheduler to control the generation process, the cache manager manages the cache resources, the cache engine physically allocates resources, the model executor executes the model inference, and finally generates the result.

[0206] (3) Return the generated results to the user.

[0207] (4) All the parameters required for inference and resource allocation in the above process are defined in the parameter module, including model parameters, sampling parameters, request scheduling parameters, distributed parallel inference parameters, etc.

[0208] In summary, the technical solutions provided in this application, by optimizing the scheduling mechanism of inference services, can significantly improve resource utilization, reduce latency, enhance system adaptability and reliability, and thus promote the widespread application and popularization of large models in various fields.

[0209] The foregoing has described specific embodiments of this application. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims may be performed in a different order than that shown in the embodiments and may still achieve the desired results. Furthermore, the processes depicted in the drawings do not necessarily require the specific or sequential order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0210] Figure 11 is a schematic diagram of the structure of an electronic device according to an embodiment of this application. Referring to Figure 11, at the hardware level, the electronic device includes a processor, and optionally also includes an internal bus, a network interface, and a memory. The memory may include RAM, such as high-speed random-access memory (RAM), or non-volatile memory, such as at least one disk storage device. Of course, the electronic device may also include other hardware required for other services.

[0211] The processor, network interface, and memory can be interconnected via an internal bus, which can be an ISA (Industry Standard Architecture) bus, a PCI (Peripheral Component Interconnect) bus, or an EISA (Extended Industry Standard Architecture) bus, etc. This bus can be categorized as an address bus, data bus, control bus, etc. For ease of illustration, only a single bidirectional arrow is used in Figure 11, but this does not imply that there is only one bus or one type of bus.

[0212] Memory is used to store programs. Programs may include program code, which includes computer operation instructions. Memory may include main memory and non-volatile memory, and provides instructions and data to the processor.

[0213] The processor reads the corresponding computer program from non-volatile memory into main memory and then runs it, forming a request processing device at the logical level. The processor executes the program stored in memory and performs the following operations: receiving inference requests for model inference using a large model; adding the inference requests to a request queue for storing multiple inference requests; determining the priority of the multiple inference requests based on their inference information, including task difficulty, waiting time, and corresponding user level; and selecting at least one inference request from the multiple inference requests to perform a single inference phase of model inference based on the priority of the multiple inference requests.

[0214] The method executed by the request processing device disclosed in the embodiment shown in Figure 11 of this application can be applied to a processor or implemented by a processor. The processor may be an integrated circuit chip with signal processing capabilities. During implementation, each step of the above method can be completed by integrated logic circuits in the processor's hardware or by instructions in software form. The processor can be a general-purpose processor, including a Central Processing Unit (CPU), a Network Processor (NP), etc.; it can also be a Digital Signal Processor (DSP), an Application Specific Integrated Circuit (ASIC), a Field-Programmable Gate Array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. It can implement or execute the methods, steps, and logic block diagrams disclosed in this application. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the method disclosed in this application can be directly embodied in the execution of a hardware decoding processor, or executed by a combination of hardware and software modules in the decoding processor. The software module can reside in a mature storage medium in the field, such as random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, or registers. This storage medium is located in memory, and the processor reads information from the memory and, in conjunction with its hardware, completes the steps of the above method.

[0215] The electronic device can also perform the methods of the embodiments shown in Figures 3 to 6, and implement the functions of the request processing device in the embodiments shown in Figures 3 to 6, which will not be described again here.

[0216] Of course, in addition to software implementation, the electronic device of this application does not exclude other implementation methods, such as logic devices or a combination of hardware and software, etc. In other words, the execution subject of the following processing flow is not limited to each logic unit, but can also be hardware or logic devices.

[0217] This application also proposes a computer-readable storage medium storing one or more programs, the programs including instructions that, when executed by a portable electronic device including multiple applications, enable the portable electronic device to perform the methods of the embodiments shown in Figures 3 to 6, and to perform the following operations: receiving an inference request for model inference using a large model; adding the inference request to a request queue for storing multiple inference requests; determining the priority of the multiple inference requests based on inference information, the inference information including the task difficulty, waiting time, and corresponding user level of the multiple inference requests; and selecting at least one inference request from the multiple inference requests for a model inference phase based on the priority of the multiple inference requests.

[0218] Figure 12 is a schematic diagram of the structure of a request processing device 120 according to an embodiment of this application. Referring to Figure 12, in one software implementation, the request processing device 120 may include: a receiving module 121, an adding module 122, a determining module 123, and a processing module 124, wherein: the receiving module 121 receives inference requests, the inference requests being used to request model inference using a large model; the adding module 122 adds the inference requests to a request queue, the request queue being used to store multiple inference requests; the determining module 123 determines the priority of the multiple inference requests based on the inference information of the multiple inference requests, the inference information including the task difficulty, waiting time, and corresponding user level of the multiple inference requests; the processing module 124 selects at least one inference request from the multiple inference requests to perform a model inference phase based on the priority of the multiple inference requests.

[0219] In some implementations, the determining module 123 determines the priority of the plurality of inference requests based on the inference information of the plurality of inference requests, including: for each inference request, performing the following operations: determining a first priority based on the task difficulty of the inference request; determining a second priority based on the waiting time of the inference request and a first parameter; determining a third priority based on the user level and a second parameter corresponding to the inference request; and determining the priority of the inference request based on the first priority, the second priority, and the third priority.

[0220] In some implementations, the task difficulty of the inference request is related to at least one of the following: the length of the prompt word of the inference request; the maximum output length specified by the user; the upper limit of the request length that the large model can infer; and the remaining number of decoding attempts of the large model for the inference request.

[0221] In some implementations, the determining module 123 determines a first priority based on the task difficulty of the inference request, including: determining the state of the inference request, the state including the inference request not being pre-filled and the inference request having been pre-filled and in the decoding stage; when the state of the inference request is that the inference request has not been pre-filled, determining the first priority based on the prompt word length, the maximum output length, and the upper limit of the request length; when the state of the inference request is that the inference request has been pre-filled and in the decoding stage, determining the first priority based on the remaining decoding attempts and the upper limit of the request length.

[0222] In some implementations, the determining module 123 determines the first priority based on the prompt word length, the maximum output length, and the upper limit of the request length, including: determining a first ratio of the difference between the upper limit of the request length and the prompt word length to the upper limit of the request length; determining a second ratio of the difference between the upper limit of the request length and the maximum output length to the upper limit of the request length; summing the first ratio and the second ratio according to a specified first coefficient and a second coefficient, and determining the first priority based on the summing result.

[0223] In some implementations, the determining module 123 determines the first priority based on the remaining number of decoding attempts for the inference request by the large model and the upper limit of the request length that the large model can infer, including: determining a third ratio of the difference between the upper limit of the request length and the remaining number of decoding attempts to the upper limit of the request length; and determining the first priority based on the third ratio and a specified third coefficient.

[0224] In some implementations, the first parameter and the second parameter are related to a time tolerance threshold, which is adjustable. The first parameter and the second parameter are adjusted by adjusting the time tolerance threshold. When the first parameter and the second parameter are adjusted by adjusting the time tolerance threshold, the priority of the plurality of inference requests satisfies at least one of the following: when the waiting times are the same, the higher the user level corresponding to the inference request, the higher the priority; when the user levels are the same, if the waiting time of the inference request exceeds the time tolerance threshold, the priority is higher than the priority of a newly arrived inference request.

[0225] In some implementations, the request queue includes a first queue, a second queue, and a third queue. The first queue stores inference requests that have not been pre-filled, the second queue stores inference requests that have completed the pre-filling stage and are in the decoding stage, and the third queue stores inference requests that are preempted for resources during the decoding stage. The processing module 124 selects at least one inference request from the plurality of inference requests for a model inference stage based on the priority of the plurality of inference requests, including: determining whether the third queue is empty; if the third queue is empty, selecting at least one inference request from the first queue for a model inference stage based on the priority of the inference requests in the first queue; if the third queue is not empty, selecting at least one inference request from the second queue for a model inference stage based on the priority of the inference requests in the second queue.

[0226] In some implementations, the processing module 124 selects at least one inference request from the first queue for a pre-filling stage model inference based on the priority of the inference requests in the first queue, including: prioritizing the inference requests in the first queue according to their priority; starting with the inference request with the highest priority, sequentially determining whether the inference request can be inferred in the current round; and performing a pre-filling stage model inference on at least one inference request that can be inferred in the current round.

[0227] In some implementations, the processing module 124, starting with the highest priority inference request, sequentially determines whether the current round can infer the inference request, including: determining whether the inference request meets a first condition; if the inference request meets the first condition, determining that the current round can infer the inference request; wherein the first condition includes at least one of the following: the number of requests already accommodated in the current round is less than the maximum number of requests that the current round can accommodate; the number of tokens already accommodated in the current round is less than the maximum number of tokens that the current round can accommodate; and there are sufficient inference resources to infer the inference request.

[0228] In some implementations, when the number of inference requests that can be inferred in this round is 0, the processing module 124 selects at least one inference request from the second queue for model inference in one decoding phase based on the priority of the inference requests in the second queue.

[0229] In some implementations, before selecting at least one inference request from the first queue for a pre-filling phase model inference based on the priority of the inference requests in the first queue, the processing module 124 determines whether the first queue meets a second condition. The second condition includes that the first queue is not empty and / or the current scheduling interval threshold of the first queue has been reached. The step of selecting at least one inference request from the first queue for a pre-filling phase model inference based on the priority of the inference requests in the first queue includes: if the first queue meets the second condition, selecting at least one inference request from the first queue for a pre-filling phase model inference based on the priority of the inference requests in the first queue.

[0230] In some implementations, if the first queue does not meet the second condition, the processing module 124 selects at least one inference request from the second queue to perform a model inference in the decoding phase based on the priority of the inference requests in the second queue.

[0231] In some embodiments, the processing module 124 selects at least one inference request from the second queue for a decoding phase model inference based on the priority of the inference requests in the second queue, including: prioritizing the inference requests in the second queue according to their priorities; starting with the highest priority inference request, sequentially determining whether the inference resource can infer the inference request; if the inference resource cannot infer the inference request, performing resource preemption starting with the lowest priority inference request in the second queue until the inference resource can infer the inference request; and performing a decoding phase model inference on at least one inference request that the inference resource can infer.

[0232] In some implementations, for any inference request that is preempted by the execution resource, the processing module 124 is used to: store the inference request in the third queue; unload the KV cache occupied by the inference request from the GPU to the CPU; or store the inference request in the first queue; release the KV cache occupied by the inference request; and save the tokens generated by the inference request to the CPU.

[0233] In some embodiments, the processing module 124 is further configured to: when at least one inference request selected from the second queue has not experienced resource preemption, select at least one inference request from the third queue for a decoding phase model inference based on the priority of the inference requests in the third queue; when the number of inference requests that the inference resource can infer is 0, select at least one inference request from the third queue for a decoding phase model inference based on the priority of the inference requests in the third queue.

[0234] In some embodiments, before selecting at least one inference request from the second queue for a decoding stage model inference based on the priority of the inference requests in the second queue, the processing module 124 determines whether the second queue satisfies a third condition, the third condition including that the second queue is not empty; wherein, selecting at least one inference request from the second queue for a decoding stage model inference based on the priority of the inference requests in the second queue includes: if the second queue satisfies the third condition, selecting at least one inference request from the second queue for a decoding stage model inference based on the priority of the inference requests in the second queue.

[0235] In some implementations, if the second queue does not meet the third condition, the processing module 124 selects at least one inference request from the third queue to perform a model inference in the decoding phase based on the priority of the inference requests in the third queue.

[0236] In some implementations, the processing module 124 selects at least one inference request from the third queue for a decoding phase model inference based on the priority of the inference requests in the third queue, including: prioritizing the inference requests in the third queue according to their priorities; starting with the highest priority inference request, sequentially determining whether the inference resource can infer the inference request; if the inference resource can infer the inference request, loading the KV cache corresponding to the inference request from the CPU to the GPU; and performing a decoding phase model inference on at least one inference request that the inference resource can infer.

[0237] In some implementations, after selecting at least one inference request from the plurality of inference requests to perform model inference in one inference phase, the processing module 124 determines whether the at least one inference request has completed model inference; for any inference request, it performs the following operations: if the inference request has completed model inference, it releases the KV cache corresponding to the inference request; if the inference request has not completed model inference, it puts the inference request into the request queue for the next inference phase of model inference.

[0238] The request processing apparatus 120 provided in this application can also execute the methods of FIG3 to FIG6 and implement the functions of the request processing apparatus 120 in the embodiments shown in FIG3 to FIG6, which will not be described again here.

[0239] This application also proposes a computer program product comprising a non-transitory computer-readable storage medium storing a computer program operable to cause a computer to perform some or all of the steps in the above-described request processing method embodiments.

[0240] In summary, the above description is merely a preferred embodiment of this application and is not intended to limit the scope of protection of this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of protection of this application.

[0241] The systems, apparatus, modules, or units described in the above embodiments can be implemented by computer chips or entities, or by products having certain functions. A typical implementation device is a computer. A computer can be, for example, a personal computer, laptop computer, cellular phone, camera phone, smartphone, personal digital assistant, media player, navigation device, email device, game console, tablet computer, wearable device, or any combination of these devices.

[0242] Computer-readable media includes both permanent and non-permanent, removable and non-removable media that can store information using any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic magnetic disk storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transient computer-readable media, such as modulated data signals and carrier waves.

[0243] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0244] The various embodiments in this application are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the system embodiments are basically similar to the method embodiments, so the description is relatively simple; relevant parts can be referred to the descriptions of the method embodiments.

Claims

1. A request processing method, comprising: Receive an inference request, the inference request being used to request model inference using a large model; The inference request is added to a request queue, which is used to store multiple inference requests. Based on the reasoning information of the multiple reasoning requests, the priority of the multiple reasoning requests is determined, wherein the reasoning information includes the task difficulty, waiting time and corresponding user level of the multiple reasoning requests; Based on the priority of the plurality of inference requests, at least one inference request is selected from the plurality of inference requests to perform model inference in one inference phase.

2. The method of claim 1, wherein determining the priority of the plurality of inference requests based on the inference information of the plurality of inference requests includes: For each of the aforementioned inference requests, perform the following operations: Based on the task difficulty of the inference request, determine the first priority; The second priority is determined based on the waiting time of the inference request and the first parameter; The third priority is determined based on the user level and the second parameter corresponding to the inference request; The priority of the inference request is determined based on the first priority, the second priority, and the third priority.

3. The method of claim 2, wherein the task difficulty of the inference request is related to at least one of the following: The length of the prompt words in the reasoning request; User-specified maximum output length; The upper limit of the request length that the large model can infer; The remaining number of decoding attempts for the inference request by the large model.

4. The method as described in claim 3, wherein determining the first priority based on the task difficulty of the reasoning request includes: Determine the state of the inference request, the state including the inference request not being pre-filled and the inference request having been pre-filled and in the decoding stage; When the state of the inference request is that the inference request has not been pre-filled, the first priority is determined based on the prompt word length, the maximum output length, and the upper limit of the request length; When the state of the inference request is that the inference request has been pre-filled and is in the decoding stage, the first priority is determined based on the remaining decoding count and the upper limit of the request length.

5. The method as described in claim 4, wherein determining the first priority based on the prompt word length, the maximum output length, and the upper limit of the request length includes: Determine a first ratio between the difference between the upper limit of the request length and the length of the prompt word and the upper limit of the request length; Determine a second ratio between the difference between the upper limit of the request length and the maximum output length and the upper limit of the request length; The first ratio and the second ratio are summed according to the specified first and second coefficients, and the first priority is determined based on the summing result.

6. The method of claim 4, wherein determining the first priority based on the remaining decoding counts of the large model for the inference request and the upper limit of the request length that the large model can infer includes: Determine a third ratio between the difference between the upper limit of the request length and the remaining number of decoding attempts and the upper limit of the request length; The first priority is determined based on the third ratio and the specified third coefficient.

7. The method as described in claim 2, wherein the first parameter and the second parameter are related to a time tolerance threshold, the time tolerance threshold is adjustable, and the first parameter and the second parameter are adjusted by adjusting the time tolerance threshold; in, When the first parameter and the second parameter are adjusted by adjusting the time tolerance threshold, the priority of the plurality of inference requests satisfies at least one of the following: Given the same waiting time, the higher the user level corresponding to the inference request, the higher the priority. If the user levels are the same, and the waiting time for an inference request exceeds the time tolerance threshold, then the inference request has a higher priority than the newly arrived inference request.

8. The method as described in claim 1, wherein the request queue comprises a first queue, a second queue and a third queue, the first queue being used to store inference requests that have not been pre-filled, the second queue being used to store inference requests that have completed the pre-filling stage and are in the decoding stage, and the third queue being used to store inference requests that are in the decoding stage and have had their resources preempted. The step of selecting at least one inference request from the plurality of inference requests for a single inference phase based on the priority of the inference requests includes: Determine whether the third queue is empty; If the third queue is empty, at least one inference request is selected from the first queue to perform a pre-filling phase model inference based on the priority of the inference requests in the first queue. If the third queue is not empty, at least one inference request is selected from the second queue for model inference in the decoding phase, based on the priority of the inference requests in the second queue.

9. The method of claim 8, wherein selecting at least one inference request from the first queue for a pre-filling phase model inference based on the priority of the inference requests in the first queue comprises: Based on the priority of the inference requests in the first queue, sort the inference requests in the first queue according to their priority. Starting with the highest priority inference request, each round is sequentially assessed to determine whether the inference request can be inferred. Perform a pre-filling phase of model inference for at least one inference request that can be inferred in this round.

10. The method of claim 9, wherein the step of sequentially determining whether the inference request can be inferred in the current round, starting from the highest priority inference request, includes: Determine whether the inference request satisfies the first condition; If the reasoning request satisfies the first condition, it is determined that the reasoning request can be reasoned in this round. The first condition includes at least one of the following: The number of requests accommodated in this round is less than the maximum number of requests that this round can accommodate; The number of lexical units already accommodated in this round is less than the maximum number of lexical units that this round can accommodate; There are sufficient reasoning resources to reason about the stated reasoning request.

11. The method of claim 9, further comprising: If the number of inference requests that can be inferred in this round is 0, at least one inference request is selected from the second queue for model inference in the decoding phase, based on the priority of the inference requests in the second queue.

12. The method of any one of claims 8 to 11, wherein before selecting at least one inference request from the first queue for a pre-filling phase model inference based on the priority of the inference requests in the first queue, the method further comprises: Determine whether the first queue meets the second condition, the second condition including the first queue is not empty and / or the current scheduling interval threshold of the first queue has been reached; The step of selecting at least one inference request from the first queue for a pre-filling phase model inference based on the priority of the inference requests in the first queue includes: If the second condition is met in the first queue, at least one inference request is selected from the first queue for a pre-filling phase model inference based on the priority of the inference requests in the first queue.

13. The method of claim 12, further comprising: If the second condition is not met in the first queue, at least one inference request is selected from the second queue for model inference in the decoding phase according to the priority of the inference requests in the second queue.

14. The method of any one of claims 8, 11, and 13, wherein selecting at least one inference request from the second queue for a single decoding phase model inference based on the priority of the inference requests in the second queue comprises: Based on the priority of the inference requests in the second queue, sort the inference requests in the second queue according to their priority. Starting with the highest priority inference request, determine inference resources in turn whether they are capable of inferring the inference request. If the inference resource is unable to infer the inference request, resource preemption is performed starting from the lowest priority inference request in the second queue until the inference resource is able to infer the inference request. A decoding phase of model reasoning is performed on at least one reasoning request that can be reasoned by the reasoning resource.

15. The method of claim 14, wherein for any inference request subject to resource preemption, the method further comprises any one of the following: The inference request is stored in the third queue; Offload the KV cache occupied by the inference request from the GPU to the CPU; The inference request is stored in the first queue; Release the KV cache occupied by the inference request; Save the generated lexical units from the inference request to the CPU.

16. The method of claim 14, further comprising at least one of the following: If no resource preemption occurs in at least one inference request selected from the second queue, at least one inference request is selected from the third queue according to the priority of the inference requests in the third queue for one decoding phase model inference. If the number of inference requests that the inference resource can infer is 0, at least one inference request is selected from the third queue for model inference in the decoding phase according to the priority of the inference requests in the third queue.

17. The method of claim 14, further comprising, before selecting at least one inference request from the second queue for a single decoding phase model inference based on the priority of the inference requests in the second queue: Determine whether the second queue satisfies the third condition, wherein the third condition includes that the second queue is not empty; The step of selecting at least one inference request from the second queue for model inference in one decoding phase based on the priority of the inference requests in the second queue includes: If the third condition is met in the second queue, at least one inference request is selected from the second queue for model inference in the decoding phase according to the priority of the inference requests in the second queue.

18. The method of claim 17, further comprising: If the third condition is not met in the second queue, at least one inference request is selected from the third queue for model inference in the decoding phase according to the priority of the inference requests in the third queue.

19. The method of claim 16 or 18, wherein selecting at least one inference request from the third queue for model inference in one decoding phase based on the priority of the inference requests in the third queue comprises: The inference requests in the third queue are prioritized according to their priority. Starting with the highest priority inference request, determine inference resources in turn whether they are capable of inferring the inference request. If the inference resource is capable of inferring the inference request, the key-value cache corresponding to the inference request is loaded from the CPU to the GPU; A decoding phase of model reasoning is performed on at least one reasoning request that can be reasoned by the reasoning resource.

20. The method of claim 1, further comprising, after selecting at least one inference request from the plurality of inference requests for a model inference phase, performing model inference in one inference phase: Determine whether the at least one inference request has completed model inference; For any reasoning request, perform the following operation: If the inference request has completed model inference, release the KV cache corresponding to the inference request; If the inference request has not completed model inference, the inference request will be placed in the request queue for the next inference stage of model inference.

21. An electronic device, comprising: processor; Memory used to store the processor's executable instructions; The processor is configured to execute the instructions to implement the method as described in any one of claims 1 to 20.

22. A computer-readable storage medium, wherein instructions in the storage medium, when executed by a processor of an electronic device, enable the electronic device to perform the method as claimed in any one of claims 1 to 20.

23. A computer program product comprising a non-transitory computer-readable storage medium storing a computer program operable to cause a computer to perform some or all of the steps of the method as claimed in any one of claims 1 to 20.

Images