Request processing method, electronic device, storage medium and program product

Abstract

The present application provides a request processing method, an electronic device, a storage medium and a program product, which belong to the field of artificial intelligence technology. The method includes: receiving an inference request, the inference request is used to request the use of a large model for model inference; adding the inference request to a request queue, the request queue is used to store multiple inference requests; according to the inference information of multiple inference requests, determining the priority of multiple inference requests, the inference information includes the task difficulty, waiting time and corresponding user level of multiple inference requests; according to the priority of multiple inference requests, selecting at least one inference request from multiple inference requests to perform model inference in one inference stage. Since the scheduling priority of the inference request can be determined according to the task difficulty, waiting time and corresponding user level, and the scheduling granularity is accurate to one inference stage of the model, it is possible to implement iterative level resource scheduling, optimize resource allocation, reduce response time, and improve overall service quality.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and in particular to a request processing method, electronic device, storage medium and program product. Background Art

[0002] With the rapid development of artificial intelligence (AI) and machine learning (ML) technologies, large-scale pre-trained models (which can be expressed as large language models (LLM), or large models for short), such as GPT-4 and PaLM, have become the core of AI research and application with their powerful multimodal processing capabilities and broad application prospects. A typical application of large models is large model reasoning, which uses trained large models to perform actual task processing. It is a key link in converting the potential of large models into actual value. It plays an important role in intelligent question and answer, automatic code generation, medical diagnosis, autonomous driving and other fields, and greatly improves the performance and user experience of various intelligent applications.

[0003] Reasoning for large models usually involves huge amounts of computing and complex resource management. Especially when dealing with highly concurrent reasoning requests, how to efficiently allocate computing resources, reduce response time, and improve overall service quality is a technical problem that needs to be solved urgently. In related technologies, when processing reasoning requests, most of them schedule reasoning requests based on task queuing and first-come-first-served strategies. However, this solution has poor flexibility and cannot take into account both user needs and resource utilization efficiency, resulting in system performance and user experience being affected. Summary of the invention

[0004] The present application provides a request processing method, electronic device, storage medium and program product, which are used to solve the problem of how to efficiently allocate computing resources, reduce response time and improve overall service quality when using large models to process inference requests.

[0005] To solve the above technical problems, this application is implemented as follows:

[0006] In a first aspect, a request processing method is provided, comprising:

[0007] receiving an inference request, wherein the inference request is used to request to use a large model to perform model inference;

[0008] Adding the inference request to a request queue, where the request queue is used to store multiple inference requests;

[0009] Determining priorities of the multiple reasoning requests according to reasoning information of the multiple reasoning requests, wherein the reasoning information includes task difficulty, waiting time, and corresponding user level of the multiple reasoning requests;

[0010] According to the priorities of the multiple inference requests, at least one inference request is selected from the multiple inference requests to perform model inference in a reasoning phase.

[0011] In a second aspect, an electronic device is provided, including:

[0012] processor;

[0013] a memory for storing instructions executable by the processor;

[0014] The processor is configured to execute the instructions to implement the method as described in the first aspect.

[0015] According to a third aspect, a computer-readable storage medium is provided. When instructions in the storage medium are executed by a processor of an electronic device, the electronic device is enabled to execute the method according to the first aspect.

[0016] According to a fourth aspect, a computer program product is provided, the computer program product comprising a non-transitory computer-readable storage medium storing a computer program, wherein the computer program is operable to cause a computer to execute part or all of the steps in the method described in the first aspect.

[0017] In an embodiment of the present application, after receiving an inference request, the inference request can be added to a request queue, which is used to store multiple inference requests, and then the priorities of the multiple inference requests are determined according to the task difficulty, waiting time and corresponding user level of the multiple inference requests. According to the priority, at least one inference request is selected from the multiple inference requests to perform a model inference in the inference phase. Since the priority of the inference request can be determined by comprehensively considering the three factors of the task difficulty, waiting time and corresponding user level of the inference request when processing the inference request, and at least one inference request is selected for inference according to the priority, not only can the preemptive scheduling of the inference request be realized, but also the user demand and resource utilization can be taken into account, thereby improving the overall performance and stability of the system. In addition, after selecting at least one inference request according to the priority, when inferring at least one inference request, since a model inference in the inference phase can be performed, such as a pre-filling or a decoding, it is possible to implement priority scheduling of the iteration level of the inference request, thereby further optimizing resource allocation, reducing response time, and improving the overall service quality. By optimizing the scheduling mechanism of the inference service, the embodiment of the present application can significantly enhance the adaptability and reliability of the system, thereby promoting the wide application and popularization of large models in various fields. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] In order to more clearly illustrate the technical solutions in the present application or the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0019] Figure 1 It is a schematic diagram of the reasoning process of a large model of an embodiment of the present application;

[0020] Figure 2 It is a schematic diagram of the pre-filling stage and the decoding stage in the large model reasoning of an embodiment of the present application;

[0021] Figure 3 It is a flowchart of a request processing method according to an embodiment of the present application;

[0022] Figure 4 It is a schematic diagram of a request processing flow when the preemptive mode is a recalculation mode according to an embodiment of the present application;

[0023] Figure 5 It is a schematic diagram of a request processing flow when the preemption mode is a replacement mode according to an embodiment of the present application;

[0024] Figure 6 is a flow chart of scheduling inference requests in a request queue according to an embodiment of the present application;

[0025] Figure 7 is a schematic diagram of performing an inference on an inference request according to an embodiment of the present application;

[0026] Figure 8 This is a schematic diagram of a system architecture for implementing large model reasoning in an embodiment of the present application;

[0027] Fig. 9 is a schematic diagram of the structure of a request processing system according to an embodiment of the present application;

[0028] Fig.10 is a schematic diagram of the structure of a scheduler according to an embodiment of the present application;

[0029] Fig.11 is a schematic diagram of the structure of an electronic device according to an embodiment of the present application;

[0030] Fig.12 It is a structural diagram of a request processing device according to an embodiment of the present application. DETAILED DESCRIPTION

[0031] In order to enable those skilled in the art to better understand the technical solutions in this application, the technical solutions in this application will be described clearly and completely below in conjunction with the drawings in one or more embodiments of this application. Obviously, the described embodiments are only part of the embodiments of this application, not all of them. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of this application.

[0032] The terms "first", "second", etc. in the present application and claims are used to distinguish similar objects, and are not used to describe a particular order or precedence. It should be understood that the data used in this way can be interchanged where appropriate so that the present application can be implemented in an order other than those illustrated or described herein. In addition, "and / or" in the present application and claims means at least one of the connected objects, and the character " / " generally means that the objects associated with each other are in an "or" relationship.

[0033] The technical solution provided in the embodiments of the present application can be applied to scenarios where a large model is used for reasoning. The large model may be, for example, GPT-4, PaLM, etc., which are not specifically limited here. Scenarios where a large model is used for reasoning may include intelligent question answering, automatic code generation, medical diagnosis, autonomous driving, etc., which are not specifically limited here.

[0034] In order to facilitate understanding of the technical solution provided by the embodiments of the present application, the reasoning process of the large model in the related technology (that is, the reasoning principle of the large model) will be explained below.

[0035] The large model is an autoregressive model. The inference process of the autoregressive model can be as follows Figure 1 shown. Figure 1 In , when using a large model for reasoning, the user can enter a prompt word (such as Figure 1 As shown in the figure, “What is the square root of 4?”), after receiving the prompt word and performing model reasoning based on the prompt word, the large model can output an answer (i.e., the reasoning result, such as Figure 1 The answer is 2, which is of length N). Figure 1 It can be seen that when the large model performs reasoning, it executes N reasonings (that is, N iterations) in the entire reasoning process. Among them, the large model outputs only one token (a basic unit in natural language processing, which can be a word, part of a word, character, or other form of text fragment) each time it performs reasoning. The output token will be concatenated with the input token and then used as the input for the next reasoning. This process is repeated until the end identifier is encountered.

[0036] exist Figure 1In the reasoning process shown, the input of the previous reasoning and the input of the next reasoning differ by only one word, so there is a problem of repeated calculation between the two reasonings. In order to avoid the problem of repeated calculation, in related technologies, the intermediate results in the large model reasoning process can be cached, so that these intermediate results can be reused in subsequent reasoning, thereby reducing repeated calculations and improving the overall reasoning speed. Among them, the cached intermediate results are generally the two variables Key and Value (such as the key parameter and value parameter in Transformer, which are used to accelerate attention calculations), so this part of the cache is also called KV cache.

[0037] The inference phase of a large model can be divided into a prefill phase and a decode phase. The complete inference process of an inference request can include one prefill phase and n decoding phases, where n is 0 or a positive integer. To understand the inference phase of a large model, see Figure 2 .

[0038] Figure 2 yes Figure 1 Schematic diagram of the pre-filling phase and decoding phase corresponding to the large model inference process shown. Figure 2 The large model reasoning process shown includes one pre-filling stage (corresponding to reasoning round 1) and two decoding stages (corresponding to reasoning rounds 2 and 3). In the pre-filling stage, the model input is "What is the square root of 4?" After the model executor performs a pre-filling according to the model input, it can output "answer". At the same time, the model executor can also cache the intermediate results generated in the pre-filling stage in the KV cache. In the first decoding stage, the model input includes the output of the pre-filling stage and the intermediate results cached in the KV cache. After the model executor performs the first decoding according to the model input, it can output "yes". At the same time, the model executor can also cache the intermediate results generated in the first decoding stage in the KV cache. In the second decoding stage, the model input includes the output of the first decoding stage and the intermediate results cached in the KV cache. After the model executor performs the second decoding according to the model input, it can output "2". At this point, the model reasoning process ends, and the model output result (that is, the reasoning result) is "the answer is 2".

[0039] It should be noted that the technical solution provided by the embodiment of the present application can realize iterative priority scheduling of reasoning requests, that is, when the embodiment of the present application schedules the reasoning requests, the scheduling granularity can be accurate to each generated word element, that is, each scheduling will obtain a group of reasoning requests (one or more reasoning requests) according to the priority to perform a model reasoning of the reasoning stage. Each reasoning request involved in the model reasoning only generates a new word element, and one pre-filling and one decoding can be regarded as a model reasoning of the reasoning stage.

[0040] The technical solutions provided by various embodiments of the present application are described in detail below in conjunction with the accompanying drawings.

[0041] Figure 3 It is a flowchart of a request processing method according to an embodiment of the present application. Figure 3 The request processing method shown can be executed by an inference server (such as a large model inference engine) for performing model inference. Figure 3 The request processing method shown includes the following steps.

[0042] Step S302: receiving an inference request, where the inference request is used to request to use a large model for model inference.

[0043] The inference request may be sent by a user to the inference server. For example, the user may send the inference request to the inference server through a user web interface provided by the inference server. The number of inference requests may be one or more (from one or more users), which is not specifically limited here.

[0044] Step S304: adding the inference request to a request queue, where the request queue is used to store multiple inference requests.

[0045] After receiving the inference request, the inference server can add the inference request to the request queue. The request queue is used to store multiple inference requests, which can be multiple inference requests received by the inference server at the same time (i.e., concurrent requests), or can be inference requests received by the inference server at different times, which are not specifically limited here. In addition, the multiple inference requests can be inference requests that have not been pre-filled, or can be inference requests that have been pre-filled and are in the decoding stage, which are not specifically limited here. In other words, as long as the inference request has not completed the inference process, it can be stored in the request queue.

[0046] In some embodiments, the request queue may include multiple sub-queues. When storing the reasoning request, the request queue may store the reasoning request in different sub-queues according to the state of the reasoning request. The state of the reasoning request here may be the reasoning stage in which the reasoning request is located, and the reasoning stage includes a pre-filling stage and a decoding stage. When storing the reasoning request according to the state of the reasoning request, for example, for the reasoning request that has not been pre-filled, it can be stored in a certain sub-queue, and for the reasoning request that has completed pre-filling and is in the decoding stage, it can be stored in other sub-queues. Further, as described above, the embodiment of the present application can realize preemptive scheduling of reasoning requests (the preempted resources can be GPU resources. When preemption occurs, the high-priority reasoning request will preempt the GPU resources of the low-priority reasoning request). Therefore, in the process of reasoning multiple reasoning requests, there may be reasoning requests whose resources are preempted. In this embodiment, for the reasoning request whose resources are preempted, its corresponding KV cache will be unloaded from the GPU to the CPU (for the reasoning request whose resources are not preempted, its corresponding KV cache is stored in the GPU). In this way, when storing inference requests, the request queue can also store the inference requests in different sub-queues according to the location of the KV cache corresponding to the inference request (or whether the inference request is preempted by resources). For example, for inference requests whose KV cache is located in the CPU (that is, inference requests whose resources are preempted), they can be stored in a certain sub-queue, and for inference requests whose KV cache is located in the GPU (that is, inference requests whose resources are not preempted), they can be stored in other sub-queues.

[0047] In some more specific implementations, the request queue may include three sub-queues, represented as a first queue, a second queue, and a third queue. The first queue is used to store inference requests that have not been pre-filled. Optionally, the first queue is also used to store inference requests that need to be recalculated after resources are preempted (i.e., inference requests that need to be re-pre-filled). The inference requests in the first queue do not occupy the KV cache. The second queue is used to store inference requests that have completed pre-filling and are in the decoding stage. The inference requests in the second queue are inference requests whose resources have not been preempted. These requests have corresponding KV caches and are stored in the GPU. The third queue is used to store inference requests that are in the decoding stage and whose resources have been preempted. Optionally, the inference requests in the third queue may be inference requests that do not need to be recalculated after resources are preempted. These requests have corresponding KV caches and are stored in the CPU.

[0048] Step S306: determining priorities of the multiple reasoning requests according to reasoning information of the multiple reasoning requests, where the reasoning information includes task difficulty, waiting time and corresponding user level of the multiple reasoning requests.

[0049] After the reasoning request is stored in the request queue, multiple reasoning requests in the request queue can be scheduled and processed. When scheduling multiple reasoning requests in the request queue, the embodiment of the present application can first determine the priority of the multiple reasoning requests based on the reasoning information of the multiple reasoning requests, and then schedule the multiple reasoning requests based on the priority of the multiple reasoning requests. Among them, for each reasoning request, its reasoning information at least includes the task difficulty, waiting time and corresponding user level of the reasoning request. This information can be carried in the reasoning request, or it can be obtained by the reasoning server through other methods, which is not specifically limited here.

[0050] The task difficulty of an inference request can represent the difficulty of reasoning about the inference request. The task difficulty of an inference request involves the length of the prompt of the inference request (i.e., the length of the model input, e.g. Figure 1 The factors include the length of the prompt word "What is the square root of 4?" shown in the figure), the maximum output length specified by the user (i.e., the length of the final output result of the large model), the upper limit of the request length that the large model can infer, the remaining number of decoding times of the large model for the inference request, the large model, the GPU model and number used by the inference service, the CPU core and the main memory size. Among them, the factors such as the large model and the GPU model and number used by the inference service, the CPU core and the main memory size are public environmental factors, which do not affect the priority of the inference request and can be excluded. Therefore, in this embodiment, the task difficulty of the inference request mainly considers the four features of the prompt word length of the inference request, the maximum output length specified by the user, the upper limit of the request length that the large model can infer, and the remaining number of decoding times of the inference request by the large model. That is to say, the task difficulty of the inference request can be related to at least one of the four features of the prompt word length of the inference request, the maximum output length specified by the user, the upper limit of the request length that the large model can infer, and the remaining number of decoding times of the inference request by the large model. Among them, the two features of the prompt word length of the inference request and the maximum output length specified by the user are negatively correlated with the difficulty of the task (because the longer the prompt word length and the maximum output length, the longer the inference request takes, and accordingly, the higher the difficulty of the task). The upper limit of the request length that can be inferred by the large model is determined by the properties of the large model and is generally a fixed value. The remaining number of decoding times of the inference request by the large model is equal to the maximum output length specified by the user minus the number of generated word units.

[0051] The waiting time of the inference request may be the time that the inference request waits for inference, and may represent the hunger level of the inference request. In this embodiment, the waiting time of the inference request may be expressed as:

[0052] T1 = now - arrival_time. Now represents the current time point, and arrival_time represents the time when the inference request arrives at the inference server.

[0053] The user level corresponding to the inference request is the level of the user who sent the inference request. Generally, user attributes (such as the user's gender, occupation, etc.) can reflect some characteristics of the user request. In this embodiment, users can be divided into different user levels according to different attributes. The higher the user level, the higher the priority of the corresponding inference request. In other words, a higher priority can be given to the inference request of a high-level user. Among them, the number of divided user levels can be determined according to actual needs and is not specifically limited here.

[0054] When determining the priorities of multiple reasoning requests according to the reasoning information of the multiple reasoning requests, in some implementations, for each reasoning request, the following steps S3061 to S3064 may be included:

[0055] Step S3061: Determine the first priority according to the task difficulty of the reasoning request.

[0056] As mentioned above, the task difficulty of the reasoning request is related to at least one of the four features: the length of the prompt word of the reasoning request, the maximum output length specified by the user, the upper limit of the request length that can be inferred by the large model, and the remaining number of decoding times of the reasoning request by the large model. Therefore, when determining the first priority, it can be determined based on at least one of these four features. In addition, since the scheduling granularity of the embodiment of the present application is at the iteration level when scheduling the reasoning request, and the task difficulty corresponding to the reasoning request at different reasoning stages is different, therefore, when determining the first priority, it is also necessary to consider the reasoning stage of the reasoning request.

[0057] In some implementations, determining the first priority according to the task difficulty of the reasoning request may include:

[0058] Determine a state of the inference request, where the state of the inference request includes the inference request not being pre-filled and the inference request being pre-filled and in a decoding phase;

[0059] When the inference request status is that the inference request is not pre-filled, the first priority is determined according to the prompt word length of the inference request, the maximum output length specified by the user, and the upper limit of the request length that can be inferred by the large model;

[0060] When the status of the inference request is that the inference request has been pre-filled and is in the decoding stage, the first priority is determined according to the remaining decoding times of the inference request by the large model and the upper limit of the request length that can be inferred by the large model.

[0061] Specifically, for inference requests that have not been pre-filled, since the inference request has not yet performed model inference on any word unit, when determining the first priority, it is necessary to consider the length of the prompt word of the inference request, the maximum output length specified by the user, and the upper limit of the request length that the large model can infer. For inference requests that have completed pre-filling and are in the decoding stage, since these requests have completed pre-filling, when determining the first priority, it is sufficient to consider the remaining number of decoding times of the inference request by the large model and the upper limit of the request length that the large model can infer.

[0062] In some implementations, determining the first priority based on the prompt word length of the inference request, the maximum output length specified by the user, and the upper limit of the request length that can be inferred by the large model may include:

[0063] Determine a first ratio of the upper limit of the request length that can be inferred by the large model and the difference between the length of the prompt word of the inference request and the upper limit of the request length;

[0064] Determine a second ratio of a difference between an upper limit of a request length that can be inferred by the large model and a maximum output length specified by a user to the upper limit of the request length;

[0065] The first ratio and the second ratio are summed according to the specified first coefficient and second coefficient, and the first priority is determined according to the summed result.

[0066] The first coefficient and the second coefficient can be obtained based on the fitting function of the time consumption of the reasoning task, the prompt word length and the maximum output length, and then the specific value is determined through experimental search. The specific size of the first coefficient and the second coefficient is not limited here. When determining the first priority based on the prompt word length of the reasoning request, the maximum output length specified by the user, and the upper limit of the request length that can be inferred by the large model, it can be determined by the following formula:

[0067] .

[0068] In the above formula, prompt_len represents the prompt word length of the inference request, output_len represents the maximum output length specified by the user, and model_len represents the upper limit of the request length that can be inferred by the large model. Among them, alpha is the first coefficient, beta is the second coefficient, and alpha and beta are hyperparameters related to the service deployment environment. They are determined as follows: the value range of alpha and beta is obtained according to the fitting function of the inference task time, the prompt word length and the maximum output length, and then the specific value is determined through experimental search. Experiments show that alpha and beta have a certain generalization ability, that is, the same set of alpha and beta can be used in different inference environments without a significant impact on performance. Optionally, the value of alpha can be fixed to 1, and the value of beta can be between 3 and 5 or between 2 and 10.

[0069] In some implementations, determining the first priority based on the remaining number of decoding times of the inference request by the large model and the upper limit of the request length that can be inferred by the large model may include:

[0070] Determine a third ratio of the upper limit of the request length that can be inferred by the large model and the difference between the remaining number of decoding times of the inference request by the large model and the upper limit of the request length;

[0071] The first priority is determined according to the third ratio and the specified third coefficient.

[0072] The third coefficient can be obtained according to the fitting function of the time consumption of the reasoning task, the length of the prompt word and the maximum output length, and then the specific value is determined by experimental search. The specific size of the third coefficient is not limited here. Optionally, in some embodiments, the third coefficient can be the same as the second coefficient mentioned above.

[0073] When determining the first priority based on the remaining number of decoding times of the inference request by the large model and the upper limit of the request length that can be inferred by the large model, it can be determined by the following formula:

[0074] .

[0075] In the above formula, model_len represents the upper limit of the request length that the large model can infer, and remaining_iter_num represents the remaining number of decoding times of the inference request by the large model, which is equal to the maximum output length specified by the user minus the number of words that have been generated. Among them, beta is the third coefficient, which is the same as the second coefficient and is a hyperparameter related to the service deployment environment. Optionally, the value of beta can be between 3 and 5 or between 2 and 10.

[0076] Based on the method for determining the first priority described above, it can be known that when determining the first priority according to the task difficulty of the reasoning request, it can be determined by the following formula:

[0077] .

[0078] It should be noted that the above-mentioned function for determining the first priority is a linear function. In other possible implementations, the function for determining the first priority may also be other functions, such as an inverse function, a trigonometric function, an exponential function, a logarithmic function, etc., which is not specifically limited here.

[0079] Step S3062: Determine the second priority according to the waiting time of the inference request and the first parameter.

[0080] The first parameter is a hyperparameter. When determining the second priority according to the waiting time of the inference request and the first parameter, it can be determined by the following formula:

[0081] , where b is the first parameter and now - arrival_time is the waiting time for the inference request.

[0082] Step S3063: Determine the third priority level according to the user level and the second parameter corresponding to the inference request.

[0083] The second parameter is a hyperparameter. When determining the third priority according to the user level and the second parameter corresponding to the inference request, it can be determined by the following formula:

[0084] , where c is the second parameter, n is the user level corresponding to the inference request, and different values ​​of n can correspond to different user levels. For example, the value range of n can be {1, 2, 3, 4, 5}, indicating that the user level is divided into five levels. The larger the value of n, the higher the user level.

[0085] Step S3064: Determine the priority of the inference request according to the first priority, the second priority, and the third priority.

[0086] When determining the priority of the reasoning request, if only the task difficulty of the reasoning request is considered, the overall optimal performance of the reasoning service can be obtained. However, when the service pressure is high, some long sequence requests may exceed the maximum waiting time of the user, resulting in a poor user experience for these requests. If only the waiting time is considered, the scheduling of the reasoning request will return to the original first-come-first-served scheduling strategy. In view of this, when determining the priority of the reasoning request, this embodiment comprehensively considers performance and fairness, and can comprehensively determine the priority of the reasoning request based on the three factors of the task difficulty, waiting time and corresponding user level of the reasoning request, that is, determine the priority of the reasoning request based on the first priority, the second priority and the third priority.

[0087] In some implementations, when determining the priority of the inference request according to the first priority, the second priority, and the third priority, the sum of the first priority, the second priority, and the third priority may be determined as the priority of the inference request. The specific formula is as follows:

[0088] .

[0089] It can be seen from the above formula that by adjusting the two hyperparameters b and c (i.e., the first parameter and the second parameter mentioned above), the degree of influence of the three factors of task difficulty, waiting time and user level on the priority can be changed. In order to sensitively adjust the performance and fairness of the system, this embodiment sets a time tolerance threshold starved_threshold (also expressed as a starvation tolerance threshold), and parameters b and c are related to the time tolerance threshold. The time tolerance threshold is adjustable, and parameters b and c can be adjusted by adjusting the time tolerance threshold. Among them, in order to ensure system performance and fairness, when adjusting parameters b and c by adjusting the time tolerance threshold, the priorities of multiple inference requests determined according to the adjusted parameters b and c need to meet at least one of the following two rules:

[0090] When the waiting time is the same, the higher the user level of the inference request, the higher the priority;

[0091] In the case of the same user level, if the waiting time of an inference request exceeds the time tolerance threshold, the priority is higher than the priority of the newly arrived inference request.

[0092] That is to say, when adjusting parameters b and c through the time tolerance threshold, no matter how they are adjusted, it is necessary to ensure that: for reasoning requests of the same user level, after the waiting time exceeds the time tolerance threshold, the priority is higher than the priority of the newly arrived reasoning request, and / or, for reasoning requests with the same waiting time, the reasoning request with a higher user level has a higher priority. From this, the values ​​of parameters b and c can be derived. For ease of understanding, the specific derivation process of parameter b will be explained below using parameter b as an example.

[0093] As mentioned above, for inference requests of the same user level, after the waiting time exceeds the time tolerance threshold starved_threshold, the priority is higher than the priority of the newly arrived inference requests of the same user level, that is, p_starved needs to be greater than p_new. The formulas of p_starved and p_new are as follows:

[0094] p_starved = priority_1 + priority_2 + priority_3 = x + b*starved_threshold+c*n.

[0095] Here, x is the value obtained by substituting prompt_len and output_len in the inference request into priority_1. Since it is assumed here that it can be any request, the value range of x is (0, alpha+beta).

[0096] p_new =priority_1 +priority_2 + priority_3 = y + c*n.

[0097] Among them, y is the same as x, which is the value obtained by substituting prompt_len and output_len in the inference request into priority_1. Since the waiting time of the inference request is 0, priority_2=0. Since it is assumed to be an arbitrary request, the value range of y is (0, alpha+beta).

[0098] Since p_starved needs to be greater than p_new, that is, x + b*starved_threshold+c*n needs to be greater than y +c*n, we can get the formula b*starved_threshold>yx. Since the formula b*starved_threshold>yx is true for any x and y, we can deduce b*starved_threshold>max(yx). Since max(yx) is equal to alpha+beta, we can further deduce b*starved_threshold>= alpha+beta. Given starved_threshold, we only need to set b=(alpha+beta) / starved_threshold to make the reasoning request of the same user level have a higher priority than the newly arrived reasoning request after the waiting time exceeds starved_threshold. From this, we can deduce the value of parameter b, that is, b=(alpha+beta) / starved_threshold.

[0099] Based on the same principle, the value of parameter c can be derived, and the derivation process of parameter c is not described in detail here. The derived value of parameter c is (alpha+beta)(1+1 / starved_threshold).

[0100] After obtaining the values ​​of parameters b and c, we can further obtain the complete form of the priority function of the inference request:

[0101] .

[0102] Among them, b=(alpha+beta) / starved_threshold, c=(alpha+beta)(1+1 / starved_threshold).

[0103] The above priority function comprehensively considers three factors: performance (determined by task difficulty), waiting time and user level, and controls the system's preference for performance and fairness through the time tolerance threshold starved_threshold. Generally, the higher the time tolerance threshold, the lower the proportion of waiting time in the priority function. When the time tolerance threshold is low, the scheduling strategy tends to be first-come-first-served, and pays more attention to fairness. When the time threshold is high, the scheduling is prioritized by efficiency. In addition, the priority function also satisfies the higher priority of requests from high-level users. In practical applications, if fairness is preferred, the time tolerance threshold can be set lower, such as 120ms. If the number of inference requests is large and the system performance is to be guaranteed, the time tolerance threshold can be set higher, such as 1200ms. Of course, the 120ms and 1200ms here are only exemplary. In specific implementations, the time tolerance threshold can be flexibly set according to actual business needs, and the size of the time tolerance threshold is not specifically limited here.

[0104] Step S308: According to the priorities of the multiple inference requests, at least one inference request is selected from the multiple inference requests to perform model inference in a reasoning phase.

[0105] After determining the priorities of multiple inference requests in the request queue by the method described in S306 above, at least one inference request can be selected from the multiple inference requests according to the priority to perform model inference in a reasoning stage. For example, one or more inference requests can be selected in order from high to low priority to perform model inference in a reasoning stage. Since the priority takes into account the task difficulty, waiting time and corresponding user level of the reasoning request, selecting at least one inference request for reasoning according to the priority can not only realize preemptive scheduling of the reasoning request, but also take into account user needs and resource utilization, thereby improving the overall performance and stability of the system. In addition, when processing at least one selected inference request, specifically performing a model inference in a reasoning stage, such as a pre-filling or a decoding, therefore, it is possible to implement iterative-level priority scheduling of the reasoning request, thereby further optimizing resource allocation, reducing response time, and improving overall service quality.

[0106] In some embodiments, as described above, the request queue may include a first queue, a second queue, and a third queue, the first queue is used to store inference requests that have not been pre-filled, the second queue is used to store inference requests that have been pre-filled and are in the decoding stage, and the third queue is used to store inference requests whose resources are preempted in the decoding stage. In this way, when selecting at least one inference request from multiple inference requests to perform model inference in an inference stage according to the priorities of multiple inference requests, the following steps may be included:

[0107] Determine whether the third queue is empty;

[0108] When the third queue is empty, selecting at least one inference request from the first queue according to the priority of the inference request in the first queue to perform a model inference in the pre-filling phase;

[0109] When the third queue is not empty, at least one inference request is selected from the second queue according to the priority of the inference request in the second queue to perform a model inference in the decoding stage.

[0110] The embodiment of the present application can set corresponding priorities for the first queue, the second queue, and the third queue. When setting the priority of the queue, in order to improve the processing efficiency of the first word (the word output in the pre-filling stage), shorten the delay, and optimize resource allocation, it can be set that the first queue has the highest priority when no resource preemption occurs. In addition, in order to ensure that the reasoning requests in the second queue and the third queue can be processed and further optimize resource allocation, it can be set that the priority of the second queue and the third queue is higher than that of the first queue when resource preemption occurs. Further, considering that the reasoning requests stored in the second queue are requests for resources that have not been preempted, and the reasoning requests stored in the third queue are requests for resources that have been preempted, and when resource preemption occurs, it is usually the high-priority reasoning requests that preempt the resources of the low-priority reasoning requests (to ensure that the high-priority reasoning requests can be processed first), therefore, it can also be set that the priority of the second queue is higher than that of the third queue when resource preemption occurs. In this way, when scheduling the inference requests in these three queues, the overall scheduling logic can be: when there is no resource preemption, the inference requests in the first queue are processed first. If the first queue is empty, that is, the number of inference requests in the first queue is 0, the inference requests in the second queue are processed. In the case of resource preemption, the inference requests in the second queue are processed first, and then the preempted requests in the third queue are processed, and finally the requests in the first queue are processed.

[0111] Based on the above-mentioned overall scheduling logic for the first queue, the second queue, and the third queue, when selecting at least one inference request according to priority to perform a model inference in the inference stage, it is possible to first determine whether the third queue is empty. If the third queue is empty, that is, the number of inference requests stored in the third queue is 0, it can be said that resource preemption has not occurred at present. At this time, at least one inference request can be selected from the first queue according to the priority of the inference request in the first queue to perform a model inference in the pre-filling stage. If the third queue is not empty, that is, the number of inference requests stored in the third queue is not 0, it can be said that resource preemption has occurred at present, and the inference request of the preempted resources has been stored in the third queue. At this time, at least one inference request can be selected from the second queue according to the priority of the inference request in the second queue to perform a model inference in the decoding stage.

[0112] When selecting at least one inference request from the first queue to perform a model inference in the pre-filling phase according to the priority of the inference request in the first queue, in some embodiments, the following steps may be included:

[0113] Prioritize the inference requests in the first queue according to the priorities of the inference requests in the first queue;

[0114] Starting from the inference request with the highest priority, determine in turn whether the inference request can be inferred in this round;

[0115] A model inference in the pre-filling phase is performed for at least one inference request that can be inferred in this round.

[0116] When starting from the inference request with the highest priority and determining in turn whether a certain inference request can be inferred in this round, taking the inference request with the highest priority in the first queue as an example, the following steps may be included:

[0117] Determine whether the inference request satisfies the first condition;

[0118] When the inference request satisfies the first condition, it is determined that the inference request can be inferred in this round.

[0119] The first condition is a condition that the inference request needs to meet when the inference request can be inferred in this round. In some implementations, the first condition may include at least one of the following:

[0120] The number of requests that have been accommodated in this round is less than the maximum number of requests that can be accommodated in this round;

[0121] The number of tokens already accommodated in this round is less than the maximum number of tokens that can be accommodated in this round;

[0122] There are sufficient inference resources to infer the inference request, where the inference resources may include but are not limited to GPU resources.

[0123] It should be noted that, in some implementations, in order to ensure that the reasoning request can be executed smoothly, the reasoning request can only be considered to be able to be inferred in this round if the reasoning request meets three of the above-mentioned first conditions. Otherwise, it can be considered that the reasoning request cannot be inferred in this round.

[0124] For the inference request with the highest priority in the first queue, when it is determined that the inference request can be inferred in this round according to the above method, the inference request with the second highest priority can be judged based on the same method. If the inference request with the second highest priority can be inferred in this round, the inference request with the third highest priority can be judged based on the same method, and so on, and so on, until a certain inference request cannot be inferred in this round (after each judgment on whether inference can be made, if a certain inference request can be inferred, the corresponding inference resources will be deducted. Therefore, in the process of cyclic judgment, the inference resources are constantly decreasing).

[0125] After performing the above-mentioned cyclic judgment on the inference requests in the first queue, for one or more inference requests that can be inferred, a pre-filling stage model inference can be performed on them in this round, and for one or more inference requests that cannot be inferred, they can wait for the model inference of the next inference stage.

[0126] It should be noted that, for the inference request with the highest priority in the first queue, after the above judgment is performed on the inference request, the judgment result may be that the inference request cannot be inferred in this round. At this time, the above-mentioned loop judgment operation can be ended, and it can be determined that there are no inference requests that can be inferred in the first queue, that is, in the first queue, the number of inference requests that can be inferred in this round is 0. In the case where the number of inference requests that can be inferred in this round in the first queue is 0, in some embodiments, the inference requests in the second queue can be processed, that is, according to the priority of the inference request in the second queue, at least one inference request is selected from the second queue to perform a model inference in the decoding stage. In this way, resource waste can be avoided and resource utilization and processing efficiency can be improved.

[0127] The above example illustrates how to select at least one inference request from the first queue to perform a model processing in the pre-filling stage when the third queue is empty, that is, when no resource preemption occurs at present. It should be noted that, in some embodiments, when selecting at least one inference request from the first queue to perform a model inference in the pre-filling stage, there may be certain prerequisites. When it is determined that the third queue is empty, the prerequisite can be judged first, and then it is determined whether to select at least one inference request from the first queue to perform a model inference in the pre-filling stage according to the judgment result. Specifically, when it is determined that the third queue is empty, before selecting at least one inference request from the first queue to perform a model inference in the pre-filling stage according to the priority of the inference request in the first queue, it can be determined whether the first queue meets the second condition. The second condition may include that the first queue is not empty and / or the scheduling interval threshold of the first queue has been reached. If the first queue meets the second condition, such as the first queue is not empty, or the first queue is not empty and the scheduling interval threshold of the first queue has been reached, then at least one inference request can be selected from the first queue according to the priority of the inference request in the first queue to perform a model inference in the pre-filling stage. If the first queue does not meet the second condition, for example, the first queue is empty, or the first queue is not empty but the scheduling interval threshold of the first queue is not reached currently, then scheduling processing may not be performed in the first queue.

[0128] Optionally, when the first queue does not meet the second condition, the inference requests in the second queue can be scheduled, that is, according to the priority of the inference requests in the second queue, at least one inference request is selected from the second queue to perform a model inference in the decoding stage.

[0129] It should be noted that the reason why the first queue is not empty is used as the second condition here, because when the first queue is empty, there is no reasoning request in the first queue, and the first queue cannot be scheduled at this time. The reason why whether the scheduling interval threshold of the first queue is reached is used as the second condition is that when there is no resource preemption, based on the overall scheduling logic of the first queue, the second queue and the third queue, the reasoning request in the first queue will be processed first, so there may be a problem that the reasoning request in the second queue cannot be processed for a period of time, resulting in a long processing delay and affecting the user experience. In order to avoid this problem, before selecting at least one reasoning request from the first queue for a model reasoning in the pre-filling stage according to the priority of the reasoning request in the first queue, it is necessary to determine whether the scheduling interval threshold of the first queue has been reached. If it has been reached, the reasoning request in the first queue can be scheduled, otherwise, the reasoning request in the first queue can be not scheduled, but the reasoning request in the second queue can be scheduled, that is, according to the priority of the reasoning request in the second queue, at least one reasoning request is selected from the second queue for a model reasoning in the decoding stage. Thus, the problem that the inference requests in the second queue cannot be processed without resource preemption can be avoided. The scheduling interval threshold of the first queue can be determined according to actual needs and is not specifically limited here.

[0130] In this embodiment, when the third queue is not empty, or the third queue is empty and the first queue is also empty, or the third queue is empty and the number of inference requests that can be inferred in this round in the first queue is 0, or the third queue is empty and the first queue does not meet the second condition, the inference requests in the second queue can be scheduled, that is, according to the priority of the inference requests in the second queue, at least one inference request is selected from the second queue to perform a model inference in the decoding stage. Among them, when selecting at least one inference request from the second queue to perform a model inference in the decoding stage according to the priority of the inference requests in the second queue, in some embodiments, the following steps may be included:

[0131] Prioritizing the inference requests in the second queue according to the priorities of the inference requests in the second queue;

[0132] Starting from the inference request with the highest priority, determine in turn whether the inference resource can infer the inference request;

[0133] When the reasoning resource cannot reason the reasoning request, resource preemption is performed starting from the reasoning request with the lowest priority in the second queue until the reasoning resource can reason the reasoning request;

[0134] A model inference of the decoding phase is performed for at least one inference request that the inference resource can infer.

[0135] Inference resources may include but are not limited to GPU resources. Since the inference requests in the second queue have been pre-filled and are in the decoding stage, and in the decoding stage, the main factor affecting whether the inference request can be inferred in this round is whether the inference resources are sufficient, therefore, when judging whether the inference request in the second queue can be inferred in this round, it is specifically to judge whether the inference resources can infer the inference request in the second queue, that is, to judge whether the inference resources can meet the demand for inference resources by the inference request in the second queue.

[0136] When judging whether the reasoning resource can infer the reasoning request in the second queue, it can be judged whether the reasoning resource can infer the reasoning request starting from the reasoning request with the highest priority in the second queue. If the reasoning resource can infer the reasoning request with the highest priority, it can continue to judge whether the reasoning resource can infer the reasoning request with the second highest priority. If the reasoning resource cannot infer the reasoning request with the highest priority, resource preemption can be performed, that is, starting from the reasoning request with the lowest priority in the second queue, resource preemption is performed for other reasoning requests in the second queue in order from low to high priority (the preempted resources will be used to infer the reasoning request with the highest priority). If the reasoning resource can infer the reasoning request with the highest priority after resource preemption, it can continue to judge whether the reasoning resource can infer the reasoning request with the second highest priority. If the reasoning resource still cannot infer the reasoning request with the highest priority after resource preemption, the judgment process can be ended, that is, the reasoning resource is not judged whether it can infer other reasoning requests in the second queue. At this time, it can be determined that the number of requests that can be inferred in the second queue in this round is 0.

[0137] When determining whether the reasoning resource can infer the second-highest priority reasoning request, if the reasoning resource can infer the second-highest priority reasoning request, continue to determine whether the reasoning resource can infer the third-highest priority reasoning request; if the reasoning resource cannot infer the second-highest priority reasoning request, perform resource preemption based on the method described above until the reasoning resource can infer the second-highest priority reasoning request.

[0138] The above uses the highest and second highest priority inference requests in the second queue as an example to illustrate how to determine whether the inference resources can infer the inference requests in the second queue. For other inference requests in the second queue, the judgment can be based on the same method, which will not be repeated here.

[0139] Starting from the inference request with the highest priority, after determining inference resources in turn whether they can infer the inference requests in the second queue, the determination results may include the following situations:

[0140] The first case: no resource preemption occurs, and the inference resources are able to infer all inference requests in the second queue.

[0141] The second situation: resource preemption occurs, and the inference resources can infer some inference requests in the second queue.

[0142] The third situation: resource preemption occurs, and the inference resources are still unable to infer the inference requests in the second queue, that is, the number of requests in the second queue that can be inferred in this round is 0.

[0143] For the first case above, since the inference resource can infer all the inference requests in the second queue, that is, the inference resource can meet the demand for inference resources of the inference requests in the second queue, therefore, for all the inference requests in the second queue, a model inference of the decoding stage can be performed in this round. In addition, since there is no resource preemption, the inference resource may still have remaining resources. In order to avoid resource waste, in some embodiments, the inference requests in the third queue can also be scheduled, that is, according to the priority of the inference request in the third queue, at least one inference request is selected from the third queue to perform a model inference of the decoding stage, thereby avoiding resource waste and improving resource utilization.

[0144] For the second situation mentioned above, since the inference resources can infer some of the inference requests in the second queue, that is, the inference resources can only meet the demand for inference resources of some of the inference requests in the second queue, therefore, for the inference requests that can be inferred in the second queue, a decoding stage model inference can be performed on them in this round, and for the inference requests that cannot be inferred, they can wait for the next round of model inference.

[0145] For the third case above, since the number of requests that can be inferred in this round in the second queue is 0, the inference requests in the second queue may not be subjected to model inference, and may wait for the next round of model inference. Optionally, considering that different inference requests may require different inference resources at different inference stages, when the inference resources cannot infer the inference requests in the second queue, the inference resources may be able to infer the inference requests in the third queue. Therefore, in order to avoid waste of resources, when the number of requests that can be inferred in this round in the second queue is 0, the inference requests in the third queue may be scheduled, that is, according to the priority of the inference requests in the third queue, at least one inference request is selected from the third queue to perform model inference in the decoding stage, thereby avoiding waste of resources and improving resource utilization.

[0146] For the second and third situations above, when resource preemption occurs, for any inference request of the preempted resource, in some implementations, any one of the following operations may be performed:

[0147] The inference request is stored in the third queue; the KV cache occupied by the inference request is unloaded from the GPU to the CPU;

[0148] The inference request is stored in the first queue; the KV cache occupied by the inference request is released; and the tokens generated by the inference request are saved to the CPU.

[0149] Specifically, for the inference request of the preempted resources, the embodiment of the present application provides two preemption modes, namely, recalculation mode and replacement mode. The recalculation mode is to put the inference request of the preempted resources from the second queue back to the first queue, and release the KV cache occupied by the inference request, but retain the generated word element (specifically, it can be saved in the CPU), and then wait for scheduling in the first queue, regenerate the KV cache, and then join the second queue to perform the decoding stage. The specific process can be as follows: Figure 4 As shown. The replacement mode is to put the inference request of the preempted resources from the second queue into the third queue, and the occupied KV cache will be unloaded from the GPU to the CPU. On the one hand, part of the GPU resources can be cleared to infer the inference request for resource preemption. On the other hand, the word units and KV cache generated by the inference request of the preempted resources can be retained in the CPU. Afterwards, as some inference requests in the second queue are completed, the corresponding KV cache can be released. Correspondingly, there will be remaining GPUs. At this time, the inference requests in the third queue can be put back into the second queue to continue decoding, and the occupied KV cache will also be loaded from the CPU to the GPU. The specific process can be as follows. Figure 5 shown.

[0150] Based on the two preemption modes provided in the embodiments of the present application, when resource preemption occurs, one of the preemption modes can be selected for the reasoning request of the preempted resource. If the replacement mode is selected, the reasoning request can be stored in the third queue, and then the KV cache occupied by the reasoning request is unloaded from the GPU to the CPU. If the recalculation mode is selected, the reasoning request can be stored in the first queue, and then the KV cache occupied by the reasoning request is released, and the word element generated by the reasoning request is saved to the CPU. Among them, when selecting the preemption mode, it can be selected according to the actual situation, and no specific limitation is made here on how to select the preemption mode. Optionally, in some embodiments, the preemption mode can be selected according to the current hardware resources. For example, if the computing power of the hardware is more advantageous than the bandwidth, such as a high computing power and a low bandwidth graphics card, so that the efficiency of recalculation is higher than replacement, the recalculation mode is used. If the bandwidth of the hardware is more advantageous than the computing power, such as a low computing power and high bandwidth graphics card, the replacement mode is used.

[0151] The above example illustrates how to select at least one inference request from the second queue to perform model processing in a decoding phase when the third queue is not empty, that is, when resource preemption occurs. It should be noted that in some embodiments, when selecting at least one inference request from the second queue to perform model processing in a decoding phase, there may be certain prerequisites. When it is determined that the third queue is not empty, the prerequisite may be judged first, and then it is determined whether to select at least one inference request from the second queue to perform model inference in a decoding phase according to the judgment result. Specifically, when it is determined that the third queue is not empty, before selecting at least one inference request from the second queue to perform model inference in a decoding phase according to the priority of the inference request in the second queue, it may be determined whether the second queue meets the third condition. Optionally, the third condition may include that the second queue is not empty. If the second queue meets the third condition, such as the second queue is not empty, at least one inference request may be selected from the second queue to perform model inference in a decoding phase according to the priority of the inference request in the second queue. If the second queue does not meet the third condition, for example, the second queue is empty, the second queue may not be scheduled.

[0152] When the second queue does not meet the third condition, the inference requests in the third queue can be processed optionally, that is, according to the priority of the inference requests in the third queue, at least one inference request is selected from the third queue to perform a model inference in the decoding stage to avoid wasting resources.

[0153] In this embodiment, when the third queue is not empty, if the inference request in the second queue does not occupy resources, or the number of requests that can be inferred in the second queue is 0, or the second queue does not meet the third condition, the third queue can be scheduled, that is, according to the priority of the inference request in the third queue, at least one inference request is selected from the third queue to perform a model inference in the decoding stage. When selecting at least one inference request from the third queue to perform a model inference in the decoding stage according to the priority of the inference request in the third queue, in some embodiments, the following steps may be included:

[0154] Prioritizing the inference requests in the third queue according to the priorities of the inference requests in the third queue;

[0155] Starting from the inference request with the highest priority, determine in turn whether the inference resource can infer the inference request;

[0156] When the inference resource is capable of inferring the inference request, the KV cache corresponding to the inference request is loaded from the CPU to the GPU;

[0157] A model inference of the decoding phase is performed for at least one inference request that the inference resource can infer.

[0158] Inference resources may include but are not limited to GPU resources. Since the inference requests in the third queue have been pre-filled and are in the decoding stage, and in the decoding stage, the main factor affecting whether the inference request can be inferred in this round is whether the inference resources are sufficient, therefore, when judging whether the inference request in the third queue can be inferred in this round, it is specifically to judge whether the inference resources can infer the inference request in the third queue, that is, to judge whether the inference resources can meet the demand for inference resources by the inference request in the third queue.

[0159] When judging whether the reasoning resource can infer the reasoning request in the third queue, you can start from the reasoning request with the highest priority in the third queue and judge whether the reasoning resource can infer the reasoning request. If the reasoning resource can infer the reasoning request with the highest priority, you can continue to judge whether the reasoning resource can infer the reasoning request with the second highest priority. If the reasoning resource can infer the reasoning request with the second highest priority, you can continue to judge whether the reasoning resource can infer the reasoning request with the third highest priority, and so on, and so on, until the reasoning resource cannot infer a certain reasoning request (after each judgment on whether reasoning can be performed, if a certain reasoning request can be inferred, the corresponding reasoning resources will be deducted. Therefore, in the process of cyclic judgment, the reasoning resources are constantly decreasing).

[0160] After the above-mentioned cyclic judgment is performed on the inference requests in the third queue, for one or more inference requests that can be inferred, a model inference of the decoding stage can be performed on them in this round, and for one or more inference requests that cannot be inferred, the next round of model inference can be waited. Among them, when a model inference of the decoding stage is performed on the inference request that can be inferred, based on the replacement mode recorded above when resource preemption occurs, the KV cache corresponding to the inference request can be loaded from the CPU to the GPU to continue decoding.

[0161] The above describes in detail how to select at least one inference request from multiple inference requests in the request queue (including the first queue, the second queue, and the third queue) to perform model inference in the inference phase. In order to facilitate understanding of the scheduling logic of inference requests in the request queue in the embodiment of the present application, a more specific implementation method will be used as an example for explanation. For details, please refer to Figure 6 . Figure 6 is a flow chart of scheduling inference requests in a request queue according to an embodiment of the present application. Figure 6 The specific implementation of the illustrated embodiment is as follows.

[0162] Step S1: Determine whether the third queue is empty.

[0163] When scheduling inference requests in the first queue, the second queue, and the third queue, based on the overall scheduling logic of the three queues (when no resource preemption occurs, the inference requests in the first queue are prioritized, and then the situations in the second and third queues are processed; when resource preemption occurs, the inference requests in the second queue are prioritized, and then the preempted inference requests in the third queue are processed, and finally the inference requests in the first queue are processed), it is possible to first determine whether the third queue is empty.

[0164] If the third queue is empty, it can be said that resource preemption has not occurred, and step S2 can be executed at this time. If the third queue is not empty, it can be said that resource preemption has occurred, and step S3 can be executed at this time.

[0165] Step S2: Determine whether the scheduling interval threshold of the first queue is reached and the first queue is not empty.

[0166] If the scheduling interval threshold of the first queue has been reached and the first queue is not empty, the inference request in the first queue can be scheduled. If the scheduling interval threshold of the first queue has not been reached or the first queue is empty, step S3 can be executed.

[0167] When scheduling the inference requests in the first queue, the priorities of the inference requests in the first queue may be calculated and sorted in descending order of priority. When calculating the priorities of the inference requests in the first queue, the following priority function may be used for calculation:

[0168] .

[0169] Among them, b=(alpha+beta) / starved_threshold, c=(alpha+beta)(1+1 / starved_threshold).

[0170] After calculating the priority of the inference request in the first queue and sorting the inference request in descending order according to the priority, the inference request in the first queue can be looped to determine whether the inference resource can infer the current request. If so, the current request is added to the queue to be inferred and resources are allocated to it, and the loop continues; if not, the loop is exited. The queue to be inferred is used to store the requests that need to be inferred in this round.

[0171] After the above-mentioned cyclic judgment of the inference requests in the first queue, it can be further judged whether the queue to be inferred is not empty. If the queue to be inferred is not empty, it can be said that the number of requests that can be inferred in this round in the first queue is not 0. At this time, the queue to be inferred can be output, and the inference requests in the queue to be inferred will be pre-filled by the inference server. Model inference. If the queue to be inferred is empty, it can be said that the number of requests that can be inferred in this round in the first queue is 0. At this time, step S3 can be executed to avoid resource waste.

[0172] Step S3: Determine whether the second queue is not empty.

[0173] If the second queue is empty, that is, the number of inference requests in the second queue is 0, step S4 can be executed to avoid wasting resources. If the second queue is not empty, that is, the number of inference requests in the second queue is not 0, the priority of the inference requests in the second queue can be calculated and sorted in descending order of priority. When calculating the priority of the inference requests in the second queue, the following priority function can be used for calculation:

[0174] .

[0175] Among them, b=(alpha+beta) / starved_threshold, c=(alpha+beta)(1+1 / starved_threshold).

[0176] After calculating the priority of the inference request in the second queue and sorting the inference requests in descending order according to the priority, the inference requests in the second queue can be looped to determine whether the inference resources can infer the current request. If so, the current request is added to the queue to be inferred and resources are allocated to it, and the loop continues; if not, the lowest priority request in the second queue is preempted in turn, and the cache of the preempted request is released until the current request can be inferred. Among them, for the inference request whose resources are preempted, it can be stored in the third queue, and then the KV cache occupied by the inference request is unloaded from the GPU to the CPU, or it can be stored in the first queue, and then the KV cache occupied by the inference request is released, and the word element generated by the inference request is saved to the CPU ( Figure 6 The corresponding processing flow is not shown).

[0177] After the above-mentioned cyclic judgment is performed on the inference requests in the second queue, it can be further judged whether resource preemption occurs in this round. If resource preemption occurs, it can be said that there are no remaining inference resources to infer the inference requests in the third queue. At this time, the queue to be inferred can be output, and the inference requests in the queue to be inferred will be subjected to a model inference in the decoding stage by the inference server. If resource preemption does not occur, it can be said that there may be remaining inference resources. In order to avoid resource waste, step S4 can be executed.

[0178] Step S4: Calculate the priorities of the inference requests in the third queue and sort them in descending order of priority.

[0179] When calculating the priority of the inference request in the third queue, it can be calculated by the following priority function:

[0180] .

[0181] Among them, b=(alpha+beta) / starved_threshold, c=(alpha+beta)(1+1 / starved_threshold).

[0182] After calculating the priority of the inference request in the third queue and sorting the inference request in descending order according to the priority, the inference request in the third queue can be looped to determine whether the inference resource can infer the current request. If so, the KV cache of the current request is loaded from the CPU to the GPU, and the current request is added to the queue to be inferred and resources are allocated to it, and the loop continues. If not, the loop is exited and the queue to be inferred is output. The inference request in the queue to be inferred will be subjected to a decoding stage model inference by the inference server.

[0183] Based on Figure 6 After a round of model reasoning, the process shown in the figure can be based on Figure 6 The process shown performs the next round of model reasoning, thereby achieving iterative priority scheduling of reasoning requests, thereby optimizing resource allocation, reducing response time, and improving overall service quality.

[0184] In some implementations, after selecting at least one inference request from multiple inference requests to perform model inference in an inference phase, the following operations may be performed:

[0185] Determine whether at least one inference request has completed model inference;

[0186] For any inference request, do the following:

[0187] When the inference request has completed model inference, the KV cache corresponding to the inference request is released;

[0188] If the inference request has not completed the model inference, the inference request is placed in the request queue for model inference in the next inference phase.

[0189] When determining whether at least one inference request has completed model inference, taking an inference request as an example, it can be determined whether an end identifier is encountered when performing model inference on the inference request. If so, it can be determined that the model inference has been completed. Alternatively, it can be determined whether the output result obtained in this inference phase reaches the output length specified by the user. If so, it can be determined that the model inference has been completed. Here, no specific limitation is made on how to determine whether the inference request has completed model inference.

[0190] After determining whether at least one inference request has completed model inference, for any of the inference requests, if the model inference has been completed, the KV cache corresponding to the inference request can be released, that is, the GPU resources occupied by the inference request can be released. If the model inference is not completed, the inference request can be placed in the request queue and the model inference of the next inference stage can be performed. For example, after the inference request is pre-filled once, if the inference request has not completed the model inference, the inference request can be placed in the second queue of the request queue (the second queue is used to store the inference request that has completed the pre-filling and is in the decoding stage) so as to decode the inference request. For another example, after the inference request is decoded once, if the inference request has not completed the model inference, the inference request can be placed back in the second queue of the request queue so as to perform the next decoding of the inference request. Among them, when the model inference of the next inference stage is performed on the inference request, the priority of multiple inference requests in the request queue can be recalculated, and at least one inference request can be selected from the multiple inference requests according to the priority to perform the model inference of the next inference stage. The specific implementation method can refer to the relevant contents recorded in S306 and S308 above, which will not be described in detail here. In this way, iterative priority scheduling of inference requests can be achieved, which can further optimize resource allocation, reduce response time, and improve overall service quality.

[0191] Based on the contents recorded in S302 to S308 above, it can be seen that the technical solution provided by the embodiment of the present application is:

[0192] (1) A comprehensive priority scheduling algorithm is proposed, which comprehensively considers factors such as the waiting time of inference requests, the corresponding user level, and the task difficulty. The algorithm is optimized through a large number of experiments to make it more flexible and generalizable to various actual production environments.

[0193] (2) A scheduling algorithm that takes into account a time tolerance threshold (starvation tolerance threshold) is proposed. Specifically, the embodiment of the present application introduces a time tolerance threshold in the priority function to avoid user starvation, while allowing the service provider to adjust the preference between efficiency and fairness according to demand.

[0194] (3) A multi-scenario generalized adaptive scheduling algorithm is proposed. Specifically, the priority function of the embodiment of the present application includes multiple hyperparameters, and these hyperparameters can derive matching values ​​according to different scenarios and experimental data. By adjusting these hyperparameters, it is possible to flexibly balance between efficiency and fairness to adapt to the needs of various scenarios.

[0195] (4) An iterative-level scheduling method is proposed. Specifically, the embodiment of the present application focuses on the autoregressive characteristics of the large language model and implements iterative-level scheduling. In each round, requests are calculated according to priority, and then the requests for reasoning in this round are selected according to priority. The reasoning results of each round are returned in a streaming manner, thereby optimizing resource allocation, improving user experience, and significantly reducing overall response latency.

[0196] Based on the above four points, the embodiments of the present application can significantly improve the system's response speed and resource utilization during large model reasoning, taking into account both performance and fairness, and adapting to a variety of actual production environments.

[0197] In order to facilitate the understanding of the technical solution provided by the embodiment of the present application, the following will be illustrated with examples in conjunction with specific scenarios, and the corresponding scenario data is shown in Table 1. In view of the fact that the complete request data is too large, the specific data of all inference requests are not displayed here, and only the key data are listed, namely, the unique identifier request_id of each request (which can be generated by the user interface of the inference server), the user level, the length of the request input prompt_len, and the output length of the request output_len. At the same time, an example of a complete request sent by a user is also shown.

[0198] Table 1

[0199]

[0200] An example of an inference request is as follows:

[0201] curl http: / / localhost:8000 / v1 / completions \

[0202] -H "Content-Type: application / json"\

[0203] -d '{

[0204] "model": " / local / code / model_tmp / llama2-13B",

[0205] "prompt": "What should I do if I don't sleep well?",

[0206] "max_tokens": 500,

[0207] "temperature": 0

[0208] }'

[0209] When processing an inference request, the specific process is as follows:

[0210] Step 1000: Create a large model inference service on the artificial intelligence platform.

[0211] Step 1001: A user sends an inference request from a web page in the user's computer to a large model inference service.

[0212] Step 1002: The large model inference service receives the inference request and parses the inference request.

[0213] Step 1003: tokenize (segment or compile) the prompt of the reasoning request, that is, convert characters that humans can understand into digital tokens that can be inferred by the large model.

[0214] Example: "prompt": "What should I do if I have trouble sleeping?" ---------tokenizer----->prompt_token_ids: [1, 29871, 234, 160, 164, 234, 159, 163, 30413, 31076, 233, 131,145, 31882, 232, 141, 161, 30882].

[0215] Step 1004: Put the inference request into the first queue to wait for scheduling and inference.

[0216] Step 1005: Scheduling and reasoning.

[0217] The first round of scheduling:

[0218] Step 100501: Calculate the priority of each inference request in the first queue, and the calculation results are shown in Table 2. Wherein, starved_threshold=120ms, for the convenience of representation, the priority function is scaled here, and the scaling value is 4096.

[0219] Table 2

[0220]

[0221] Step 100502: sort the multiple inference requests in the first queue in descending order of priority. The sorting result is shown in Table 3.

[0222] Table 3

[0223]

[0224] Step 100503: loop the inference requests in the first queue, determine whether the inference cache (inference resource) can infer the current inference request, if so, add the current inference request to the queue to be inferred, and continue looping to determine other inference requests in the first queue; if not, exit the loop.

[0225] In this embodiment, 9 inference requests may be selected from the first queue, as shown in Table 4.

[0226] Table 4

[0227]

[0228] Step 100504: Call the model executor to perform reasoning. The reasoning process is as follows Figure 7 As shown, a token is generated for each reasoning.

[0229] Before and after prefilling the 9 inference requests in the first queue, the status of the first queue, the second queue, and the third queue are shown in Table 5.

[0230] Table 5

[0231]

[0232] Step 100505: Determine whether there is any reasoning request in this round of reasoning to complete the model reasoning. If so, remove the request from the first queue, globally update the request status, and recycle resources. Put the unfinished requests in this round of reasoning into the second queue.

[0233] Second round of scheduling:

[0234] Step 100506: Determine whether the remaining inference resources can continue to be pre-filled. In this embodiment, it is taken as an example that the pre-filling cannot continue. In this case, the second queue can be scheduled.

[0235] Step 100507: Calculate the priority of each inference request in the second queue.

[0236] Step 100508: Sort the inference requests in the second queue according to priority from high to low.

[0237] Step 100509: loop the inference requests in the second queue and determine whether the inference cache (inference resource) can infer the current inference request. If so, add the current inference request to the queue to be inferred and continue the loop; if not, preempt the inference request with the lowest priority in the second queue in turn, release the cache of the preempted inference request, until the current inference request can be inferred. If all the inference requests with a lower priority than the current inference request are preempted and still do not meet the resource requirements for inferring the current inference request, the current inference request will also be treated as preempted. In this embodiment, no preemption occurs in the second queue, and all 9 inference requests can be inferred.

[0238] Step 100510: Call the model executor for inference.

[0239] Continue to execute according to the above scheduling process. Table 6 lists the states of the first queue, the second queue and the third queue during the scheduling process. The states of the first queue, the second queue and the third queue in Table 6 are the states before this inference stage. After inference, the states become the states of the next row.

[0240] Table 6

[0241]

[0242] From Table 6, we can see that no resource preemption occurs during the entire inference process. In fact, a careful analysis of the scheduling strategy reveals that the probability of preemption is not high (because in the decoding stage of model inference, the inference resources required for inference requests are generally small, so the probability of sending resource preemption is small).

[0243] Step 1006: Detokenize the inference request that has completed inference, that is, convert the digital token into characters that can be understood by humans.

[0244] Step 1007: Post-processing, packaging the inference request and the result of the inference request.

[0245] Step 1008: Return the inference result to the user.

[0246] Example of inference results:

[0247] There may be many reasons for poor sleep, including lifestyle, diet, stress, mental state, etc. Here are some suggestions that may help:

[0248] 1. Maintain a fixed sleep schedule: The sleep schedule should be fixed every night and should not be changed at will. This can help your body and mind establish a fixed sleep pattern.

[0249] 2. Create a comfortable sleeping environment: Your bed and bedroom should be comfortable, quiet and warm. You can use a soft mattress, soft sweaters and soft blankets to create a comfortable sleeping environment.

[0250] 3. Avoid eating dinner and drinking alcohol: Eating dinner and drinking alcohol may make it difficult for you to sleep at night because they may cause stimulation in your body and make it difficult for you to fall asleep.

[0251] 4. Avoid using electronic devices: Using electronic devices such as mobile phones, computers and televisions may make it difficult for you to fall asleep because their screens and sounds may disturb your sleep.

[0252] In summary, after receiving the reasoning request, the embodiment of the present application can add the reasoning request to the request queue, and the request queue is used to store multiple reasoning requests, and then determine the priority of the multiple reasoning requests according to the task difficulty, waiting time and corresponding user level of the multiple reasoning requests, and select at least one reasoning request from the multiple reasoning requests according to the priority to perform a model reasoning of the reasoning stage. Since when processing the reasoning request, the three factors of the task difficulty, waiting time and corresponding user level of the reasoning request can be comprehensively considered to determine the priority of the reasoning request, and at least one reasoning request can be selected for reasoning according to the priority, therefore, not only can the preemptive scheduling of the reasoning request be realized, but also the user demand and resource utilization can be taken into account, thereby improving the overall performance and stability of the system. In addition, after selecting at least one reasoning request according to the priority, when reasoning at least one reasoning request, since the model reasoning of the reasoning stage can be performed once, such as one pre-filling or one decoding, therefore, the priority scheduling of the iteration level of the reasoning request can be realized, so that the resource allocation can be further optimized, the response time can be reduced, and the overall service quality can be improved. By optimizing the scheduling mechanism of the reasoning service, the embodiment of the present application can significantly enhance the adaptability and reliability of the system, thereby promoting the wide application and popularization of large models in various fields.

[0253] The request processing method provided in the embodiment of the present application can be provided in the form of an inference service, which can be deployed in an artificial intelligence platform. In some embodiments, when a user performs model inference, he can use the web application interface to upload the inference request to the inference service through the network. The inference service can use the method provided in the embodiment of the present application to schedule the inference request and feed back the inference result to the user through the network. The corresponding system architecture (application environment) example can be as follows: Figure 8 shown.

[0254] The present application also provides a request processing system. In some implementations, the structure of the request processing system can be as follows: Fig. 9 shown. Fig. 9 The request processing system shown includes a user interface, a parameter module and a large model reasoning engine (LLM Engine). These three modules will be described separately below.

[0255] The large model inference engine is a key part of the system and the core part of large model inference. It can use iteration-level scheduling and efficient memory management to maximize service throughput. The large model inference engine is responsible for receiving inference requests and generating inference results, which involves scheduling and resource allocation of the entire inference process. Its specific functions include:

[0256] (1) Initialization: configuration rationalization check, tokenizer model initialization, large model loading, request scheduling initialization, distributed parallel reasoning environment initialization, worker process initialization, and cache management initialization.

[0257] (2) Manage the inference process of inference requests: add inference requests to the scheduler's request pool, call the scheduler to schedule the inference requests, call the model executor to perform model inference, encapsulate the inference results, return the inference results, etc.

[0258] (3) Resource management: allocate and release CPU main memory and GPU video memory, and monitor the remaining status of CPU main memory and GPU video memory.

[0259] The large model inference engine coordinates the entire inference process and resource allocation, among which the two core functions of inference and resource management can be implemented by distributed systems and central controllers.

[0260] The distributed system is mainly responsible for the initialization of the distributed environment, distributed parallel reasoning, model initialization, model loading, physical cache management, and model execution. The physical cache management and model execution functions are implemented through each worker process (Worker), where the worker process (Worker) is an abstraction of the hardware GPU. At the software level, it refers to the Worker instance (each process on the GPU maintains its own Worker instance). In each Worker instance, the following two important instances are controlled:

[0261] Model executor (model_runner): responsible for loading models and performing inference.

[0262] Cache Engine: Responsible for managing the KV cache physical blocks on the GPU and CPU. The cache manager in the scheduler determines the allocation and release of physical blocks, and the cache engine performs data management on the hardware based on the allocation results of the physical blocks.

[0263] The central controller is mainly responsible for request scheduling, logical management and allocation of memory resources. There are two main components under the central controller: the cache manager and the scheduler. The cache manager is responsible for allocating KV cache physical blocks for inference requests and logically managing KV cache physical blocks virtually. The main function of the scheduler is to implement which inference requests are passed to the model executor for model inference during an inference phase. Among them, the scheduler is the core part of the embodiment of this application, and its scheduling logic can be found in Figure 3 The corresponding contents in the illustrated embodiment will not be described in detail here.

[0264] In some implementations, the structure of the scheduler may be as follows: Fig.10 shown. Fig.10 The scheduler shown in includes scheduling budget, preemption mode, scheduling strategy and request queue. The scheduler needs to select the request for the next reasoning from the request queue according to the scheduling budget, preemption mode and scheduling strategy. The scheduling budget records the maximum number of requests and tokens that can be accommodated in the current reasoning round, and also records the number of requests and tokens that have been accommodated in this round of reasoning. The scheduling budget is responsible for judging whether the request can be executed currently based on these attribute values. It should be noted that whether the current request has enough resources to execute also needs to call the function in the cache manager to judge.

[0265] based on Fig. 9 In the request processing system shown, when the large model inference engine receives a new inference request, it will put the inference request into the first queue to wait for pre-filling. After the pre-filling is completed, it will enter the second queue for loop decoding until the end. The specific implementation process is as follows:

[0266] (1) The user calls the user interface to pass the request to the Large Model Reasoning Engine (LLM Engine);

[0267] (2) The large model inference engine receives requests, uses the scheduler to control the generation process, the cache manager manages cache resources, the cache engine physically allocates resources, the model executor performs model inference, and finally generates results;

[0268] (3) Return the generated results to the user;

[0269] (4) Various parameters required for reasoning and resource allocation in the above process are defined in the parameter module, including model parameters, sampling parameters, request scheduling parameters, distributed parallel reasoning parameters, etc.

[0270] To sum up, the technical solution provided in the embodiments of the present application can significantly improve resource utilization, reduce latency, enhance system adaptability and reliability by optimizing the scheduling mechanism of reasoning services, thereby promoting the widespread application and popularization of large models in various fields.

[0271] The above describes specific embodiments of the present application. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in an order different from that in the embodiments and still achieve the desired results. In addition, the processes depicted in the accompanying drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0272] Fig.11 This is a schematic diagram of the structure of an electronic device according to an embodiment of the present application. Fig.11 At the hardware level, the electronic device includes a processor, and optionally also includes an internal bus, a network interface, and a memory. The memory may include a memory, such as a high-speed random access memory (RAM), and may also include a non-volatile memory (non-volatile memory), such as at least one disk storage. Of course, the electronic device may also include hardware required for other services.

[0273] The processor, network interface and memory can be interconnected through an internal bus, which can be an ISA (Industry Standard Architecture) bus, a PCI (Peripheral Component Interconnect) bus or an EISA (Extended Industry Standard Architecture) bus. The bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Fig.11 Only one bidirectional arrow is used in the diagram, but this does not mean that there is only one bus or only one type of bus.

[0274] The memory is used to store the program. Specifically, the program may include a program code, and the program code includes a computer operation instruction. The memory may include a memory and a non-volatile memory, and provides instructions and data to the processor.

[0275] The processor reads the corresponding computer program from the non-volatile memory into the memory and then runs it, forming a request processing device at the logical level. The processor executes the program stored in the memory and is specifically used to perform the following operations:

[0276] receiving an inference request, wherein the inference request is used to request to use a large model to perform model inference;

[0277] Adding the inference request to a request queue, where the request queue is used to store multiple inference requests;

[0278] Determining priorities of the multiple reasoning requests according to reasoning information of the multiple reasoning requests, wherein the reasoning information includes task difficulty, waiting time, and corresponding user level of the multiple reasoning requests;

[0279] According to the priorities of the multiple inference requests, at least one inference request is selected from the multiple inference requests to perform model inference in a reasoning phase.

[0280] The above application Fig.11 The method executed by the request processing device disclosed in the illustrated embodiment can be applied to a processor or implemented by a processor. The processor may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method can be completed by an integrated logic circuit of hardware in the processor or an instruction in the form of software. The above processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The methods, steps and logic block diagrams disclosed in this application can be implemented or executed. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor, etc. The steps of the method disclosed in this application can be directly embodied as a hardware decoding processor for execution, or can be executed by a combination of hardware and software modules in the decoding processor. The software module can be located in a storage medium mature in the art such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory, or an electrically erasable programmable memory, a register, etc. The storage medium is located in the memory, and the processor reads the information in the memory and completes the steps of the above method in combination with its hardware.

[0281] The electronic device may also perform Figures 3 to 6 The method of the embodiment shown in the figure is implemented by the request processing device in Figures 3 to 6 The functions in the illustrated embodiments will not be described in detail in this application.

[0282] Of course, in addition to software implementation methods, the electronic device of the present application does not exclude other implementation methods, such as logic devices or a combination of software and hardware, etc. That is to say, the execution subject of the following processing flow is not limited to each logic unit, but can also be hardware or logic devices.

[0283] The present application also proposes a computer-readable storage medium, which stores one or more programs, wherein the one or more programs include instructions, which, when executed by a portable electronic device including a plurality of application programs, enable the portable electronic device to execute Figures 3 to 6 The method of the embodiment shown is specifically used to perform the following operations:

[0284] receiving an inference request, wherein the inference request is used to request to use a large model to perform model inference;

[0285] Adding the inference request to a request queue, where the request queue is used to store multiple inference requests;

[0286] Determining priorities of the multiple reasoning requests according to reasoning information of the multiple reasoning requests, wherein the reasoning information includes task difficulty, waiting time, and corresponding user level of the multiple reasoning requests;

[0287] According to the priorities of the multiple inference requests, at least one inference request is selected from the multiple inference requests to perform model inference in a reasoning phase.

[0288] Fig.12 is a schematic diagram of the structure of a request processing device 120 according to an embodiment of the present application. Fig.12 In a software implementation, the request processing device 120 may include: a receiving module 121, an adding module 122, a determining module 123 and a processing module 124, wherein:

[0289] A receiving module 121 receives an inference request, wherein the inference request is used to request to use a large model to perform model inference;

[0290] An adding module 122, adding the inference request to a request queue, wherein the request queue is used to store multiple inference requests;

[0291] A determination module 123 determines priorities of the multiple reasoning requests according to reasoning information of the multiple reasoning requests, wherein the reasoning information includes task difficulty, waiting time, and corresponding user level of the multiple reasoning requests;

[0292] The processing module 124 selects at least one inference request from the multiple inference requests according to the priorities of the multiple inference requests to perform model inference in an inference phase.

[0293] In some implementations, the determining module 123 determines the priorities of the multiple reasoning requests according to the reasoning information of the multiple reasoning requests, including:

[0294] For each of the inference requests, the following operations are performed:

[0295] Determining a first priority according to the task difficulty of the reasoning request;

[0296] Determining a second priority according to the waiting time of the inference request and the first parameter;

[0297] Determining a third priority level according to the user level corresponding to the reasoning request and the second parameter;

[0298] A priority of the inference request is determined according to the first priority, the second priority, and the third priority.

[0299] In some implementations, the task difficulty of the reasoning request is related to at least one of the following:

[0300] The length of the prompt word of the inference request;

[0301] User-specified maximum output length;

[0302] The upper limit of the request length that the large model can reason about;

[0303] The remaining number of decoding times of the inference request by the large model.

[0304] In some implementations, the determining module 123 determines the first priority according to the task difficulty of the reasoning request, including:

[0305] Determine a state of the inference request, the state comprising: the inference request is not pre-filled and the inference request is pre-filled and in a decoding stage;

[0306] When the state of the inference request is that the inference request is not pre-filled, determining the first priority according to the prompt word length, the maximum output length, and the upper limit of the request length;

[0307] When the state of the inference request is that the inference request has completed pre-filling and is in the decoding stage, the first priority is determined according to the remaining decoding times and the upper limit of the request length.

[0308] In some implementations, the determination module 123 determines the first priority according to the prompt word length, the maximum output length, and the upper limit of the request length, including:

[0309] Determine a first ratio of a difference between the upper limit of the request length and the length of the prompt word to the upper limit of the request length;

[0310] Determine a second ratio of a difference between the upper limit of the request length and the maximum output length to the upper limit of the request length;

[0311] The first ratio and the second ratio are added according to the specified first coefficient and the second coefficient, and the first priority is determined according to the addition result.

[0312] In some implementations, the determining module 123 determines the first priority according to the remaining number of decoding times of the inference request by the large model and the upper limit of the request length that can be inferred by the large model, including:

[0313] Determine a third ratio of a difference between the upper limit of the request length and the remaining number of decoding times to the upper limit of the request length;

[0314] The first priority is determined according to the third ratio and a specified third coefficient.

[0315] In some embodiments, the first parameter and the second parameter are related to a time tolerance threshold, the time tolerance threshold is adjustable, and the first parameter and the second parameter are adjusted by adjusting the time tolerance threshold;

[0316] Wherein, in the case where the first parameter and the second parameter are adjusted by adjusting the time tolerance threshold, the priorities of the multiple inference requests satisfy at least one of the following:

[0317] When the waiting time is the same, the higher the user level corresponding to the inference request, the higher the priority;

[0318] In the case where the user levels are the same, if the waiting time of the inference request exceeds the time tolerance threshold, the priority is higher than the priority of the newly arrived inference request.

[0319] In some embodiments, the request queue includes a first queue, a second queue, and a third queue, the first queue is used to store inference requests that have not been pre-filled, the second queue is used to store inference requests that have completed the pre-filling stage and are in the decoding stage, and the third queue is used to store inference requests whose resources are preempted in the decoding stage; the processing module 124 selects at least one inference request from the multiple inference requests to perform model inference in an inference stage according to the priority of the multiple inference requests, including:

[0320] Determining whether the third queue is empty;

[0321] When the third queue is empty, selecting at least one inference request from the first queue according to the priority of the inference request in the first queue to perform a model inference in the pre-filling stage;

[0322] When the third queue is not empty, at least one inference request is selected from the second queue according to the priority of the inference requests in the second queue to perform a model inference in the decoding stage.

[0323] In some implementations, the processing module 124 selects at least one inference request from the first queue to perform a model inference in a pre-filling phase according to the priority of the inference request in the first queue, including:

[0324] Prioritizing the inference requests in the first queue according to the priorities of the inference requests in the first queue;

[0325] Starting from the reasoning request with the highest priority, determine in turn whether the reasoning request can be reasoned in this round;

[0326] A model inference in the pre-filling phase is performed for at least one inference request that can be inferred in this round.

[0327] In some implementations, the processing module 124, starting from the reasoning request with the highest priority, sequentially determines whether the reasoning request can be reasoned in the current round, including:

[0328] Determining whether the inference request satisfies a first condition;

[0329] When the inference request satisfies the first condition, determining that the inference request can be inferred in this round;

[0330] The first condition includes at least one of the following:

[0331] The number of requests that have been accommodated in this round is less than the maximum number of requests that can be accommodated in this round;

[0332] The number of tokens already accommodated in this round is less than the maximum number of tokens that can be accommodated in this round;

[0333] There are sufficient inference resources to infer the inference request.

[0334] In some embodiments, when the number of inference requests that can be inferred in this round is 0, the processing module 124 selects at least one inference request from the second queue according to the priority of the inference requests in the second queue to perform a model inference in the decoding stage.

[0335] In some implementations, the processing module 124, before selecting at least one inference request from the first queue to perform a model inference in a pre-filling phase according to the priority of the inference request in the first queue, determines whether the first queue satisfies a second condition, where the second condition includes that the first queue is not empty and / or a scheduling interval threshold of the first queue has been reached;

[0336] The step of selecting at least one inference request from the first queue to perform a model inference in a pre-filling phase according to the priority of the inference request in the first queue includes:

[0337] When the first queue satisfies the second condition, at least one inference request is selected from the first queue according to the priority of the inference requests in the first queue to perform a model inference in the pre-filling stage.

[0338] In some implementations, when the first queue does not satisfy the second condition, the processing module 124 selects at least one inference request from the second queue to perform a model inference in a decoding phase according to the priority of the inference requests in the second queue.

[0339] In some implementations, the processing module 124 selects at least one inference request from the second queue to perform a model inference in a decoding phase according to the priority of the inference request in the second queue, including:

[0340] Prioritizing the inference requests in the second queue according to the priorities of the inference requests in the second queue;

[0341] Starting from the inference request with the highest priority, determining in turn whether the inference resource can infer the inference request;

[0342] When the reasoning resource cannot reason the reasoning request, resource preemption is performed starting from the reasoning request with the lowest priority in the second queue until the reasoning resource can reason the reasoning request;

[0343] A model inference at a decoding stage is performed on at least one inference request that can be inferred by the inference resource.

[0344] In some implementations, for any inference request that is preempted by an execution resource, the processing module 124 is configured to perform any of the following:

[0345] storing the inference request in the third queue; and unloading the KV cache occupied by the inference request from the GPU to the CPU;

[0346] The inference request is stored in the first queue; the KV cache occupied by the inference request is released; and the word element generated by the inference request is saved to the CPU.

[0347] In some embodiments, the processing module 124 is further configured to:

[0348] In a case where resource preemption does not occur for at least one inference request selected from the second queue, selecting at least one inference request from the third queue according to the priority of the inference request in the third queue to perform a model inference in a decoding phase;

[0349] When the number of inference requests that can be inferred by the inference resource is 0, at least one inference request is selected from the third queue according to the priority of the inference requests in the third queue to perform a model inference in the decoding stage.

[0350] In some implementations, the processing module 124, before selecting at least one inference request from the second queue to perform a model inference at a decoding stage according to the priority of the inference request in the second queue, determines whether the second queue satisfies a third condition, where the third condition includes that the second queue is not empty;

[0351] The selecting at least one inference request from the second queue to perform a model inference in a decoding phase according to the priority of the inference request in the second queue includes:

[0352] When the second queue satisfies the third condition, at least one inference request is selected from the second queue according to the priority of the inference requests in the second queue to perform a model inference in the decoding stage.

[0353] In some embodiments, when the second queue does not satisfy the third condition, the processing module 124 selects at least one inference request from the third queue to perform a model inference in the decoding stage according to the priority of the inference requests in the third queue.

[0354] In some implementations, the processing module 124 selects at least one inference request from the third queue to perform a model inference in a decoding phase according to the priority of the inference request in the third queue, including:

[0355] Prioritizing the inference requests in the third queue according to the priorities of the inference requests in the third queue;

[0356] Starting from the inference request with the highest priority, determining in turn whether the inference resource can infer the inference request;

[0357] If the inference resource is capable of inferring the inference request, loading the KV cache corresponding to the inference request from the CPU to the GPU;

[0358] A model inference at a decoding stage is performed on at least one inference request that can be inferred by the inference resource.

[0359] In some implementations, after the processing module 124 selects at least one inference request from the multiple inference requests to perform a model inference in the inference phase:

[0360] Determining whether the at least one inference request has completed model inference;

[0361] For any inference request, do the following:

[0362] When the inference request has completed model inference, releasing the KV cache corresponding to the inference request;

[0363] In the case that the inference request has not completed the model inference, the inference request is put into the request queue to perform model inference in the next inference phase.

[0364] The request processing device 120 provided in the present application can also execute Figures 3 to 6 The method is implemented by the request processing device 120 Figures 3 to 6 The functions of the illustrated embodiments will not be described in detail in this application.

[0365] The present application also proposes a computer program product, which includes a non-transitory computer-readable storage medium storing a computer program, and the computer program is operable to enable a computer to execute part or all of the steps in the above request processing method embodiment.

[0366] In short, the above description is only a preferred embodiment of the present application and is not intended to limit the protection scope of the present application. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application.

[0367] The systems, devices, modules or units described in the above embodiments may be implemented by computer chips or entities, or by products with certain functions. A typical implementation device is a computer. Specifically, the computer may be, for example, a personal computer, a laptop computer, a cellular phone, a camera phone, a smart phone, a personal digital assistant, a media player, a navigation device, an email device, a game console, a tablet computer, a wearable device, or a combination of any of these devices.

[0368] Computer readable media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.

[0369] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0370] Each embodiment in this application is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the system embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

Claims

1. A request processing method, comprising: receiving an inference request, wherein the inference request is used to request to use a large model to perform model inference; Adding the inference request to a request queue, where the request queue is used to store multiple inference requests; Determining priorities of the multiple inference requests according to inference information of the multiple inference requests, the inference information including task difficulty, waiting time, and corresponding user level of the multiple inference requests, wherein the task difficulty is related to at least one of a prompt word length of the inference request, a maximum output length specified by a user, an upper limit of a request length that can be inferred by the large model, and a remaining number of decoding times of the inference request by the large model; According to the priorities of the multiple reasoning requests, selecting at least one reasoning request from the multiple reasoning requests to perform model reasoning in a reasoning phase; Wherein, determining the priorities of the multiple reasoning requests according to the reasoning information of the multiple reasoning requests includes: For each of the inference requests, determining a state of the inference request, the state comprising: the inference request is not pre-filled and the inference request is pre-filled and is in a decoding stage; When the state of the inference request is that the inference request is not pre-filled, determining a first priority according to the prompt word length, the maximum output length, and the upper limit of the request length; When the state of the inference request is that the inference request has completed pre-filling and is in the decoding stage, the first priority is determined according to the remaining decoding times and the upper limit of the request length.

2. The method of claim 1, wherein determining priorities of the multiple reasoning requests based on the reasoning information of the multiple reasoning requests further comprises: For each of the inference requests, the following operations are performed: Determining a second priority according to the waiting time of the inference request and the first parameter; Determining a third priority level according to the user level corresponding to the reasoning request and the second parameter; A priority of the inference request is determined according to the first priority, the second priority, and the third priority.

3. The method according to claim 1, wherein determining the first priority according to the prompt word length, the maximum output length and the upper limit of the request length comprises: Determine a first ratio of a difference between the upper limit of the request length and the length of the prompt word to the upper limit of the request length; Determine a second ratio of a difference between the upper limit of the request length and the maximum output length to the upper limit of the request length; The first ratio and the second ratio are added according to the specified first coefficient and the second coefficient, and the first priority is determined according to the addition result.

4. The method according to claim 1, wherein determining the first priority according to the remaining number of decoding times of the inference request by the large model and the upper limit of the request length that can be inferred by the large model comprises: Determine a third ratio of a difference between the upper limit of the request length and the remaining number of decoding times to the upper limit of the request length; The first priority is determined according to the third ratio and a specified third coefficient.

5. The method according to claim 2, wherein the first parameter and the second parameter are related to a time tolerance threshold, the time tolerance threshold is adjustable, and the first parameter and the second parameter are adjusted by adjusting the time tolerance threshold; in, In the case where the first parameter and the second parameter are adjusted by adjusting the time tolerance threshold, the priorities of the multiple inference requests satisfy at least one of the following: When the waiting time is the same, the higher the user level corresponding to the inference request, the higher the priority; In the case where the user levels are the same, if the waiting time of the inference request exceeds the time tolerance threshold, the priority is higher than the priority of the newly arrived inference request.

6. The method according to claim 1, wherein the request queue comprises a first queue, a second queue and a third queue, wherein the first queue is used to store inference requests that have not been pre-filled, the second queue is used to store inference requests that have completed the pre-filling stage and are in the decoding stage, and the third queue is used to store inference requests whose resources are preempted in the decoding stage; The selecting at least one reasoning request from the multiple reasoning requests to perform model reasoning in a reasoning phase according to the priorities of the multiple reasoning requests includes: Determining whether the third queue is empty; When the third queue is empty, selecting at least one inference request from the first queue according to the priority of the inference request in the first queue to perform a model inference in the pre-filling stage; When the third queue is not empty, at least one inference request is selected from the second queue according to the priority of the inference requests in the second queue to perform a model inference in the decoding stage.

7. The method of claim 6, wherein selecting at least one inference request from the first queue to perform a model inference in a pre-filling phase according to the priority of the inference request in the first queue comprises: Prioritizing the inference requests in the first queue according to the priorities of the inference requests in the first queue; Starting from the reasoning request with the highest priority, determine in turn whether the reasoning request can be reasoned in this round; A model inference in the pre-filling phase is performed for at least one inference request that can be inferred in this round.

8. The method according to claim 7, wherein starting from the inference request with the highest priority, determining in turn whether the inference request can be inferred in the current round comprises: Determining whether the inference request satisfies a first condition; When the inference request satisfies the first condition, determining that the inference request can be inferred in this round; The first condition includes at least one of the following: The number of requests that have been accommodated in this round is less than the maximum number of requests that can be accommodated in this round; The number of tokens already accommodated in this round is less than the maximum number of tokens that can be accommodated in this round; There are sufficient inference resources to infer the inference request.

9. The method of claim 7, further comprising: When the number of inference requests that can be inferred in this round is 0, at least one inference request is selected from the second queue according to the priority of the inference requests in the second queue to perform a model inference in the decoding stage.

10. The method according to any one of claims 6 to 9, before selecting at least one inference request from the first queue to perform a model inference in a pre-filling phase according to the priority of the inference request in the first queue, the method further comprises: Determine whether the first queue satisfies a second condition, where the second condition includes that the first queue is not empty and / or a scheduling interval threshold of the first queue has been reached; The step of selecting at least one inference request from the first queue to perform a model inference in a pre-filling phase according to the priority of the inference request in the first queue includes: When the first queue satisfies the second condition, at least one inference request is selected from the first queue according to the priority of the inference requests in the first queue to perform a model inference in the pre-filling stage.

11. The method of claim 10, further comprising: When the first queue does not satisfy the second condition, at least one inference request is selected from the second queue according to the priority of the inference requests in the second queue to perform a model inference in the decoding stage.

12. The method according to any one of claims 6, 9 and 11, wherein selecting at least one inference request from the second queue to perform a model inference in a decoding phase according to the priority of the inference request in the second queue comprises: Prioritizing the inference requests in the second queue according to the priorities of the inference requests in the second queue; Starting from the inference request with the highest priority, determining in turn whether the inference resource can infer the inference request; When the reasoning resource cannot reason the reasoning request, resource preemption is performed starting from the reasoning request with the lowest priority in the second queue until the reasoning resource can reason the reasoning request; A model inference at a decoding stage is performed on at least one inference request that can be inferred by the inference resource.

13. The method according to claim 12, further comprising, for any inference request preempted by an execution resource, any of the following: storing the inference request in the third queue; Unloading the KV cache occupied by the inference request from the GPU to the CPU; storing the inference request in the first queue; Release the KV cache occupied by the inference request; The word-grams generated by the inference request are saved in the CPU.

14. The method of claim 12, further comprising at least one of the following: In a case where resource preemption does not occur for at least one inference request selected from the second queue, selecting at least one inference request from the third queue to perform model inference in a decoding phase according to the priority of the inference requests in the third queue; When the number of inference requests that can be inferred by the inference resource is 0, at least one inference request is selected from the third queue according to the priority of the inference requests in the third queue to perform a model inference in the decoding stage.

15. The method according to claim 12, before selecting at least one inference request from the second queue to perform a model inference in a decoding phase according to the priority of the inference request in the second queue, the method further comprises: Determine whether the second queue satisfies a third condition, where the third condition includes that the second queue is not empty; The selecting at least one inference request from the second queue to perform a model inference in a decoding phase according to the priority of the inference request in the second queue includes: When the second queue satisfies the third condition, at least one inference request is selected from the second queue according to the priority of the inference requests in the second queue to perform a model inference in the decoding stage.

16. The method of claim 15, further comprising: When the second queue does not satisfy the third condition, at least one inference request is selected from the third queue according to the priority of the inference requests in the third queue to perform a model inference in the decoding stage.

17. The method according to claim 14 or 16, wherein selecting at least one inference request from the third queue to perform a model inference in a decoding phase according to the priority of the inference request in the third queue comprises: Prioritizing the inference requests in the third queue according to the priorities of the inference requests in the third queue; Starting from the inference request with the highest priority, determining in turn whether the inference resource can infer the inference request; If the inference resource is capable of inferring the inference request, loading the KV cache corresponding to the inference request from the CPU to the GPU; A model inference at a decoding stage is performed on at least one inference request that can be inferred by the inference resource.

18. The method according to claim 1, after selecting at least one inference request from the multiple inference requests to perform a model inference in an inference phase, the method further comprises: Determining whether the at least one inference request has completed model inference; For any inference request, do the following: When the inference request has completed model inference, releasing the KV cache corresponding to the inference request; In the case that the inference request has not completed the model inference, the inference request is put into the request queue to perform model inference in the next inference phase.

19. An electronic device comprising: processor; a memory for storing instructions executable by the processor; The processor is configured to execute the instructions to implement the method according to any one of claims 1 to 18.

20. A computer-readable storage medium, when instructions in the storage medium are executed by a processor of an electronic device, the electronic device is enabled to execute the method according to any one of claims 1 to 18.

21. A computer program product, comprising a non-transitory computer-readable storage medium storing a computer program, wherein the computer program is operable to cause a computer to execute part or all of the steps in the method according to any one of claims 1 to 18.

Images