Model reasoning request management method, device and system

Abstract

The invention discloses a request management method, device and system for model reasoning. The method comprises the following steps: obtaining an observation index of a model reasoning service; index prediction is carried out according to the observation indexes, prediction indexes are obtained, and the prediction indexes comprise the target memory utilization rate of the central processing unit and the target video memory utilization rate of the graphics processing unit; determining a scheduling strategy according to the prediction index, determining a scheduling suggestion according to the scheduling strategy in combination with the request queue, and adding the scheduling suggestion into a suggestion buffer queue; when the model reasoning service executes request scheduling, the scheduling suggestion is read from the suggestion buffer queue, a target scheduling decision is determined according to the scheduling suggestion in combination with a scheduling decision of the model reasoning service, and the model reasoning service is controlled to schedule corresponding request execution model reasoning according to the target scheduling decision; the problem of unreasonable request scheduling in the model reasoning process is solved, and efficient utilization and reasonable allocation of resources are realized.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to a request management method and device for model reasoning, and a request management system. Background Art

[0002] In the field of large model inference services, with the advancement of technology and the growth of market demand, users are using large model inference services more and more frequently, and the load pressure on the enterprise's inference cluster is getting higher and higher. At the same time, since the identities of users using large model inference services are different, and the tasks that initiate large model inference requests are also diverse, this leads to different large model inference requests receiving different service priorities and urgency. When hardware resources are limited, high-priority and high-urgency requests should be served before low-priority requests.

[0003] In order to meet the diverse needs of users, the reasoning service framework not only needs to process large-scale data efficiently, but also needs to be able to reasonably allocate resources in a multi-user environment. However, although the current mainstream large-model reasoning framework has excellent performance in processing large-scale data and complex computing tasks, it has many shortcomings in user priority differentiation and resource scheduling. In the field of cloud computing, the resource quotas allocated to high-priority requests and low-priority requests can be differentiated by dividing the request resource pool, and different resource pools with different resource amounts can be specified for high-priority requests and low-priority requests, respectively, so as to provide differentiated services for requests of different priorities and urgency.

[0004] However, the service requirements of requests with different priorities and urgency levels are relatively rigid in request processing by dividing resource pools. When a request is routed to the corresponding resource pool, the request can only be served by the service instance in the resource pool. When another resource pool has vacancies, the request cannot be reallocated to the vacancies in the resource pool. Even if the request can be rescheduled, the KVCache of the request prompt needs to be recalculated, resulting in a waste of computing power. Therefore, the resource pool-based method of providing services for requests of different priorities will inevitably cause a waste of computing power, which is particularly evident when overall hardware resources are insufficient. Therefore, how to reasonably schedule requests during model inference has become a problem to be solved. Summary of the invention

[0005] The present invention provides a model reasoning request management method, device and request management system to solve the problem of unreasonable request scheduling in the model reasoning process.

[0006] According to one aspect of the present invention, a model reasoning request management method is provided, which is applied to a request processing engine in a request management system, wherein the request management system also includes a model reasoning service, and the method includes:

[0007] Obtaining observation indicators of the model inference service;

[0008] Performing indicator prediction based on the observed indicators to obtain predicted indicators, wherein the predicted indicators include a target memory utilization rate of a central processing unit and a target video memory utilization rate of a graphics processing unit;

[0009] Determine a scheduling strategy according to the prediction indicator, determine a scheduling suggestion according to the scheduling strategy combined with the request queue, and add the scheduling suggestion to the suggestion buffer queue;

[0010] When the model reasoning service executes request scheduling, the scheduling suggestion is read from the suggestion buffer queue, and the target scheduling decision is determined based on the scheduling suggestion and the scheduling decision of the model reasoning service, and the model reasoning service is controlled to schedule the corresponding request to execute model reasoning according to the target scheduling decision.

[0011] According to another aspect of the present invention, a model reasoning request management device is provided, which is applied to a request processing engine in a request management system, wherein the request management system also includes a model reasoning service, and the device includes:

[0012] An observation module, used to obtain observation indicators of the model reasoning service;

[0013] A state prediction module, used for performing indicator prediction according to the observed indicators to obtain predicted indicators, wherein the predicted indicators include a target memory utilization rate of a central processing unit and a target video memory utilization rate of a graphics processing unit;

[0014] A request scheduler, configured to determine a scheduling strategy according to the prediction indicator, determine a scheduling suggestion according to the scheduling strategy in combination with the request queue, and add the scheduling suggestion to a suggestion buffer queue;

[0015] The side-loaded scheduling module is used to read scheduling suggestions from the suggestion buffer queue when the model reasoning service executes request scheduling, and determine the target scheduling decision based on the scheduling suggestions and the scheduling decision of the model reasoning service, and control the model reasoning service to schedule the corresponding request to execute model reasoning according to the target scheduling decision.

[0016] According to another aspect of the present invention, there is provided a request management system, comprising: a request processing engine and a model reasoning service;

[0017] The request processing engine is used to: obtain observed indicators of the model reasoning service; perform indicator prediction based on the observed indicators to obtain predicted indicators, wherein the predicted indicators include a target memory utilization of a central processing unit and a target video memory utilization of a graphics processing unit; determine a scheduling strategy based on the predicted indicators, determine a scheduling suggestion based on the scheduling strategy in combination with a request queue, and add the scheduling suggestion to a suggestion buffer queue; when the model reasoning service executes request scheduling, read the scheduling suggestion from the suggestion buffer queue, and determine a target scheduling decision based on the scheduling suggestion in combination with the scheduling decision of the model reasoning service, and control the model reasoning service to schedule the corresponding request to execute model reasoning based on the target scheduling decision.

[0018] According to another aspect of the present invention, an electronic device is provided, the electronic device comprising:

[0019] at least one processor, and a memory communicatively coupled to the at least one processor;

[0020] The memory stores a computer program that can be executed by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute the request management method for model reasoning described in any embodiment of the present invention.

[0021] According to another aspect of the present invention, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the model reasoning request management method described in any embodiment of the present invention when executed.

[0022] According to another aspect of the present invention, a computer program product is provided. The computer program product includes a computer program. When the computer program is executed by a processor, the method for managing model reasoning requests according to any embodiment of the present invention is implemented.

[0023] The technical solution of the embodiment of the present invention obtains the observed indicators of the model reasoning service; predicts the indicators based on the observed indicators to obtain the predicted indicators, and the predicted indicators include the target memory utilization of the central processing unit and the target video memory utilization of the graphics processing unit; determines the scheduling strategy based on the predicted indicators, determines the scheduling suggestion based on the scheduling strategy combined with the request queue, and adds the scheduling suggestion to the suggestion buffer queue; when the model reasoning service executes the request scheduling, reads the scheduling suggestion from the suggestion buffer queue, and determines the target scheduling decision based on the scheduling suggestion combined with the scheduling decision of the model reasoning service, and controls the model reasoning service to schedule the corresponding request to execute the model reasoning according to the target scheduling decision, thereby solving the problem of unreasonable request scheduling in the model reasoning process, and predicts the predicted indicators based on the observed indicators of the model reasoning service to obtain the predicted indicators, and the predicted indicators include the target memory utilization of the central processing unit and the target video memory utilization of the graphics processing unit; determines the scheduling suggestion based on the scheduling strategy combined with the request queue, and adds the scheduling suggestion to the suggestion buffer queue; when the model reasoning service executes the request scheduling, reads the scheduling suggestion from the suggestion buffer queue, and determines the target scheduling decision based on the scheduling suggestion combined with the scheduling decision of the model reasoning service, and controls the model reasoning service to schedule the corresponding request to execute the model reasoning according to the target scheduling decision, thereby solving the problem of unreasonable request scheduling in the model reasoning process, and predicts the predicted indicators based on the observed indicators of the model reasoning service, and obtains the predicted indicators, and the predicted indicators include the target memory utilization of the central processing unit and the target video memory utilization of the graphics processing unit; determines the scheduling suggestion based on the scheduling suggestion and the target scheduling decision, and adds the scheduling suggestion to the suggestion buffer queue ... based on the target scheduling decision, and adds the scheduling suggestion based on the target The target memory utilization and the target video memory utilization of the graphics processor, the prediction indicators in the embodiment of the present application can be used to represent different indicators of the model reasoning service in the next time, that is, to predict the indicators of the model reasoning service in the subsequent time; determine the scheduling strategy through the prediction indicators, and further determine the scheduling suggestions in combination with the request queue, and add them to the suggestion buffer queue; when the model reasoning service executes request scheduling, read the scheduling suggestions from the suggestion buffer queue, and determine the target scheduling decision based on the scheduling suggestions and the scheduling decision of the model reasoning service, intervene in the request scheduling of the model reasoning service, and finally the target scheduling decision controls the model reasoning service to schedule the corresponding request to execute model reasoning; interfere with the scheduling decision of the model reasoning service through indicator prediction, provide the model reasoning service with the target scheduling decision, and control the model reasoning service to execute model reasoning through reasonable request scheduling, so as to realize efficient utilization and reasonable allocation of resources.

[0024] It should be understood that the contents described in this section are not intended to identify the key or important features of the embodiments of the present invention, nor are they intended to limit the scope of the present invention. Other features of the present invention will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0026] Figure 1 is a flowchart of a model reasoning request management method provided according to the first embodiment of the present invention;

[0027] Figure 2 is a flowchart of a model reasoning request management method provided according to Embodiment 2 of the present invention;

[0028] Figure 3 is an example diagram of implementing a request scheduling according to Embodiment 2 of the present invention;

[0029] Figure 4 is a schematic diagram of the structure of a model reasoning request management device provided in Embodiment 3 of the present invention;

[0030] Figure 5 is a schematic diagram of the structure of a request management system provided according to Embodiment 4 of the present invention;

[0031] Figure 6 is an example diagram of implementing request management according to Embodiment 4 of the present invention;

[0032] Figure 7 It is a structural diagram of an electronic device for implementing the request management method for model reasoning according to an embodiment of the present invention. DETAILED DESCRIPTION

[0033] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0034] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0035] Embodiment 1

[0036] Figure 1 This is a flowchart of a model reasoning request management method provided in the first embodiment of the present invention. This embodiment is applicable to the case of managing requests executed by a model reasoning service. The method is applied to a request processing engine in a request management system, and the request management system also includes a model reasoning service. Figure 1 As shown, the method includes:

[0037] S101. Obtain observation indicators of model inference service.

[0038] In this embodiment, the model inference service can be an inference service for any type of model, such as an inference service for a large language model. The observation indicator can be understood as an indicator used to describe relevant information in the model inference service process. The observation indicator can be the memory / video memory utilization of the CPU / GPU, the number of related requests in the model inference process, the key-value pair cache size of the token obtained by inference, and so on.

[0039] When the model inference service receives a user request related to inference, it performs the corresponding model inference. The CPU central processing unit and GPU graphics processing unit are used in the model inference process. For example, when the GPU performs the model inference task, the GPU's video memory utilization, core utilization and other indicators will change during the inference process. During the model inference process, some data will be stored in the CPU memory, and the CPU memory utilization will change. At the same time, as the model inference progresses, some parameters involved in the model inference itself will also change, such as the number of requests left for prefilling and the request length.

[0040] The observation indicators of the model reasoning service can be obtained through the interface. The request processing engine obtains the observation indicators of the model reasoning service by calling the corresponding interface. For example, the model reasoning service can be exposed to the request processing engine in the form of an API. The observation indicators can be obtained according to a certain period or after a certain trigger condition is met. For example, the trigger condition is that the model reasoning service needs to schedule a new request, that is, after the model reasoning service completes the scheduling of a request, it schedules other requests waiting to be processed.

[0041] S102: performing indicator prediction according to the observed indicators to obtain predicted indicators, where the predicted indicators include a target memory utilization rate of the central processing unit and a target video memory utilization rate of the graphics processing unit.

[0042] In this embodiment, the prediction index can be understood as the index information of the model inference service obtained through prediction, and the prediction index is the index information of a period of time later than the collection time of the observation index; for example, the collection time of the observation index is t1, which represents the observation index of the model inference service at time t1, and the prediction index is the observation index of the model inference service at the predicted time (t1+10s). The target memory utilization can be understood as the predicted memory utilization of the central processing unit; the target video memory utilization can be understood as the predicted video memory utilization of the graphics processing unit.

[0043] Analyze observation indicators. Observation indicators can reflect information about model inference services from different perspectives. By analyzing observation indicators, the future load status of model inference services can be predicted to obtain prediction indicators, including the target memory utilization of the CPU and the target video memory utilization of the GPU. Models and functions can be pre-built to predict observation indicators.

[0044] S103: Determine a scheduling strategy according to the prediction index, determine a scheduling suggestion according to the scheduling strategy and the request queue, and add the scheduling suggestion to the suggestion buffer queue.

[0045] In this embodiment, the scheduling policy can be understood as a policy indication for assisting the model inference service in scheduling, such as pausing a request, resuming a request, etc. The request queue can be understood as a queue formed by requests from different users, and the request queue can be a first-in-first-out FIFO queue or other types of queues. The suggestion buffer queue can be understood as a queue formed by scheduling suggestions, and can be a first-in-first-out queue or other types of queues.

[0046] Analyze the processing capabilities of the CPU and GPU based on the prediction indicators, and determine the scheduling strategy based on the processing capabilities of the CPU and GPU; for example, when the GPU processing capability is low, suspend some requests to ensure the request processing speed, and when the GPU processing capability is high, resume some requests to ensure fast request processing. Select appropriate requests from the request queue based on the scheduling strategy to generate scheduling suggestions, for example, the scheduling suggestion is to suggest request 1 to be suspended, request 2 to be resumed, and so on; add the scheduling suggestions to the suggestion buffer queue.

[0047] Since the request processing engine performs request processing and scheduling decisions asynchronously with the internal operation of the model inference service, when scheduling decisions are made based on the collected indicators, these indicator information has lagged behind the state of the model inference service itself when the decision was made. Therefore, when making scheduling decisions, scheduling decisions should be made based on the predicted future state of the model inference service.

[0048] S104. When the model reasoning service executes request scheduling, the scheduling suggestion is read from the suggestion buffer queue, and the target scheduling decision is determined based on the scheduling suggestion and the scheduling decision of the model reasoning service. The model reasoning service is controlled to schedule the corresponding request to execute model reasoning based on the target scheduling decision.

[0049] In this embodiment, the target scheduling decision can be understood as the strategy ultimately adopted by the model reasoning service when scheduling requests.

[0050] The model reasoning service can execute requests sent by multiple users at the same time. For example, the model reasoning service provides model reasoning related services for multiple users. After receiving the request sent by the user, the model reasoning service can directly execute the request for model reasoning if the execution conditions are met. If the execution conditions are not met, the request can be placed in a waiting queue and executed after the execution conditions are met. The execution conditions can be that the number of requests being executed by the model reasoning service is less than a certain threshold. After executing a request, the model reasoning service continues to obtain requests from the waiting queue and executes them. When the model reasoning service executes request scheduling, that is, when the model reasoning service obtains a new request and executes it, the scheduling suggestion is read from the suggestion buffer queue. The scheduling suggestion and the scheduling decision of the model reasoning service are integrated to determine the target scheduling decision. For example, the request recommended to be suspended or resumed in the scheduling suggestion is determined, and it is judged whether the scheduling decision of the model reasoning service uses the same processing method to schedule this request. If so, it is determined that this request is suspended or resumed; if not, it can be determined according to the pre-set priority and other information to determine what kind of processing to be performed on this request. According to the target scheduling decision, the model reasoning service is controlled to schedule the corresponding request to perform model reasoning.

[0051] After reading the scheduling suggestion from the suggestion buffer queue, the scheduling suggestion can be deleted from the suggestion buffer queue. The embodiment of the present application can also set the trigger condition that the scheduling suggestion in the suggestion buffer queue is 0, that is, after the scheduling suggestion in the suggestion buffer queue is read, there is no scheduling suggestion at this time, determine that the trigger condition is met, and obtain the observation indicator to generate a new scheduling suggestion.

[0052] A request management method for model reasoning in an embodiment of the present invention solves the problem of unreasonable request scheduling in the model reasoning process. Prediction is performed through the observation indicators of the model reasoning service to obtain prediction indicators. The prediction indicators include the target memory utilization of the central processing unit and the target video memory utilization of the graphics processing unit. The prediction indicators in the embodiment of the present application can be used to represent different indicators of the model reasoning service in the next time, that is, to predict the indicators of the model reasoning service in the subsequent time; determine the scheduling strategy through the prediction indicators, further determine the scheduling suggestions in combination with the request queue, and add them to the suggestion buffer queue; when the model reasoning service executes request scheduling, read the scheduling suggestions from the suggestion buffer queue, and determine the target scheduling decision based on the scheduling suggestions and the scheduling decision of the model reasoning service, intervene in the request scheduling of the model reasoning service, and finally the target scheduling decision controls the model reasoning service to schedule the corresponding request to execute model reasoning; interfere with the scheduling decision of the model reasoning service through indicator prediction, provide the model reasoning service with a target scheduling decision, and control the model reasoning service to execute model reasoning through reasonable request scheduling, so as to achieve efficient utilization and reasonable allocation of resources.

[0053] Embodiment 2

[0054] Figure 2 This is a flowchart of a model reasoning request management method provided in the second embodiment of the present invention. This embodiment is refined on the basis of the above embodiment. Figure 2 As shown, the method includes:

[0055] S201. Obtain observation indicators of model inference service.

[0056] The observation indicator includes at least one of a processor parameter indicator and a request indicator.

[0057] In this embodiment, the processor parameter indicator can be understood as a parameter of a processor, which can be a parameter of the same type for different types of processors, or different types of parameters for the same processor. The request indicator can be understood as relevant information of a request.

[0058] The processor parameter indicator includes at least one of the following:

[0059] CPU memory utilization;

[0060] The graphics processor's video memory utilization;

[0061] GPU core utilization;

[0062] The request indicator includes at least one of the following:

[0063] The number of requests left for prefilling;

[0064] The length of the request left for prefilling;

[0065] The number of requests in the decoding phase;

[0066] The size of the requested key-value cache to be swapped into the CPU's memory;

[0067] The requested key-value cache size to be swapped into the GPU's video memory.

[0068] Among them, CPU memory usage and GPU video memory usage are used to measure the usage of memory and video memory by the current model inference service, so as to judge whether the current hardware storage resources are sufficient;

[0069] GPU core usage rate is used to measure the usage of GPU cores by the current model inference service, so as to judge whether the computing resources of the current hardware are sufficient;

[0070] The number of requests left for prefilling, used to predict the future load status of the model inference service;

[0071] The request length left for prefilling is used to predict the future load status of the model inference service;

[0072] The number of requests in the decoding phase, which is used to predict the future load status of the model inference service;

[0073] The KVCache size of the request to be swapped into the CPU memory, which is used to predict the future load status of the model inference service;

[0074] The KVCache size of the request to be swapped into the GPU memory, which is used to predict the future load status of the model inference service.

[0075] The core of the model based on the Transformer architecture is the Attention mechanism. Reasoning is divided into two stages: prefill (input understanding and initialization) and decoding (recursive reasoning and decoding output). The reasoning process generated after each request is initiated will first go through a Prefill process. The prefill process will calculate all the user's inputs and generate the corresponding KV cache, and then go through several decoding processes. In each decoding process, the server will generate a character and put it into the KV cache. The inferred prediction result is put into the input again, and this cycle repeats until the final result is inferred. After the prefill is completed, the new request will be continuously iterated for decoding. After each decoding stage, the result will be returned to the user on the spot. This generation process is called streaming.

[0076] In the decoding phase, the attention of the current token and all previously generated tokens needs to be calculated, so the k and v vectors of all tokens need to be calculated. However, the KV values ​​of the previous tokens are repeatedly calculated in each round of decoding, so they can be saved as two [seq_len-1, inner_dim] Tensors. In each round of calculation, only the kv value of the current token needs to be calculated.

[0077] The number of requests waiting for Prefilling refers to the number of requests that need to be processed in the Prefilling stage; the length of requests waiting for Prefilling refers to the length of requests that need to be processed in the Prefilling stage. The length can be the number of tokens obtained after tokenization of the input sequence corresponding to all user inputs calculated in the prefill process. The number of requests in the Decoding stage refers to the number of requests in the Decoding stage.

[0078] The requested key-value cache size to be swapped to the memory of the central processing unit, i.e., the KVCaChe size, is the size of the memory actually used. The requested key-value cache size to be swapped to the video memory of the graphics processor, i.e., the KVCaChe size, is the size of the video memory actually used. During the model inference process, CPU memory and GPU video memory are applied in advance to store the key-value pairs generated during the inference process. The KVCaChe size in the embodiment of the present application is the size of the video memory actually used.

[0079] Optionally, the processor parameter indicator is updated when the first indicator update condition is met;

[0080] The first indicator update condition includes at least one of the following:

[0081] The system time meets the periodic update conditions;

[0082] The model inference service completes the scheduling of a request.

[0083] In the present embodiment, the first indicator update condition can be understood as a condition for updating the processor parameter indicator. The periodic update condition can be understood as a condition for periodically updating the processor parameter indicator. For example, the update period is set to T. When the time difference between the system time and the time of the last update is equal to T (or not less than T), the periodic update condition is met and the processor parameter indicator needs to be periodically updated. The model reasoning service completes the scheduling of a request. At this time, the processor parameter indicator will also change, and the processor parameter indicator needs to be updated. The processor parameter indicator in the embodiment of the present application can be updated when any of the above conditions are met.

[0084] Optionally, the request indicator is updated when a second indicator update condition is met.

[0085] Among them, the second indicator update condition includes: the model inference service completes the scheduling of a request.

[0086] In this embodiment, the second indicator update condition can be understood as a condition for updating the processor parameter indicator. When the model reasoning service completes the scheduling of a request, the request indicator will change and the request indicator needs to be updated. The request indicator in the embodiment of the present application can be updated after the model reasoning service completes the scheduling of a request.

[0087] The embodiment of the present application does not need to collect various indicators in the model reasoning service in a synchronous process when collecting observation indicators, but collects indicators in an asynchronous manner in a certain time period and puts them into the internal cache, and updates some indicator information according to the request scheduling decision inside each model reasoning. When the request processing engine accesses the indicator information through the API, the indicator information is directly taken out of the cache and returned. Compared with the synchronous collection method, the cache strategy can effectively guarantee the performance of the indicator collection interface and the overall performance of the reasoning service framework.

[0088] S202: Taking the memory size of the central processing unit, the video memory size of the graphics processing unit, and the observed index as input for prediction, and determining the target memory utilization of the central processing unit and the target video memory utilization of the graphics processing unit.

[0089] Among them, the memory size of the central processing unit and the video memory size of the graphics processing unit are applied for when the program starts.

[0090] When the program starts, the memory size of the CPU and the video memory size of the GPU are requested. The memory size of the CPU, the video memory size of the GPU, and the observed indicators are input into the pre-trained model, function, etc. for prediction to obtain the target memory utilization of the CPU and the target video memory utilization of the GPU.

[0091] Exemplarily, the indicator prediction of the present application can be implemented by algorithms such as linear regression. For example, a linear regression algorithm is used for fitting, and the linear relationship between the memory utilization of the CPU and the video memory utilization of the GPU and the memory size of the CPU, the video memory size of the GPU, and the observed indicators is fitted to obtain a function expression. After determining the memory size of the CPU, the video memory size of the GPU, and the observed indicators, they are substituted into the function expression to obtain the target memory utilization and the target video memory utilization; wherein the target memory utilization and the target video memory utilization can be calculated by one function expression or by two function expressions. When predicting indicators, the memory utilization of the CPU, the video memory utilization of the GPU, the core utilization of the GPU, the number of requests to be prefilled, the length of the requests to be prefilled, the number of requests in the decoding stage, the key-value cache size of the requests to be exchanged to the memory of the CPU, and the key-value cache size of the requests to be exchanged to the video memory of the GPU can be used as input for prediction.

[0092] S203: If the target video memory utilization is higher than the first preset threshold and the target memory utilization is lower than the second preset threshold, determine that the scheduling strategy is a pause request.

[0093] In this embodiment, the first preset threshold and the second preset threshold can be understood as pre-set thresholds, which can be set according to the requirements of processor performance, speed, business type, etc. Compare the target video memory utilization with the first preset threshold, and the target memory utilization with the second preset threshold. The target video memory utilization is higher than the first preset threshold and the target memory utilization is lower than the second preset threshold, indicating that the GPU has limited memory for processing requests at this time, and the scheduling strategy is determined to suspend requests to ensure the speed and performance of request processing. When suspending requests, it can be recommended to suspend some requests with lower priority and urgency, so that the remaining requests can preempt the hardware resources used by these suspended requests, and the KVCache of the preempted requests will be swapped to the CPU KVCache storage space.

[0094] S204: If the target video memory utilization is lower than a third preset threshold, determine that the scheduling strategy is a recovery request.

[0095] In this embodiment, the third preset threshold value can be understood as a preset threshold value, which can be set according to the requirements of processor performance, speed, business type, etc. If the target video memory utilization rate is lower than the third preset threshold value, it means that the GPU has sufficient resources for processing requests at this time, and the scheduling strategy is determined to be a resume request to resume the suspended request. The resume request can give priority to resuming the preempted request with higher priority and urgency.

[0096] If both the target memory utilization of the GPU and the target video memory utilization of the CPU are higher than the corresponding thresholds, no recommendation is made because the exchange between the GPU KVCache and the CPU KVCache is no longer possible.

[0097] S205 . Determine a scheduling suggestion according to the scheduling policy and the request queue, and add the scheduling suggestion to the suggestion buffer queue.

[0098] Optionally, the request queue includes a running queue and a suspended queue. The scheduling suggestion is determined according to the scheduling policy and the request queue, including:

[0099] A1. When the scheduling strategy is to pause request, traverse the running queue, sort the requests in the running queue according to the priority level and urgency of the requests in the running queue, select at least one request as a pause request according to the sorting result, add the pause request to the pause request queue, and generate a scheduling suggestion according to the pause request queue.

[0100] In this embodiment, the running queue is the Running queue; the pause request can be understood as a request from a user who needs to pause; and the pause request queue can be understood as a queue for caching requests that need to be paused.

[0101] When the scheduling strategy is to pause request, the running queue is traversed to determine the priority level and urgency of each request in the running queue, and the requests in the running queue are sorted according to the priority level and urgency. For example, requests with higher priority levels and higher urgency are put in front, and at least one request with lower priority levels and lower urgency is screened out as a pause request based on the sorting result, and all the screened pause requests are added to the pause request queue, and a scheduling suggestion is generated according to the pause request queue, for example, the scheduling suggestion is to pause the pause requests in the pause request queue.

[0102] A2. When the scheduling strategy is to recover requests, traverse the suspended queue, sort the requests in the suspended queue according to their priority levels and urgency, select at least one request as a recovery request based on the sorting result, add the recovery request to the recovery request queue, and generate a scheduling suggestion based on the recovery request queue.

[0103] In this embodiment, the suspend queue is the Hanging queue; the resume request can be understood as a request from a user that needs to be suspended; and the resume request queue can be understood as a queue for caching requests that need to be resumed.

[0104] When the scheduling strategy is to recover requests, the suspended queue is traversed to determine the priority level and urgency of each request in the suspended queue, and the requests in the running queue are sorted according to the priority level and urgency. For example, requests with higher priority levels and higher urgency are put in front. At least one request with higher priority levels and higher urgency is screened out as a recovery request based on the sorting result, and all the screened recovery requests are added to the recovery request queue. A scheduling suggestion is generated based on the recovery request queue. For example, the scheduling suggestion is to recover the recovery requests in the recovery request queue.

[0105] Optionally, the forecasting indicators also include: the number of requests;

[0106] Correspondingly, the number of pause requests or resume requests is the number of requests.

[0107] In this embodiment, the number of requests is the number of requests that should be paused / resumed in this round of scheduling. The number of requests can be predicted together with the target memory utilization and the target video memory utilization when performing indicator prediction. When the prediction indicator also includes the number of requests, when screening the resume requests and the pause requests, a preset number of pause requests or resume requests are screened out.

[0108] In the model reasoning service, requests of different priorities and urgency have different priorities for the use of hardware. Requests with high priority and high urgency should be given priority in occupying hardware resources for service. Therefore, these tasks need to be scheduled. When hardware resources are insufficient or the system load is high, it is necessary to decide which tasks should preempt other tasks for priority. The embodiment of the present application screens out some requests with the lowest priority and urgency, adds them to the pause request queue (Pause list), and then issues a pause scheduling suggestion to the model reasoning service; or screens out some requests with the highest priority and urgency from the hanging queue, adds them to the resume request queue (Resume list), and then issues a resume scheduling suggestion to the model reasoning service.

[0109] S206. When the model inference service executes a request scheduling, a scheduling suggestion is read from the suggestion buffer queue.

[0110] S207: intercept and obtain the scheduling decision of the model reasoning service, and record it as the first scheduling decision.

[0111] In this embodiment, the first scheduling decision is a scheduling decision, for example, running request 1, pausing request 1, etc. The model reasoning service will schedule the next request after a request is completed, and the model reasoning service itself will determine the scheduling decision. The embodiment of the present application can intercept and obtain the scheduling decision of the model reasoning service through an interface or the like, and record the scheduling decision of the model reasoning service as the first scheduling decision.

[0112] S208. Determine the request to be scheduled and the scheduling type according to the scheduling suggestion.

[0113] In this embodiment, the request to be scheduled can be understood as a request that needs to be scheduled; the scheduling type can be pause, resume, etc., which is used to indicate the type of operation performed on the request in this round of scheduling. The scheduling suggestion is analyzed to determine the request to be scheduled and the scheduling type, wherein the number of the request to be scheduled can be one or more.

[0114] S209: If the scheduling type of the request to be scheduled is pause, and the scheduling mode of the request to be scheduled is determined to be swap out or discard according to the first scheduling decision, the scheduling decision of the request to be scheduled is set to swap out, and the request to be scheduled is added to the exchange queue.

[0115] In this embodiment, the scheduling mode is the scheduling mode performed on the request, such as swapping out, discarding, swapping back, etc.; the swap queue is the Swapped queue. If the scheduling type of the request to be scheduled is pause, it means that the current round of scheduling needs to pause the request to be scheduled, and the first scheduling decision is analyzed. The first scheduling decision includes which request to schedule and the scheduling mode. If the first scheduling decision determines that the scheduling mode of the request to be scheduled is swapping out or discarding, the scheduling decision of the request to be scheduled is set to swapping out, and the request to be scheduled is added to the swap queue to keep it "transparent" to the model reasoning service, and the model reasoning service will no longer generate scheduling decisions related to the request in the future.

[0116] S210: If the scheduling type of the request to be scheduled is recovery, set the scheduling decision of the request to be scheduled to recovery.

[0117] If the scheduling type of the pending request is recovery, since the pending request is in the Swaped queue of the request processing engine, the model inference service will not generate any relevant scheduling decisions. Therefore, the scheduling decision of the pending request can be directly set to recovery.

[0118] S211. Generate a target scheduling decision based on the scheduling decision of the request to be scheduled, and control the model reasoning service to schedule the corresponding request to execute model reasoning according to the target scheduling decision.

[0119] When there are multiple requests to be scheduled, steps S209-S210 are executed for each request to be scheduled to set its scheduling decision; a final target scheduling decision is generated based on the scheduling decisions of all requests to be scheduled to implement scheduling of the requests.

[0120] The existing solutions for model reasoning services have the following shortcomings: 1. Simple task queues and basic scheduling algorithms: Some frameworks use simple task queues and scheduling algorithms to manage user requests. These algorithms usually use the "FIFO" principle and ignore the differences in user priority and request urgency. When the number of users increases or the load is high, this approach may result in high-priority users not being able to get a timely response, affecting user experience and service satisfaction. 2. Static resource allocation strategy: Some model reasoning services use a pre-defined static resource allocation strategy to allocate computing resources and memory according to fixed rules. This approach is simple and direct, but lacks flexibility and is difficult to cope with dynamic changes in user requests. 3. Strategies based on resource optimization and distributed processing: Some services focus more on the optimization of computing resources and distributed processing capabilities, aiming to improve the overall throughput of the system and model reasoning efficiency. However, these optimizations are mainly focused on hardware resources and computing tasks, rather than the priority of user requests. Therefore, in applications for multi-user scenarios, these frameworks find it difficult to effectively distinguish and differentiate services for users of different priorities.

[0121] The embodiment of the present application generates scheduling suggestions based on the system's observation indicators, request priority, urgency, etc., intervenes in the scheduling strategy of the model reasoning service itself through the scheduling suggestions, and generates a target scheduling strategy to control the model reasoning service to execute the corresponding scheduling request. While taking into account the efficiency of hardware resource utilization, high-priority and high-urgency requests can be served before low-priority and high-urgency requests.

[0122] Optionally, before determining the target scheduling decision according to the scheduling suggestion combined with the scheduling decision of the model reasoning service, the method further includes:

[0123] B1. If the type corresponding to the scheduling suggestion is a pause request, determine the first candidate request to be paused according to the scheduling suggestion, take the first candidate request out of the running requests, update the status of the first candidate request from running to paused, and put the first candidate request into the exchange queue.

[0124] In this embodiment, the first candidate request can be understood as a request that needs to be scheduled. If the type corresponding to the scheduling suggestion is a pause request, that is, the scheduling suggestion indicates that the request should be paused, the scheduling suggestion is analyzed to determine the requests that need to be paused, and this part of the requests is used as the first candidate request, and the first candidate request is taken out from the running requests, and the status of the first candidate request is recorded, and the status of the first candidate request is updated from running to paused, and the first candidate request is put into the exchange queue.

[0125] B2. If the type corresponding to the scheduling suggestion is a recovery request, determine the second candidate request for recovery according to the scheduling suggestion, take the second candidate request out of the exchange queue, update the status of the second candidate request from pause to run, and put the second candidate request into the running request.

[0126] In this embodiment, the second candidate request can be understood as a request that needs to be scheduled. If the type corresponding to the scheduling suggestion is a recovery request, that is, the scheduling suggestion indicates that the request should be recovered, the scheduling suggestion is analyzed to determine the requests that need to be recovered, and this part of the requests is used as the second candidate request, and the second candidate request is taken out from the running requests, and the status of the second candidate request is recorded, and the status of the second candidate request is updated from paused to running, and the second candidate request is put into the running queue.

[0127] The embodiment of the present application slices the scheduling methods or functions in the model reasoning service to influence the internal scheduling decisions of the model reasoning service in a low-intrusive manner, thereby avoiding direct modification of the code of the scheduling methods or functions. When the scheduling suggestion of the external request processing engine arrives, it will first enter the suggestion buffer queue (Suggestions) queue. Each time a request scheduling method or function inside the model reasoning service is executed, the request processing engine will take out the scheduling suggestion from the suggestion buffer queue (Suggestions), and generate a scheduling decision to take the request out of the running request or resume the operation of some requests based on the scheduling suggestion:

[0128] For the scheduling suggestion of pausing the specified request (i.e., the type corresponding to the scheduling suggestion is a pause request), take it out of the running requests, record the current status of the request and put it into the swap queue (Swapped) (i.e., take the first candidate request out of the running requests, update the status of the first candidate request from running to paused, and put the first candidate request into the swap queue);

[0129] For the scheduling suggestion of Resume specified request (i.e., the type of scheduling suggestion corresponding to the resume request), take it out of the Swapped queue, restore its status and put it back into the running request (take the second candidate request out of the swap queue, update the status of the second candidate request from pause to run, and put the second candidate request into the running request);

[0130] Finally, these decisions are integrated with the scheduling decisions generated by the scheduling methods or functions within the model reasoning service to obtain the final target scheduling decision, and the update of the corresponding indicators is indicated based on the target scheduling decision.

[0131] For example, Figure 3A sample diagram for implementing request scheduling is provided. Taking the model inference service vLLM as an example, the internal scheduling module of vLLM (hereinafter referred to as Inner Scheduler) sets up three queues: Running, Swapped, and Waiting. The Waiting queue records requests waiting to be executed, and all newly arrived requests are first added to the queue for queuing. The Running queue records the requests currently being processed, and the Swapped queue records the requests that have been suspended after the internal scheduling decision. Each time vLLM executes the internal scheduling method, it decides which requests need to be processed (taken out of the Waiting queue and put into the Running queue, i.e., compute), which requests need to be discarded (taken out of the Running queue and put back into Waiting, i.e., drop out), which requests need to be swapped out (taken out of the Running queue and put into Swapped, i.e., swap out in the dotted part of the figure), and which requests need to be swapped back (from Swapped to Running, i.e., swap in in the dotted part of the figure). Taking the request processing engine including the sidecar scheduling module Sidecar Scheduler as an example, Sidecar Scheduler adds an additional Swapped queue internally to record requests that are suspended due to external scheduling decisions (outer decision). By slicing the internal scheduling method of vLLM, each time the internal scheduling method of vLLM is executed and returns a scheduling decision, Sidecar Scheduler intercepts these decisions and decides to modify these scheduling decisions based on the external scheduling decisions it receives (placed in the suggestion buffer Suggestions queue). The specific strategies are as follows:

[0132] For requests that need to be paused in Suggestions, if the Inner Scheduler makes a decision to swap out or drop out the request, it is modified to a swap-out scheduling decision and the request is not placed in the Inner Scheduler's Swapped queue, but in the Sidecar Scheduler's Swapped queue, making it "transparent" to the Inner Scheduler. The Inner Scheduler will not make scheduling decisions related to the request later.

[0133] For the request that needs to be restored in Suggestions, since the request is in the Swapped queue of the Sidar Scheduler, the Inner Scheduler will not make any relevant scheduling decisions. Therefore, the release scheduling decision of the request can be directly placed in the final target scheduling decision, or directly placed in the Running queue of the Inner Scheduler. Figure 3 Take putting it directly into the Running queue of the Inner Scheduler as an example.

[0134] For requests not included in Suggestions, no processing is done and they are retained as is.

[0135] Through the above method, the decision-making of the internal scheduling method of the Sidecar Scheduler and the model reasoning service can be integrated.

[0136] As an optional embodiment of the present embodiment, this optional embodiment is further optimized to include: when a new request is received, traversing the running queue to determine the user request currently being executed; determining the execution time corresponding to each user request; if the execution time is greater than the service time limit of the new request, rejecting the new request; otherwise, accepting the new request and sending the new request to the model inference service.

[0137] The request processing engine can receive requests sent by different users. The user's request usually needs to be sent to the model inference service so that the model inference service can provide the corresponding service to the user; for example, the user sends a request to the request processing engine, and the request processing engine forwards the request to the model inference service, or the user sends a request to the model inference service, and the request processing engine intercepts or obtains the request sent by the user through an interface. After receiving a new request, the request processing engine traverses the running queue and determines the requests in the running queue. These requests are the user requests currently being executed; based on each user request, the request is analyzed, for example, according to the type of request, time limit requirements, etc., to determine the execution time required for normal execution of each user request. The service time limit of the new request is determined based on the relevant information of the new request. If the execution time is greater than the service time limit of the new request, it means that the service time limit requirements of the new request may not be met, and the new request is rejected; otherwise, the new request is accepted and sent to the model inference service.

[0138] The embodiment of the present application can also determine whether the system load is within an acceptable range by analyzing the currently executing user request. If the load is too high, the new request may not meet the service time limit requirements, so the request will be directly rejected. If the load is within an acceptable range, the request is accepted and sent to the model reasoning service through the API of the model reasoning service for specific processing. The execution time can be used as a parameter to determine the load.

[0139] Optionally, after accepting the new request, the method further includes: adding the new request to a running queue.

[0140] After accepting a new request, the request will be added to the running queue and regulated by the request processing engine.

[0141] The embodiment of the present application provides a request management method for model reasoning, through the request processing engine, based on the collection and prediction of the indicator information of the model reasoning service, a scheduling suggestion for the request is issued, and the request processing engine introduces a suggestion buffer queue by slicing the scheduling function and method, and interferes with the request scheduling decision inside the reasoning framework according to the suggestion through a low-intrusive method, thereby achieving the effect of request preemption. The preemption effect of the request is achieved by pausing and resuming the computing task of the reasoning request; by slicing the request scheduling method or function inside the model reasoning service, the scheduling decision of the request scheduling method or function is merged with the decision of the sliced ​​scheduling function or method to obtain the final target scheduling decision, thereby achieving interference with the scheduling decision; on the basis of the existing model reasoning service, the scheduling optimization is performed through an external priority queue, and the scheduling mechanism inside the model reasoning service is expanded using a low-intrusive method, thereby avoiding major changes to the existing model reasoning service. This low-intrusive method simplifies the integration and maintenance of the system, so that the scheduling mechanism of the embodiment of the present application can be easily integrated into the existing model reasoning service. By introducing state prediction, the state of the model inference service seen by the request processing engine at the time of scheduling decision is aligned with the actual state of the model inference service at that moment, alleviating scheduling decision errors caused by the asynchrony inside and outside the model inference service. The request processing engine schedules requests based on the predicted future state of the model inference service, making the scheduling more accurate. Dynamic optimization of resource allocation is achieved through an external scheduling mechanism. According to the current system load, request priority, and urgency requirements, requests occupying hardware resources are adjusted in real time. In the case of resource constraints, high-priority and high-urgency requests are prioritized to achieve the effect of request preemption, thereby ensuring efficient use and reasonable allocation of hardware resources while taking into account the priority and urgency requirements of the task.

[0142] Embodiment 3

[0143] Figure 4This is a schematic diagram of the structure of a model reasoning request management device provided in Embodiment 3 of the present invention, which is applied to a request processing engine in a request management system, and the request management system also includes a model reasoning service. Figure 4 As shown, the device includes: an observation module 31, a state prediction module 32, a request scheduler 33 and a side load scheduling module 34.

[0144] An observation module 31 is used to obtain observation indicators of the model reasoning service;

[0145] A state prediction module 32, configured to perform indicator prediction based on the observed indicators to obtain prediction indicators, wherein the prediction indicators include a target memory utilization rate of a central processing unit and a target video memory utilization rate of a graphics processing unit;

[0146] A request scheduler 33, configured to determine a scheduling strategy according to the prediction indicator, determine a scheduling suggestion according to the scheduling strategy in combination with the request queue, and add the scheduling suggestion to a suggestion buffer queue;

[0147] The side-load scheduling module 34 is used to read the scheduling suggestions from the suggestion buffer queue when the model reasoning service executes request scheduling, and determine the target scheduling decision based on the scheduling suggestions and the scheduling decision of the model reasoning service, and control the model reasoning service to schedule the corresponding request to execute model reasoning according to the target scheduling decision.

[0148] A request management device for model reasoning in an embodiment of the present invention solves the problem of unreasonable request scheduling in the model reasoning process. It predicts through the observation indicators of the model reasoning service to obtain the predicted indicators. The predicted indicators include the target memory utilization of the central processing unit and the target video memory utilization of the graphics processing unit. The predicted indicators in the embodiment of the present application can be used to represent different indicators of the model reasoning service in the next time, that is, predict the indicators of the model reasoning service in the subsequent time; determine the scheduling strategy through the predicted indicators, further determine the scheduling suggestions in combination with the request queue, and add them to the suggestion buffer queue; when the model reasoning service executes request scheduling, read the scheduling suggestions from the suggestion buffer queue, and determine the target scheduling decision based on the scheduling suggestions and the scheduling decision of the model reasoning service, intervene in the request scheduling of the model reasoning service, and finally the target scheduling decision controls the model reasoning service to schedule the corresponding request to execute model reasoning; interfere with the scheduling decision of the model reasoning service through indicator prediction, provide the model reasoning service with the target scheduling decision, control the model reasoning service to execute model reasoning through reasonable request scheduling, and realize efficient utilization and reasonable allocation of resources.

[0149] Among them, the side-load scheduling module is deployed in the model reasoning service 32 so as to intervene in the scheduling strategy of the model reasoning service 32.

[0150] Optionally, the observation indicator includes at least one of a processor parameter indicator and a request indicator;

[0151] The processor parameter indicator includes at least one of the following:

[0152] CPU memory utilization;

[0153] The graphics processor's video memory utilization;

[0154] GPU core utilization;

[0155] The request indicator includes at least one of the following:

[0156] The number of requests left for prefilling;

[0157] The length of the request left for prefilling;

[0158] The number of requests in the decoding phase;

[0159] The size of the requested key-value cache to be swapped into the CPU's memory;

[0160] The requested key-value cache size to be swapped into the GPU's video memory.

[0161] Optionally, the processor parameter indicator is updated when a first indicator update condition is met;

[0162] The request indicator is updated when the second indicator update condition is met;

[0163] The first indicator update condition includes at least one of the following:

[0164] The system time meets the periodic update conditions;

[0165] The model inference service completes the scheduling of a request;

[0166] The second indicator update condition includes:

[0167] The model inference service completes the scheduling of a request.

[0168] Optionally, a state prediction module is specifically used to: use the memory size of the central processing unit, the video memory size of the graphics processing unit, and the observed index as inputs for prediction, and determine a target memory utilization rate of the central processing unit and a target video memory utilization rate of the graphics processing unit;

[0169] The memory size of the central processing unit and the video memory size of the graphics processing unit are applied for when the program is started.

[0170] Optionally, the request scheduler is specifically used to: if the target video memory utilization is higher than a first preset threshold and the target memory utilization is lower than a second preset threshold, determine the scheduling policy as a pause request; if the target video memory utilization is lower than a third preset threshold, determine the scheduling policy as a resume request.

[0171] Optionally, the request queue includes a running queue and a suspended queue;

[0172] The request scheduler is specifically used to: when the scheduling policy is a pause request, traverse the running queue, sort the requests in the running queue according to the priority level and urgency of the requests in the running queue, screen out at least one request as a pause request according to the sorting result, add the pause request to the pause request queue, and generate scheduling suggestions according to the pause request queue; when the scheduling policy is a resume request, traverse the suspend queue, sort the requests in the suspend queue according to the priority level and urgency of the requests in the suspend queue, screen out at least one request as a resume request according to the sorting result, add the resume request to the resume request queue, and generate scheduling suggestions according to the resume request queue.

[0173] Optionally, the forecasting indicators also include: the number of requests;

[0174] Correspondingly, the number of the pause requests or the resume requests is the request number.

[0175] Optionally, the side-load scheduling module is also used to: if the type corresponding to the scheduling suggestion is a pause request, determine a first candidate request to be paused according to the scheduling suggestion, take the first candidate request out of the running request, update the status of the first candidate request from running to pause, and put the first candidate request into the exchange queue; if the type corresponding to the scheduling suggestion is a resume request, determine a second candidate request to be resumed according to the scheduling suggestion, take the second candidate request out of the exchange queue, update the status of the second candidate request from pause to running, and put the second candidate request into the running request.

[0176] Optionally, a side-loaded scheduling module is specifically used to: intercept and obtain the scheduling decision of the model inference service, and record it as the first scheduling decision; determine the request to be scheduled and the scheduling type based on the scheduling suggestion; if the scheduling type of the request to be scheduled is pause, and the scheduling mode of the request to be scheduled is determined to be swap out or discard according to the first scheduling decision, set the scheduling decision of the request to be scheduled to swap out, and add the request to be scheduled to the exchange queue; if the scheduling type of the request to be scheduled is recovery, set the scheduling decision of the request to be scheduled to recovery; generate a target scheduling decision based on the scheduling decision of the request to be scheduled.

[0177] Optionally, the request processing engine 31 further includes:

[0178] The early rejection module is used to traverse the running queue to determine the currently executing user request when a new request is received; determine the execution time corresponding to each user request; if the execution time is greater than the service time limit of the new request, reject the new request; otherwise, accept the new request and send the new request to the model inference service.

[0179] Optionally, the early rejection module is further used to: add the new request to the running queue.

[0180] The request management system provided by the embodiment of the present invention can execute the request management method of model reasoning provided by any embodiment of the present invention, and has the corresponding functional modules and beneficial effects of the execution method.

[0181] Embodiment 4

[0182] Figure 5 This is a schematic diagram of the structure of a request management system provided by Embodiment 4 of the present invention. Figure 5 As shown, the system includes: a request processing engine 41 and a model reasoning service 42.

[0183] The request processing engine 41 is used to: obtain the observed indicators of the model reasoning service 42; perform indicator prediction based on the observed indicators to obtain predicted indicators, wherein the predicted indicators include the target memory utilization of the central processing unit and the target video memory utilization of the graphics processing unit; determine a scheduling strategy based on the predicted indicators, determine a scheduling suggestion based on the scheduling strategy in combination with the request queue, and add the scheduling suggestion to the suggestion buffer queue; when the model reasoning service 42 executes request scheduling, read the scheduling suggestion from the suggestion buffer queue, and determine a target scheduling decision based on the scheduling suggestion in combination with the scheduling decision of the model reasoning service, and control the model reasoning service 42 to schedule the corresponding request to execute model reasoning according to the target scheduling decision.

[0184] A request management system according to an embodiment of the present invention solves the problem of unreasonable request scheduling in a model reasoning process. Prediction is performed through observation indicators of a model reasoning service to obtain prediction indicators, wherein the prediction indicators include a target memory utilization rate of a central processing unit and a target video memory utilization rate of a graphics processing unit. In an embodiment of the present application, the prediction indicators can be used to represent different indicators of the model reasoning service in the following time, that is, to predict the indicators of the model reasoning service in the subsequent time; a scheduling strategy is determined through the prediction indicators, and a scheduling suggestion is further determined in combination with a request queue, and the scheduling suggestion is added to a suggestion buffer queue; when the model reasoning service executes request scheduling, the scheduling suggestion is read from the suggestion buffer queue, and a target scheduling decision is determined based on the scheduling suggestion and the scheduling decision of the model reasoning service, and the request scheduling of the model reasoning service is intervened. Finally, the target scheduling decision controls the model reasoning service to schedule the corresponding request to execute model reasoning; the scheduling decision of the model reasoning service is interfered with through indicator prediction, and a target scheduling decision is provided for the model reasoning service, and the model reasoning service is controlled to execute model reasoning through reasonable request scheduling, thereby realizing efficient utilization and reasonable allocation of resources.

[0185] For example, Figure 6 An implementation example diagram of request management is provided, where the request processing engine 31 includes: an early rejection module 411, an observation module 412, a state prediction module 413, a request scheduler 414, a side load scheduling module 415, and an indicator collection module 416. The side load scheduling module 415 and the indicator collection module 416 are deployed on the model reasoning service 42.

[0186] When a new request arrives, the Early Rejection module 411 checks the current system load to determine whether the service time limit requirement of the new request is met. If not, the request is rejected; if so, the request is accepted and sent to the model inference service through the API of the model inference service for processing. Figure 6 Take the example of a request sent to a model inference service through an inference request processing interface; at the same time, the request will be added to the Running queue and regulated by the request processing engine 41. After the request is received by the model inference service 42, the request can feedback the corresponding result to the user through the request processing engine 31 to complete the response to the request.

[0187] The Observer module 42 obtains the observation indicators of the model reasoning service through the API. Figure 6Take obtaining observation indicators through the indicator observation interface as an example. In each round of scheduling, the observation module 42 will request the indicator API exposed by the model reasoning service to obtain the indicators of the model reasoning service, and hand it over to the state prediction (State Predictor) module 413 to predict the future load of the model reasoning service, so that the module requesting scheduling can provide better scheduling suggestions.

[0188] The state prediction module 413 is responsible for predicting the state of the model reasoning service when the request scheduler 414 makes a scheduling decision after a short period of time based on the observation indicators of the model reasoning service collected by the observation module 42. Since the external request processing engine 41 is asynchronous with the internal operation of the model reasoning service in terms of request processing and scheduling decisions, when the request scheduler 414 makes a scheduling decision based on the observation indicators collected by the observation module 412, these indicator information has lagged behind the state of the model reasoning service itself when the decision was made. Therefore, when making a scheduling decision, the request scheduler 414 should make a scheduling decision based on the future state of the model reasoning service predicted by the state prediction module 413 (i.e., the prediction indicator).

[0189] In large model inference services, requests of different priorities and urgency have different priorities for the use of hardware. Requests with high priority and high urgency should be given priority in occupying hardware resources for service. Therefore, the request scheduler 414 is required to schedule these tasks. When hardware resources are insufficient or the system load is high, the request scheduler 414 needs to decide which tasks should preempt other tasks for priority. The request scheduler 414 determines the scheduling strategy to be executed based on the predicted GPU memory utilization and CPU memory utilization.

[0190] The request scheduler 414 includes a scanner and a priority selector. If a pause request is required, the scanner will first traverse the Running queue and sort the requests in the Running queue according to the priority and urgency of the requests in the queue; if a resume request is required, the scanner will first traverse the Hanging queue and sort the requests in the Hanging queue according to the priority and urgency of the requests in the queue. The priority selector component will filter out some requests with the lowest priority and urgency from the Running queue that has been sorted according to priority, and add them to the pause request queue (Pause list), and then issue a pause suggestion to the model reasoning service; or filter out some requests with the highest priority and urgency from the hanging queue, and add them to the resume request queue (Resume list), and then issue a resume suggestion to the model reasoning service. Figure 6 Take the example of sending the scheduling suggestion to the side-load scheduling module 415 through the request suggestion interface.

[0191] The indicator collection (Collector) module 416 will collect various indicators of the model reasoning service at a certain period and expose them to the request processing engine through the API interface for scheduling decisions. Due to the existence of the state prediction module 412 in the request processing engine 41, the indicator collection module 416 does not need to collect various indicators in the model reasoning service in a synchronous process, but collects indicators in an asynchronous manner at a certain time period and puts them into the internal cache, and updates some indicator information according to each request scheduling decision within the model reasoning service. When the request processing engine accesses the indicator information through the API, the indicator collection module 416 will directly take the indicator information from the cache and return it as an observation indicator. Compared with the synchronous collection method, the cache strategy of the indicator collection module 416 can effectively guarantee the performance of the indicator collection interface and the overall performance of the reasoning service framework.

[0192] The sidecar scheduling module 415 slices the scheduling methods or functions in the model reasoning service to influence the internal scheduling decisions of the model reasoning service in a low-intrusive manner, avoiding direct modification of the code of the scheduling methods or functions. When the scheduling suggestions of the external request processing engine arrive, they will first enter the suggestion buffer queue (Suggestions) queue of the sidecar scheduling module 415. Each time the request scheduling method or function inside the model reasoning service is executed, the sidecar scheduling module 415 will take these suggestions from the suggestion buffer queue (Suggestions) and make scheduling decisions based on the suggestions to take the request out of the running request or resume the operation of some requests:

[0193] For suggestions to pause the specified request, the side-load scheduling module 415 will take it out from the running request, record the current status of the request and put it into the swap queue (Swapped); for suggestions to resume the specified request, the side-load scheduling module 415 will take it out from the Swapped queue, restore its status and put it back into the running request.

[0194] The side-loaded scheduling module 415 will integrate these decisions with the scheduling decisions generated by the scheduling method or function within the model reasoning service to obtain the final target scheduling decision, and notify the indicator collection (Collector) module 416 of the update of the corresponding indicators based on the target scheduling decision.

[0195] Embodiment 5

[0196] Figure 7 A schematic diagram of an electronic device 50 that can be used to implement an embodiment of the present invention is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital assistants, cellular phones, smart phones, wearable devices (such as helmets, glasses, watches, etc.), and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present invention described and / or required herein.

[0197] like Figure 7As shown, the electronic device 50 includes at least one processor 51, and a memory connected to the at least one processor 51 in communication, such as a read-only memory (ROM) 52, a random access memory (RAM) 53, etc., wherein the memory stores a computer program that can be executed by at least one processor, and the processor 51 can perform various appropriate actions and processes according to the computer program stored in the read-only memory (ROM) 52 or the computer program loaded from the storage unit 58 to the random access memory (RAM) 53. In the RAM 53, various programs and data required for the operation of the electronic device 50 can also be stored. The processor 51, the ROM 52, and the RAM 53 are connected to each other via a bus 54. An input / output (I / O) interface 55 is also connected to the bus 54.

[0198] A number of components in the electronic device 50 are connected to the I / O interface 55, including: an input unit 56, such as a keyboard, a mouse, etc.; an output unit 57, such as various types of displays, speakers, etc.; a storage unit 58, such as a disk, an optical disk, etc.; and a communication unit 59, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 59 allows the electronic device 50 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0199] The processor 51 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the processor 51 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any appropriate processor, controller, microcontroller, etc. The processor 51 executes the various methods and processes described above, such as a request management method for model reasoning.

[0200] In some embodiments, the request management method for model reasoning may be implemented as a computer program, which is tangibly contained in a computer-readable storage medium, such as a storage unit 58. In some embodiments, part or all of the computer program may be loaded and / or installed on the electronic device 50 via the ROM 52 and / or the communication unit 59. When the computer program is loaded into the RAM 53 and executed by the processor 51, one or more steps of the request management method for model reasoning described above may be performed. Alternatively, in other embodiments, the processor 51 may be configured to execute the request management method for model reasoning in any other appropriate manner (e.g., by means of firmware).

[0201] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0202] Computer programs for implementing the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that when the computer program is executed by the processor, the functions / operations specified in the flow chart and / or block diagram are implemented. The computer program may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0203] An embodiment of the present invention provides a computer program product, which includes a computer program. When the computer program is executed by a processor, the method for managing model reasoning requests described in any embodiment of the present invention is implemented.

[0204] In the context of the present invention, a computer-readable storage medium may be a tangible medium that may contain or store a computer program for use by or in combination with an instruction execution system, device or equipment. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices or equipment, or any suitable combination of the foregoing. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0205] To provide interaction with a user, the systems and techniques described herein may be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and a pointing device (e.g., a mouse or trackball) through which the user can provide input to the electronic device. Other types of devices may also be used to provide interaction with the user; for example, the feedback provided to the user may be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user may be received in any form (including acoustic input, voice input, or tactile input).

[0206] The systems and techniques described herein may be implemented in a computing system that includes backend components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes frontend components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such backend components, middleware components, or frontend components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), a blockchain network, and the Internet.

[0207] A computing system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The client and server relationship is generated by computer programs running on the corresponding computers and having a client-server relationship with each other. The server may be a cloud server, also known as a cloud computing server or cloud host, which is a host product in the cloud computing service system to solve the defects of difficult management and weak business scalability in traditional physical hosts and VPS services.

[0208] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps described in the present invention can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solution of the present invention can be achieved, and this document does not limit this.

[0209] The above specific implementations do not constitute a limitation on the protection scope of the present invention. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A request management method for model reasoning, characterized in that: A request processing engine applied to a request management system, wherein the request management system also includes a model reasoning service, and the method includes: Obtaining observation indicators of the model inference service; Performing indicator prediction based on the observed indicators to obtain predicted indicators, wherein the predicted indicators include a target memory utilization rate of a central processing unit and a target video memory utilization rate of a graphics processing unit; Determine a scheduling strategy according to the prediction indicator, determine a scheduling suggestion according to the scheduling strategy combined with the request queue, and add the scheduling suggestion to the suggestion buffer queue; When the model reasoning service executes request scheduling, the scheduling suggestion is read from the suggestion buffer queue, and the target scheduling decision is determined based on the scheduling suggestion and the scheduling decision of the model reasoning service, and the model reasoning service is controlled to schedule the corresponding request to execute model reasoning according to the target scheduling decision.

2. The method according to claim 1, characterized in that The observation indicator includes at least one of a processor parameter indicator and a request indicator; The processor parameter indicator includes at least one of the following: CPU memory utilization; The graphics processor's video memory utilization; GPU core utilization; The request indicator includes at least one of the following: The number of requests left for prefilling; The length of the request left for prefilling; The number of requests in the decoding phase; The size of the requested key-value cache to be swapped into the CPU's memory; The requested key-value cache size to be swapped into the GPU's video memory.

3. The method according to claim 2, characterized in that The processor parameter indicator is updated when a first indicator update condition is met; The request indicator is updated when the second indicator update condition is met; The first indicator update condition includes at least one of the following: The system time meets the periodic update conditions; The model inference service completes the scheduling of a request; The second indicator update condition includes: The model inference service completes the scheduling of a request.

4. The method according to claim 1, characterized in that: The step of performing indicator prediction according to the observed indicator to obtain the predicted indicator includes: The memory size of the central processing unit, the video memory size of the graphics processing unit, and the observed index are used as inputs for prediction to determine a target memory utilization rate of the central processing unit and a target video memory utilization rate of the graphics processing unit; The memory size of the central processing unit and the video memory size of the graphics processing unit are applied for when the program is started.

5. The method according to claim 1, characterized in that The determining of the scheduling strategy according to the prediction index comprises: If the target video memory utilization is higher than a first preset threshold and the target memory utilization is lower than a second preset threshold, determining the scheduling strategy to be a pause request; If the target video memory utilization is lower than a third preset threshold, the scheduling strategy is determined to be a recovery request.

6. The method according to claim 1, characterized in that The request queue includes a running queue and a suspended queue, and the determining of the scheduling suggestion according to the scheduling policy in combination with the request queue includes: When the scheduling strategy is a pause request, traverse the run queue, sort the requests in the run queue according to the priority level and urgency of the requests in the run queue, select at least one request as a pause request according to the sorting result, add the pause request to a pause request queue, and generate a scheduling suggestion according to the pause request queue; When the scheduling strategy is a recovery request, the suspension queue is traversed, the requests in the suspension queue are sorted according to the priority level and urgency of the requests in the suspension queue, at least one request is screened out as a recovery request according to the sorting result, the recovery request is added to the recovery request queue, and a scheduling suggestion is generated according to the recovery request queue.

7. The method according to claim 6, characterized in that The prediction indicators also include: the number of requests; Correspondingly, the number of the pause requests or the resume requests is the request number.

8. The method according to claim 1, characterized in that Before determining the target scheduling decision according to the scheduling suggestion combined with the scheduling decision of the model reasoning service, the method further includes: If the type corresponding to the scheduling suggestion is a pause request, determine a first candidate request to be paused according to the scheduling suggestion, take the first candidate request out of the running requests, update the state of the first candidate request from running to paused, and put the first candidate request into the exchange queue; If the type corresponding to the scheduling suggestion is a recovery request, determine the second candidate request for recovery according to the scheduling suggestion, take the second candidate request out of the exchange queue, update the status of the second candidate request from paused to running, and put the second candidate request into the running request.

9. The method according to claim 1, characterized in that: Determine the target scheduling decision based on the scheduling suggestion combined with the scheduling decision of the model reasoning service, including: Intercept and obtain the scheduling decision of the model inference service and record it as the first scheduling decision; Determine the scheduling request and the scheduling type according to the scheduling suggestion; If the scheduling type of the request to be scheduled is pause, and it is determined according to the first scheduling decision that the scheduling mode of the request to be scheduled is swap out or discard, setting the scheduling decision of the request to be scheduled to swap out, and adding the request to be scheduled to the exchange queue; If the scheduling type of the request to be scheduled is recovery, setting the scheduling decision of the request to be scheduled to recovery; A target scheduling decision is generated based on the scheduling decision of the request to be scheduled.

10. The method according to any one of claims 1 to 9, characterized in that: Also includes: When a new request is received, the running queue is traversed to determine the currently executing user request; Determine the execution time corresponding to each of the user requests; If the execution time is greater than the service time limit of the new request, the new request is rejected; otherwise, the new request is accepted and sent to the model inference service.

11. The method according to claim 10, characterized in that After accepting the new request, the method further includes: The new request is added to the running queue.

12. A model reasoning request management device, characterized in that: A request processing engine applied to a request management system, wherein the request management system also includes a model reasoning service, wherein the device includes: An observation module, used to obtain observation indicators of the model reasoning service; A state prediction module, used for performing indicator prediction according to the observed indicators to obtain predicted indicators, wherein the predicted indicators include a target memory utilization rate of a central processing unit and a target video memory utilization rate of a graphics processing unit; A request scheduler, configured to determine a scheduling strategy according to the prediction indicator, determine a scheduling suggestion according to the scheduling strategy in combination with the request queue, and add the scheduling suggestion to a suggestion buffer queue; The side-loaded scheduling module is used to read scheduling suggestions from the suggestion buffer queue when the model reasoning service executes request scheduling, and determine the target scheduling decision based on the scheduling suggestions and the scheduling decision of the model reasoning service, and control the model reasoning service to schedule the corresponding request to execute model reasoning according to the target scheduling decision.

13. A request management system, characterized in that: include: Request processing engine and model inference service; The request processing engine is used to obtain observation indicators of the model reasoning service; Perform indicator prediction based on the observed indicator to obtain predicted indicators, wherein the predicted indicators include a target memory utilization of the central processing unit and a target video memory utilization of the graphics processing unit; determine a scheduling strategy based on the predicted indicators, determine a scheduling suggestion based on the scheduling strategy in combination with the request queue, and add the scheduling suggestion to the suggestion buffer queue; when the model inference service executes request scheduling, read the scheduling suggestion from the suggestion buffer queue, and determine a target scheduling decision based on the scheduling suggestion in combination with the scheduling decision of the model inference service, and control the model inference service to schedule the corresponding request to execute model inference based on the target scheduling decision.

14. An electronic device, characterized in that: The electronic device comprises: at least one processor, and a memory communicatively coupled to the at least one processor; The memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute the model reasoning request management method described in any one of claims 1-11.

15. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the model reasoning request management method described in any one of claims 1-11 when executed.

16. A computer program product, characterized in that The computer program product comprises a computer program, which, when executed by a processor, implements the request management method for model reasoning according to any one of claims 1 to 11.

Images