Model reasoning scheduling method, device and server cluster

Abstract

The present application provides a model reasoning scheduling method, device and server cluster, which relates to the field of computer technology, including: obtaining a first real-time load of the pre-filled resource pool and a second real-time load of the decoding resource pool; when both the first real-time load and the second real-time load exceed the preset load, scheduling the pre-filled request processed by the pre-filled resource pool to the decoding resource pool, so that the decoding resource pool processes the pre-filled request and the decoding request at the same time. The method and device provided by the present application improve the utilization rate of hardware resources in the server cluster and improve the response speed of large model reasoning.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a model reasoning scheduling method, device and server cluster. Background Art

[0002] With the development of deep learning technology, large language models (LLM), also known as big models, have shown excellent performance in fields such as natural language processing (NLP) and computer vision (CV).

[0003] The reasoning stage of a large model includes prefill and decode. Although the relevant technology can separate prefill and decode for online reasoning scheduling, so that prefill and decoding do not affect each other, the separation method also makes the utilization rate of the graphics processing unit (GPU) in the decoding stage low, resulting in a waste of computing resources and a slow response speed for large model reasoning. Summary of the invention

[0004] The present application provides a model reasoning scheduling method, device and server cluster, which are used to solve the technical problems of low utilization of hardware resources and slow response speed of large model reasoning during the model reasoning process.

[0005] The present application provides a model reasoning scheduling method, which is applied to a server cluster, wherein the server cluster includes a pre-filled resource pool and a decoding resource pool, and the method includes:

[0006] Acquire a first real-time load of the pre-filled resource pool and a second real-time load of the decoding resource pool;

[0007] When both the first real-time load and the second real-time load exceed a preset load, the prefilling request processed by the prefilling resource pool is scheduled to the decoding resource pool, so that the decoding resource pool processes the prefilling request and the decoding request simultaneously.

[0008] In some embodiments, scheduling the prefill request processed by the prefill resource pool to the decoding resource pool includes:

[0009] determining pre-fill requests to be scheduled;

[0010] Segmenting the pre-filling request to be scheduled to obtain a plurality of pre-filling request segments;

[0011] splicing the pre-filled request fragment with the decoded request to obtain a hybrid inference request;

[0012] The hybrid inference request is dispatched to a node in the decoding resource pool.

[0013] In some embodiments, the method further comprises:

[0014] Establishing a first request processing queue, a second request processing queue and a third request processing queue in the pre-filled resource pool;

[0015] Determining the processing priority of each request processing queue based on the text sequence length of each request processing queue; the processing priority is in inverse proportion to the text sequence length;

[0016] Adding each pre-filling request to each request processing queue based on the text sequence length of each received pre-filling request;

[0017] Get the current pre-filled request from each request processing queue in turn;

[0018] The current pre-fill request is dispatched to a node in the pre-fill resource pool.

[0019] In some embodiments, the method further comprises:

[0020] Establishing a first preempted queue in the pre-filled resource pool;

[0021] Obtaining the processing time of the current pre-fill request;

[0022] When the processing time of the current pre-filling request is longer than the first preset time, suspending the processing of the current pre-filling request, and obtaining the next pre-filling request from the first request processing queue, the second request processing queue, and the third request processing queue;

[0023] Add the current pre-filled request to the first preempted queue and start pausing processing timing;

[0024] In a case where the suspension processing duration of the current pre-filling request is greater than a second preset duration, determining a target request processing queue corresponding to the current pre-filling request based on a remaining text sequence length of the current pre-filling request;

[0025] The current pre-filled request is added to the target request processing queue.

[0026] In some embodiments, the method further comprises:

[0027] Determine the first word delay time setting value;

[0028] Searching the historical operation data of the server cluster, and obtaining the batch size corresponding to each node in the pre-filling resource pool when processing the pre-filling request when the first word delay time is less than or equal to the first word delay time setting value;

[0029] The maximum value of the batch sizes corresponding to each node when processing the pre-filling request is determined as the batch size setting value in the pre-filling resource pool.

[0030] In some embodiments, the method further comprises:

[0031] Establishing a fourth request processing queue and a fifth request processing queue in the decoding resource pool;

[0032] adding each decoding request to the fourth request processing queue based on the request time of each received decoding request;

[0033] Based on the request time of each pre-filling request scheduled from the pre-filling resource pool, each decoding request is added to the fifth request processing queue.

[0034] In some embodiments, the method further comprises:

[0035] Establishing a second intercepted queue in the decoding resource pool;

[0036] Get the processing time of the current decoding request;

[0037] When the processing time of the current decoding request is longer than the third preset time, suspending the processing of the current decoding request and obtaining the next decoding request from the fourth request processing queue;

[0038] Add the current decoding request to the second intercepted queue and start pausing processing timing;

[0039] When the pause processing time of the current decoding request is greater than a fourth preset time, the current decoding request is added to the fourth request processing queue.

[0040] In some embodiments, the method further comprises:

[0041] Determine the word unit output interval setting value;

[0042] Searching the historical operation data of the server cluster, and obtaining the batch size corresponding to each node in the decoding resource pool when processing a decoding request when the word unit output interval is less than or equal to the word unit output interval setting value;

[0043] The maximum value of the batch sizes corresponding to each node when processing a decoding request is determined as a batch size setting value in the decoding resource pool.

[0044] In some embodiments, the server cluster further includes a hybrid resource pool, and the method further includes:

[0045] When the first real-time load exceeds a preset load, the prefilling requests processed by the nodes in the prefilling resource pool are scheduled to the hybrid resource pool so that the nodes in the hybrid resource pool process the prefilling requests.

[0046] In some embodiments, the method further comprises:

[0047] When the second real-time load exceeds a preset load, the decoding requests processed by the nodes in the decoding resource pool are scheduled to the hybrid resource pool, so that the nodes in the hybrid resource pool process the decoding requests.

[0048] In some embodiments, the method further comprises:

[0049] Determining a processing priority of each inference request based on a request time of each received inference request; the inference request includes a pre-filling request scheduled from the pre-filling resource pool and / or a decoding request scheduled from the decoding resource pool;

[0050] Based on the processing priority of each inference request, each inference request is scheduled to a node in the hybrid resource pool.

[0051] In some embodiments, the method further comprises:

[0052] Acquire a third real-time load of the hybrid resource pool;

[0053] Based on the first real-time load, the second real-time load, and the third real-time load, the number of nodes in the pre-filled resource pool, the decoding resource pool, and the hybrid resource pool are adjusted.

[0054] In some embodiments, the method further comprises:

[0055] In the server cluster initialization phase, determining an inference request to be processed;

[0056] Determining a first load estimate of the pre-filled resource pool and a second load estimate of the decoding resource pool based on a text sequence length of the to-be-processed inference request;

[0057] Based on the first load estimate and the second load estimate, each node in the server cluster is allocated to the pre-filled resource pool and the decoding resource pool.

[0058] In some embodiments, the method further comprises:

[0059] Receive the current inference request;

[0060] In a case where the current inference request is a pre-filling request, obtaining the task queue length of each node in the pre-filling resource pool;

[0061] The current reasoning request is dispatched to a node corresponding to a minimum task queue length.

[0062] In some embodiments, the method further comprises:

[0063] Receive the current inference request;

[0064] When the current inference request is a decoding request, obtaining the task queue length of each node in the decoding resource pool;

[0065] The current reasoning request is dispatched to a node corresponding to a minimum task queue length.

[0066] In some embodiments, the method further comprises:

[0067] When each prefilling request processed by a node in the prefilling resource pool is completed, the key value cache and the first word element corresponding to each prefilling request are transmitted to the node corresponding to the decoding request corresponding to each prefilling request in the decoding resource pool.

[0068] The present application provides a model reasoning scheduling device, which is applied to a server cluster, wherein the server cluster includes a pre-filled resource pool and a decoding resource pool, and the device includes:

[0069] An acquisition module, used to acquire a first real-time load of the pre-filled resource pool and a second real-time load of the decoding resource pool;

[0070] A scheduling module is used to schedule the prefilling request processed by the prefilling resource pool to the decoding resource pool when both the first real-time load and the second real-time load exceed a preset load, so that the decoding resource pool processes the prefilling request and the decoding request at the same time.

[0071] The present application provides a server cluster, including multiple nodes and a model reasoning scheduling device;

[0072] The node is connected to the model reasoning scheduling device and is used to perform model reasoning;

[0073] The model reasoning scheduling device is used to execute the model reasoning scheduling method.

[0074] The present application provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the model reasoning scheduling method as described when executing the computer program.

[0075] The present application provides a non-transitory computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the model reasoning scheduling method as described is implemented.

[0076] The model reasoning scheduling method, device and server cluster provided in the present application are applied to a server cluster, wherein the server cluster includes a pre-filling resource pool and a decoding resource pool, and obtains a first real-time load of the pre-filling resource pool and a second real-time load of the decoding resource pool; when the first real-time load and the second real-time load both exceed the preset load, the pre-filling request processed by the node in the pre-filling resource pool is scheduled to the decoding resource pool, so that the nodes in the decoding resource pool process the pre-filling request and the decoding request at the same time, so that when the server cluster executes the model reasoning task, it can make full use of the graphics processor of each node in the decoding resource pool, avoid wasting computing power resources, improve the utilization rate of hardware resources in the server cluster, and improve the response speed of large model reasoning. BRIEF DESCRIPTION OF THE DRAWINGS

[0077] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0078] In order to more clearly illustrate the technical solutions in the present application or the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0079] Figure 1 It is a flowchart of the model reasoning scheduling method provided in this application.

[0080] Figure 2 It is a schematic diagram of a hybrid reasoning request provided by this application.

[0081] Figure 3 It is a structural diagram of the model reasoning scheduling device provided in this application.

[0082] Figure 4 It is a schematic diagram of the structure of the server cluster provided in this application.

[0083] Figure 5 This is a model reasoning architecture diagram of the server cluster provided in this application.

[0084] Figure 6 It is a schematic diagram of the structure of the electronic device provided by this application. DETAILED DESCRIPTION

[0085] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present application.

[0086] It should be noted that the terms "first", "second" etc. in the present application are used to distinguish similar objects, and need not be used to describe a specific order or sequential order. It should be understood that the data used in this way can be interchangeable in appropriate circumstances, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, the process, method, system, product or equipment comprising a series of steps or units need not be limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or equipment.

[0087] The development and application of large models can be divided into three process steps: pre-training, fine-tuning, and reasoning. Large model reasoning refers to the actual application and inference based on the pre-trained model. The input length of the large model is fixed, and the output length is not fixed, so it can only rely on partial information for reasoning; the model usually requires multiple cards on a single machine or multiple cards on multiple machines for reasoning, so communication between cards and between machines will increase the complexity of reasoning.

[0088] The reasoning of large models mainly includes pre-filling and decoding. Pre-filling is mainly used to load the model and prepare input data, which usually involves initializing model parameters and optimizers. This stage is a highly parallel matrix operation, which requires high computing resources and is a computationally intensive stage. Decoding uses a method of gradually generating tokens, which has high real-time requirements and requires fast processing and output generation. The demand for latency optimization and high throughput is more obvious at this stage, which is a bandwidth-intensive stage.

[0089] In the process of online reasoning and scheduling of large models, different methods can be used to accelerate the online reasoning of large models through different scheduling methods. Although pre-filling and decoding can be separated for online reasoning scheduling so that pre-filling and decoding do not affect each other, the separation method also makes the utilization rate of the graphics processor in the decoding stage low, resulting in a waste of computing resources.

[0090] In order to solve the shortcomings of related technologies, Figure 1 is a flowchart of the model reasoning scheduling method provided by this application, such as Figure 1 As shown, the method is applied to a server cluster, and the server cluster includes a pre-filled resource pool and a decoding resource pool. The method includes steps 110 and 120.

[0091] Step 110: Obtain a first real-time load of a pre-filled resource pool and a second real-time load of a decoding resource pool.

[0092] Specifically, the execution subject of the model reasoning scheduling method provided in the embodiment of the present application is a model reasoning scheduling device. The device can be implemented by software, such as a model reasoning scheduling program; it can also be implemented by hardware, such as a computer or server that executes the model reasoning scheduling method. The method provided in the embodiment of the present application is applicable to various large models, and the embodiment of the present application does not specifically limit the model structure and number of parameters of the large model.

[0093] A server cluster is a system consisting of multiple servers connected through a network, working together to provide higher availability, reliability and performance. The cluster can share the load, achieve failover and load balancing, and is suitable for scenarios with large-scale applications, data processing and high availability requirements. By integrating multiple servers, the cluster can continue to ensure the continuity of services when a server fails. Each server in the server cluster can serve as a node in the cluster. In an embodiment of the present application, the server cluster is used to perform model reasoning tasks.

[0094] In an embodiment of the present application, the model reasoning scheduling device may be a computer or server independently arranged in a server cluster, or may be a node in the server cluster that executes the model reasoning scheduling method.

[0095] A resource pool is a combination of hardware resources that are obtained after planning and allocating hardware resources in a server cluster according to different task requirements. The hardware resources in a resource pool can be dynamically allocated and adjusted to meet the needs of different applications and improve resource utilization and flexibility. For example, the computing resources (CPU, memory, storage resources, GPU) and network resources of multiple nodes are integrated into a unified resource collection, which is called a resource pool.

[0096] In the embodiment of the present application, each node in the server cluster can be divided into a pre-filled resource pool and a decoding resource pool according to different tasks performed by the node. The number of nodes in the pre-filled resource pool and the decoding resource pool can be set and adjusted as needed.

[0097] The inference requests received by the server cluster can be divided into pre-filling requests and decoding requests according to the processing stage. For example, a user can enter a text into the server cluster for semantic understanding. The server cluster generates an inference request based on the input text. First, the text input needs to be pre-filled into the resource pool, and each node needs to convert the text into an embedding vector, generate an initial key value vector (key value) and the first word, in preparation for the decoding stage. At this time, the inference request in the pre-filling stage is a pre-filling request. After the pre-filling process is completed, the generated initial key value vector and the first word can be sent to the decoding resource pool, and each node generates the output text based on the initial key value vector and the first word. At this time, the inference request in the decoding stage is a decoding request.

[0098] Load refers to the amount of work and resource usage of each server in the cluster. Load can include metrics such as processor utilization, memory usage, and network bandwidth. By monitoring and managing these loads, the cluster can achieve load balancing and ensure that each server shares the work evenly, thereby improving overall performance and reliability.

[0099] The first real-time load refers to the real-time value of the load when each node in the pre-filling resource pool processes a pre-filling request. The second real-time load refers to the real-time value of the load when each node in the decoding resource pool processes a decoding request.

[0100] Step 120: When both the first real-time load and the second real-time load exceed a preset load, the prefilling request processed by the prefilling resource pool is scheduled to the decoding resource pool, so that the decoding resource pool processes the prefilling request and the decoding request at the same time.

[0101] Specifically, in order to improve the stability and response speed of the server cluster in executing model reasoning, a preset load can be set for the pre-filled resource pool and the decoding resource pool to limit the real-time load and avoid node overload.

[0102] The preset load can be set as needed. For different resource pools, different load indicators can be selected and different preset loads can be set. For example, for the pre-filling resource pool, since the pre-filling request needs to consume computing resources, the preset load can be set according to the processor utilization to obtain the first preset load corresponding to the first real-time load; for the decoding resource pool, since the decoding request needs to consume bandwidth resources, the preset load can be set according to the bandwidth to obtain the second preset load corresponding to the second real-time load.

[0103] For nodes in the server cluster, model inference tasks are mainly performed through the graphics processors set in the nodes. For graphics processors, the most important thing is computing power, which provides computing resources to the outside world.

[0104] When both the first real-time load and the second real-time load exceed the preset load, for each node in the pre-filled resource pool, the computing power resources provided by its graphics processor have reached the upper limit; and for each node in the decoding resource pool, the bandwidth resources provided by its graphics processor have reached the upper limit, but the computing power resources provided are not fully utilized, resulting in a waste of computing power resources.

[0105] Therefore, the prefilling requests processed by the nodes in the prefilling resource pool can be scheduled to the decoding resource pool, so that the nodes in the decoding resource pool can process the prefilling requests and the decoding requests at the same time. In this case, the computing power resources of the graphics processors set in the nodes in the decoding resource pool will be fully released and used, thereby improving the query per second (QPS) response speed of the large model.

[0106] The model inference scheduling method provided in the embodiment of the present application is applied to a server cluster, wherein the server cluster includes a pre-filling resource pool and a decoding resource pool, and obtains a first real-time load of the pre-filling resource pool and a second real-time load of the decoding resource pool; when the first real-time load and the second real-time load both exceed the preset load, the pre-filling request processed by the node in the pre-filling resource pool is scheduled to the decoding resource pool, so that the nodes in the decoding resource pool process the pre-filling request and the decoding request at the same time, so that the server cluster can make full use of the graphics processor of each node in the decoding resource pool when executing the model inference task, avoid wasting computing power resources, improve the utilization rate of hardware resources in the server cluster, and improve the response speed of large model inference.

[0107] It should be noted that each implementation method of the present application can be freely combined, the order can be changed, or it can be executed separately, and does not need to rely on or depend on a fixed execution order.

[0108] In some embodiments, scheduling a prefill request processed by a node in a prefill resource pool to a decoding resource pool includes:

[0109] determining pre-fill requests to be scheduled;

[0110] The pre-filling request to be scheduled is segmented to obtain multiple pre-filling request fragments;

[0111] Concatenate the pre-filled request fragment with the decoded request to obtain a hybrid inference request;

[0112] Schedule hybrid inference requests to nodes in the decoding resource pool.

[0113] Specifically, when scheduling the prefilling request processed by the node in the prefilling resource pool to the decoding resource pool, the prefilling request to be scheduled may be determined first. For example, the prefilling request to be scheduled may be an inference request newly input into the server cluster.

[0114] For each node in the decoding resource pool, if a decoding request priority strategy is adopted, the pre-filling requests scheduled to the decoding resource pool may not be processed for a long time, resulting in delayed responses to these pre-filling requests, which in turn causes delayed responses to the decoding requests corresponding to these pre-filling requests, and ultimately makes the entire inference request unable to respond in time.

[0115] Therefore, the pre-filling request to be scheduled can be segmented to obtain multiple pre-filling request fragments, and the pre-filling request fragments can be spliced ​​with the existing decoding request to obtain a hybrid inference request. The hybrid inference request is scheduled to the nodes in the decoding resource pool so that these nodes can process the pre-filling request and the decoding request at the same time.

[0116] Figure 2 is a schematic diagram of a hybrid reasoning request provided by this application, such as Figure 2 As shown in the figure, the decoding request Ad is the request corresponding to the inference request A in the decoding stage, the decoding request Bd is the request corresponding to the inference request B in the decoding stage, and the pre-filling request Cp is the request corresponding to the inference request C in the pre-filling stage. The pre-filling request Cp is divided into segments Cp1 and Cp2. The segments Cp1 and Cp2 are concatenated with the decoding request Ad and the decoding request Bd and processed as a batch. Each square represents an iterative calculation process in a batch.

[0117] The model inference scheduling method provided in the embodiment of the present application divides the pre-filling request to be scheduled into multiple pre-filling request fragments, splices the pre-filling request fragments with the decoding request, and schedules them to the nodes in the decoding resource pool. This can make full use of the graphics processor of each node in the decoding resource pool, avoid wasting computing power resources, and improve the utilization rate of hardware resources in the server cluster.

[0118] In some embodiments, the method further comprises:

[0119] Establishing a first request processing queue, a second request processing queue, and a third request processing queue in the pre-filled resource pool;

[0120] Based on the length of the text sequence of each request processing queue, the processing priority of each request processing queue is determined; the processing priority is inversely proportional to the length of the text sequence;

[0121] Adding each pre-filling request to each request processing queue based on the text sequence length of each received pre-filling request;

[0122] Get the current pre-filled request from each request processing queue in turn;

[0123] Schedules the current prefill request to a node in the prefill resource pool.

[0124] Specifically, a machine-level prefill scheduler (MLPS) may be set to schedule nodes in the prefill resource pool to process prefill requests.

[0125] The prefill phase is a computationally intensive phase that requires a lot of computing power. The amount of computing increases with the length of the user input token. Usually, only one model inference is required. In this phase, the model will use parallel processing capabilities to process all input tokens at once and output the first token (only occupies 1 token). At the same time, the prefill phase will generate the initial key-value vector.

[0126] A first request processing queue, a second request processing queue and a third request processing queue are established in the pre-filled resource pool. For example, a first request processing queue, a second request processing queue and a third request processing queue are established in the pre-filled resource pool, and these three request processing queues respectively place requests of light, medium and heavy levels.

[0127] The text sequence length refers to the number of words in the input text to be processed. The processing priority of each request processing queue is determined based on the text sequence length of each request processing queue, and the processing priority is inversely proportional to the text sequence length. For example, the text sequence length of the first request processing queue is less than or equal to 18 words, the text sequence length of the second request processing queue is greater than 18 words and less than or equal to 512 words, and the text sequence length of the third request processing queue is greater than 512 words. The processing priority of the first request processing queue is greater than that of the second request processing queue, and the processing priority of the second request processing queue is greater than that of the third request processing queue.

[0128] The prefill resource pool receives multiple prefill requests, and adds each prefill request to each request processing queue according to the length of the text sequence of each received prefill request. A round-bin method can be adopted to obtain the current prefill request from each request processing queue in turn, and schedule the current prefill request to a node in the prefill resource pool for processing.

[0129] The model inference scheduling method provided in the embodiment of the present application establishes a first request processing queue, a second request processing queue and a third request processing queue in a pre-filled resource pool, can process multiple concurrent requests in an orderly manner, give priority to pre-filled requests with a smaller text sequence length, and improve the response speed of large model reasoning.

[0130] In some embodiments, the method further comprises:

[0131] Establishing a first preempted queue in the pre-filled resource pool;

[0132] Get the processing time of the current pre-fill request;

[0133] When the processing time of the current pre-filling request is longer than the first preset time, suspend processing of the current pre-filling request, and obtain the next pre-filling request from the first request processing queue, the second request processing queue, and the third request processing queue;

[0134] Add the current pre-filled request to the first preempted queue and start the pause processing timer;

[0135] When the suspension processing duration of the current pre-filling request is greater than the second preset duration, determining a target request processing queue corresponding to the current pre-filling request based on the remaining text sequence length of the current pre-filling request;

[0136] Add the current pre-populated request to the target request processing queue.

[0137] Specifically, a first preempted queue can be established in the prefill resource pool to suspend some prefill requests that have timed out and speed up the processing of other prefill requests. The reason for the prefill request processing timeout may be that the text sequence length is long, such as greater than 2048 words.

[0138] The processing time of each pre-fill request is monitored to obtain the processing time of the current pre-fill request.

[0139] If the processing time of the current pre-fill request is longer than the first preset time, it means that the processing time of the current pre-fill request is too long, which affects the response speed of the model inference. At this time, the processing of the current pre-fill request can be suspended, and the current pre-fill request can be removed from the currently processed batch. The next pre-fill request is obtained from the first request processing queue, the second request processing queue and the third request processing queue for processing.

[0140] Add the current pre-fill request to the first pre-empted queue and start the pause processing timer. If the pause processing duration of the current pre-fill request is greater than the second preset duration, the current pre-fill request can be removed from the first pre-empted queue, and the target request processing queue corresponding to the current pre-fill request is determined according to the remaining text sequence length of the current pre-fill request, and the current pre-fill request is added to the target request processing queue. The target request processing queue refers to a request processing queue that is re-determined based on the number of unprocessed words in the current pre-fill request. For example, the current pre-fill request originally had 1024 words and belonged to the third request processing queue. Now there are still 200 words left to be processed, so the current pre-fill request is added to the second request processing queue.

[0141] The first preset time length and the second preset time length can be set as needed, for example, 1 second.

[0142] The model inference scheduling method provided in the embodiment of the present application suspends some pre-filling requests whose processing time has timed out, speeds up the processing of other pre-filling requests, improves the processing efficiency of pre-filling requests in the case of a large number of concurrent requests, and improves the response speed of large model reasoning.

[0143] In some embodiments, in order to avoid the problem of throughput degradation caused by too many tokens processed in a batch during the pre-filling stage, the total number of tokens processed in a batch can be limited to 2048. Since the performance of pre-filling decreases after 2048 tokens, its operation will be interrupted when the number of service tokens exceeds 2048.

[0144] In some embodiments, the method further comprises:

[0145] Determine the first word delay time setting value;

[0146] Search the historical operation data of the server cluster, and when the first word delay time is less than or equal to the first word delay time setting value, obtain the batch size corresponding to each node in the pre-filling resource pool when processing the pre-filling request;

[0147] The maximum value of the batch sizes corresponding to each node when processing the pre-filling request is determined as the batch size setting value in the pre-filling resource pool.

[0148] Specifically, Time to First Token (TTFT) refers to the time from sending a request to the system returning the first token response. This metric is used to measure the response speed of large models, especially in natural language processing and generation tasks. A shorter TTFT indicates that the system can respond quickly to user requests and improve user experience.

[0149] You can set the first-word latency setting value to measure the response time of the server cluster performing model inference.

[0150] Collect historical operation data of the server cluster. Search the historical operation data (you can use a brute force search algorithm, also known as an exhaustive search algorithm). When the first word delay time is less than or equal to the first word delay time setting value, obtain the batch size corresponding to each node in the pre-fill resource pool when processing the pre-fill request. The batch size refers to the number of requests processed in one training or inference.

[0151] For example, the VLLM framework (online reasoning framework) and BentoML are used for brute force search. VLLM (Very Large Language Model) and BentoML are tools related to large model deployment and management.

[0152] The maximum value of the batch size corresponding to each node when processing the pre-filling request is determined as the batch size setting value in the pre-filling resource pool. For example, if there are multiple request processing queues in the pre-filling resource pool, the maximum value of the batch size corresponding to each node when processing the pre-filling request in each request processing queue can be used as the batch size setting value in the pre-filling resource pool. According to the batch size setting value, the batch size of the request for executing model inference in the pre-filling resource pool is restricted.

[0153] The model inference scheduling method provided in the embodiment of the present application determines the batch size setting value in the pre-filled resource pool according to the first word delay time setting value, thereby improving the response speed of large model inference.

[0154] In some embodiments, the method further comprises:

[0155] Establishing a fourth request processing queue and a fifth request processing queue in the decoding resource pool;

[0156] Adding each decoding request to a fourth request processing queue based on the request time of each received decoding request;

[0157] Based on the request time of each pre-filling request scheduled from the pre-filling resource pool, each decoding request is added to a fifth request processing queue.

[0158] Specifically, a machine-level decode scheduler (MLDS) may be set to schedule nodes in the pre-filling resource pool to process pre-filling requests.

[0159] The decoding phase is a bandwidth-intensive phase with high bandwidth requirements. The model uses the initial key-value vector and the first token generated in the pre-filling phase to calculate the probability distribution of the next token. Then, based on this probability distribution, a token is selected as the next token to be generated and appended to the previous text sequence. This process is repeated until the entire text sequence is generated.

[0160] A fourth request processing queue and a fifth request processing queue may be established in the decoding resource pool.

[0161] According to the request time of each received decoding request, each decoding request is added to a fourth request processing queue. The fourth request processing queue may also be called a waiting queue. The decoding resource pool schedules the decoding requests in the queue using a First-Come First-Served (FCFS) scheduling algorithm.

[0162] According to the request time of each pre-filling request scheduled from the pre-filling resource pool, each decoding request is added to the fifth request processing queue. The fifth request processing queue is because the inference request in the pre-filling stage (that is, the pre-filling request) is inserted into the fifth request processing queue after the load of both pre-filling and decoding exceeds the threshold, and is spliced ​​into a batch (batch) with the decoding request for inference.

[0163] The model inference scheduling method provided in the embodiment of the present application establishes a fourth request processing queue and a fifth request processing queue in the decoding resource pool, which can process multiple concurrent decoding requests and pre-filling requests in an orderly manner, thereby improving the response speed of large model inference.

[0164] In some embodiments, the method further comprises:

[0165] Establishing a second intercepted queue in the decoding resource pool;

[0166] Get the processing time of the current decoding request;

[0167] When the processing time of the current decoding request is longer than the third preset time, suspending the processing of the current decoding request and obtaining the next decoding request from the fourth request processing queue;

[0168] Add the current decoding request to the second intercepted queue and start the pause processing timer;

[0169] When the pause processing duration of the current decoding request is greater than the fourth preset duration, the current decoding request is added to the fourth request processing queue.

[0170] Specifically, a second preempted queue can be established in the decoding resource pool to suspend some decoding requests with timeout and speed up the processing of other decoding requests. The reason for the processing timeout of the decoding request may be that there are too many word units generated by decoding, such as more than 512 word units.

[0171] The processing time of each decoding request is monitored to obtain the processing time of the current decoding request.

[0172] If the processing time of the current decoding request is greater than the third preset time, it means that the processing time of the current decoding request has timed out. The processing of the current decoding request can be suspended, the current decoding request can be removed from the current processing batch, and the next decoding request can be obtained from the fourth request processing queue for processing.

[0173] The current decoding request is added to the second intercepted queue and the pause processing timing is started. If the pause processing time of the current decoding request is longer than the fourth preset time, the current decoding request can be re-added to the fourth request processing queue for processing.

[0174] The third preset time length and the fourth preset time length can be set as needed, for example, 1 second.

[0175] The model inference scheduling method provided in the embodiment of the present application suspends some decoding requests that have timed out, speeds up the processing of other decoding requests, improves the processing efficiency of decoding requests in the case of a large number of concurrent requests, and improves the response speed of large model inference.

[0176] In some embodiments, the method further comprises:

[0177] Determine the word unit output interval setting value;

[0178] Search the historical operation data of the server cluster, and when the word unit output interval is less than or equal to the word unit output interval setting value, obtain the batch size corresponding to each node in the decoding resource pool when processing the decoding request;

[0179] The maximum value of the batch sizes corresponding to each node when processing a decoding request is determined as a batch size setting value in the decoding resource pool.

[0180] Specifically, the Time Between Tokens (TBT) refers to the time interval between two adjacent tokens when generating model output. The Time Between Tokens measures the speed at which the model generates text, usually in milliseconds. A shorter Time Between Tokens means that the model can generate output more quickly, improving the fluency of real-time interaction.

[0181] You can set a time setting for the word output interval to measure the response time of the server cluster executing model inference.

[0182] Collect historical operation data of the server cluster. Search the historical operation data (you can use a brute force search algorithm, also known as an exhaustive search algorithm). When the word unit output interval is less than or equal to the word unit output interval setting value, obtain the batch size (batch size) corresponding to each node in the decoding resource pool when processing the decoding request, and determine the maximum value of the batch size corresponding to each node when processing the decoding request as the batch size setting value in the decoding resource pool. For example, use the VLLM framework (online reasoning framework) and BentoML for brute force search. According to the batch size setting value, limit the batch size of the request to perform model reasoning in the decoding resource pool.

[0183] The model inference scheduling method provided in the embodiment of the present application determines the batch size setting value in the decoding resource pool according to the word unit output interval time setting value, thereby improving the response speed of large model inference.

[0184] In some embodiments, when determining the batch size setting value in the decoding resource pool according to the word unit output interval setting value, the number of word units in the decoding request can be set to 1 or a random number. The number of input word units can be continuously increased, and the maximum number of input word units that can meet the delay condition (the maximum number that can be spliced ​​into a batch) can be obtained by searching in the historical operation data.

[0185] In some embodiments, the server cluster further includes a hybrid resource pool, and the method further includes:

[0186] When the first real-time load exceeds the preset load, the prefilling requests processed by the nodes in the prefilling resource pool are scheduled to the hybrid resource pool, so that the nodes in the hybrid resource pool process the prefilling requests.

[0187] Specifically, when the resources in the server cluster are sufficient, a hybrid resource pool can be set up. The nodes in the hybrid resource pool can be deployed in a combined manner to perform batch processing data mixed with pre-filled requests and decoded requests. When the request load changes drastically and the nodes cannot scale up or switch roles in time, the hybrid resource pool will perform the work.

[0188] If the first real-time load exceeds the preset load, it means that the nodes in the pre-filling resource pool cannot process the pre-filling requests currently scheduled to the pre-filling resource pool in time. These pre-filling requests can be scheduled to the hybrid resource pool so that the nodes in the hybrid resource pool process the pre-filling requests.

[0189] The nodes in the hybrid resource pool can be pre-allocated nodes or surplus nodes in the decoding resource pool. For example, nodes that process decoding requests that do not exceed the request quantity threshold limit can be temporarily allocated to the hybrid resource pool and returned to the decoding resource pool after processing the pre-filled requests.

[0190] The model inference scheduling method provided in the embodiment of the present application can process pre-filled requests by mixing nodes in the resource pool, thereby improving the utilization rate of hardware resources in the server cluster and improving the response speed of large model reasoning.

[0191] In some embodiments, the method further comprises:

[0192] When the second real-time load exceeds the preset load, the decoding requests processed by the nodes in the decoding resource pool are scheduled to the hybrid resource pool, so that the nodes in the hybrid resource pool process the decoding requests.

[0193] Specifically, if the second real-time load exceeds the preset load, it means that the nodes in the decoding resource pool cannot process the decoding requests currently scheduled to the decoding resource pool in a timely manner. These decoding requests can be scheduled to the hybrid resource pool so that the nodes in the hybrid resource pool process the decoding requests.

[0194] The nodes in the hybrid resource pool can be pre-allocated nodes or surplus nodes in the pre-filled resource pool. For example, nodes that process pre-filled requests but do not exceed the request quantity threshold limit can be temporarily allocated to the hybrid resource pool and returned to the pre-filled resource pool after processing the pre-filled requests.

[0195] The model inference scheduling method provided in the embodiment of the present application can process decoding requests by mixing nodes in the resource pool, thereby improving the utilization rate of hardware resources in the server cluster and improving the response speed of large model inference.

[0196] In some embodiments, based on the request time of each received inference request, a processing priority of each inference request is determined; the inference request includes a pre-filling request scheduled from a pre-filling resource pool and / or a decoding request scheduled from a decoding resource pool;

[0197] Based on the processing priority of each inference request, each inference request is scheduled to a node in the hybrid resource pool.

[0198] Specifically, the processing priority of each inference request can be determined according to the request time of each received inference request. The earlier the request time of the inference request, the higher the processing priority. It can be understood that the inference request processed by the hybrid resource pool includes the pre-filling request scheduled from the pre-filling resource pool and / or the decoding request scheduled from the decoding resource pool.

[0199] The model inference scheduling method provided in the embodiment of the present application determines the processing priority in the hybrid resource pool according to the request time of each inference request, thereby improving the response speed of large model inference.

[0200] In some embodiments, the method further comprises:

[0201] Obtaining the third real-time load of the hybrid resource pool;

[0202] Based on the first real-time load, the second real-time load, and the third real-time load, the number of nodes in the pre-filled resource pool, the decoding resource pool, and the hybrid resource pool is adjusted.

[0203] Specifically, a cluster-level scheduler (CLS) may be set up to schedule hardware resources in the entire server cluster.

[0204] When a pre-filled resource pool, a decoding resource pool, and a hybrid resource pool are set in a server cluster, the real-time load of each resource pool can be monitored. The first real-time load is used to measure the load of the pre-filled resource pool, the second real-time load is used to measure the load of the decoding resource pool, and the third real-time load is used to measure the load of the hybrid resource pool.

[0205] According to the first real-time load, the second real-time load and the third real-time load, the number of nodes in the pre-filled resource pool, the decoding resource pool and the hybrid resource pool is adjusted. For example, the nodes in the resource pool with lower real-time load are allocated to the resource pool with higher real-time load to realize dynamic allocation of resources.

[0206] The model reasoning scheduling method provided in the embodiment of the present application dynamically adjusts the number of nodes in each resource pool according to the real-time load of each resource pool, realizes resource sharing, and improves the overall resource utilization of the server cluster.

[0207] In some embodiments, the method further comprises:

[0208] During the server cluster initialization phase, the inference requests to be processed are determined;

[0209] Determine, based on a text sequence length of an inference request to be processed, a first load estimate of a prefilled resource pool and a second load estimate of a decoding resource pool;

[0210] Based on the first load estimate and the second load estimate, each node in the server cluster is allocated to a pre-filling resource pool and a decoding resource pool.

[0211] Specifically, during the server cluster initialization phase, pending inference requests that need to be processed in parallel can be determined, and based on the text sequence length of the pending inference requests, the load of the pre-filled resource pool and the load of the decoding resource pool can be estimated to obtain a first load estimate and a second load estimate.

[0212] According to the first load estimate and the second load estimate, as well as the load that each node can handle, each node in the server cluster is allocated to a pre-filled resource pool and a decoding resource pool to achieve optimal configuration of nodes in each resource pool.

[0213] The model inference scheduling method provided in the embodiment of the present application allocates each node in the server cluster to a pre-filled resource pool and a decoding resource pool according to the load estimate during the server cluster initialization phase, thereby achieving resource sharing and improving the overall resource utilization of the server cluster.

[0214] In some embodiments, the method further comprises:

[0215] Receive the current inference request;

[0216] When the current inference request is a pre-filling request, the task queue length of each node in the pre-filling resource pool is obtained;

[0217] Schedule the current inference request to the node corresponding to the minimum task queue length.

[0218] Specifically, the server cluster receives the current inference request. If the current inference request is a pre-filled request, the task queue length of each node in the pre-filled resource pool can be obtained. The cluster-level scheduler adopts the Join the Shortest Queue (JSQ) method for the routing strategy of the request to schedule the current inference request to the node corresponding to the minimum task queue length in the pre-filled resource pool.

[0219] The model inference scheduling method provided in the embodiment of the present application schedules pre-filling requests according to the task queue length of each node in the pre-filling resource pool, thereby improving the response speed of large model inference.

[0220] In some embodiments, the method further comprises:

[0221] Receive the current inference request;

[0222] When the current inference request is a decoding request, the task queue length of each node in the decoding resource pool is obtained;

[0223] Schedule the current inference request to the node corresponding to the minimum task queue length.

[0224] Specifically, the server cluster receives the current inference request. If the current inference request is a decoding request, the task queue length of each node in the decoding resource pool can be obtained. The cluster-level scheduler adopts the Join the Shortest Queue (JSQ) method for the routing strategy of the request to schedule the current inference request to the node corresponding to the minimum task queue length in the decoding resource pool.

[0225] The model inference scheduling method provided in the embodiment of the present application schedules decoding requests according to the task queue length of each node in the decoding resource pool, thereby improving the response speed of large model inference.

[0226] In some embodiments, the method further comprises:

[0227] When each prefilling request processed by a node in the prefilling resource pool is completed, the key value cache and the first word element corresponding to each prefilling request are transmitted to the node corresponding to the decoding request corresponding to each prefilling request in the decoding resource pool.

[0228] Specifically, the server cluster receives an inference request, allocates a node in a pre-filling resource pool to process a pre-filling request corresponding to the inference request, and allocates a node in a decoding resource pool to process a decoding request corresponding to the inference request.

[0229] After the pre-fill request is executed, the corresponding key-value cache (KV cache) and the first word are generated and transmitted to the node corresponding to the decoding request corresponding to the pre-fill request in the decoding resource pool, and decoding is continued to complete the model inference.

[0230] The model inference scheduling method provided in the embodiment of the present application transmits the key value cache and the first word element corresponding to each pre-filled request to the node corresponding to the decoding request corresponding to each pre-filled request in the decoding resource pool, thereby improving the response speed of large model inference.

[0231] In some embodiments, nodes in a server cluster perform inference using a tensor parallel strategy.

[0232] Specifically, the tensor parallel strategy divides and distributes the tensors of large models to multiple computing devices during inference to achieve higher computing efficiency. By distributing different tensor operations on multiple graphics processors, the memory burden of each device can be reduced while accelerating the inference process.

[0233] The device provided in the embodiment of the present application is described below. The device described below and the method described above can be referenced to each other.

[0234] Figure 3 is a schematic diagram of the structure of the model reasoning scheduling device provided by this application, such as Figure 3As shown, the device is applied to a server cluster, the server cluster includes a pre-filled resource pool and a decoding resource pool, and the device includes:

[0235] An acquisition module 310 is used to acquire a first real-time load of a pre-filled resource pool and a second real-time load of a decoding resource pool;

[0236] The scheduling module 320 is used to schedule the prefilling requests processed by the nodes in the prefilling resource pool to the decoding resource pool when both the first real-time load and the second real-time load exceed the preset load, so that the nodes in the decoding resource pool can process the prefilling requests and the decoding requests at the same time.

[0237] The model inference scheduling device provided in the embodiment of the present application is applied to a server cluster, wherein the server cluster includes a pre-filling resource pool and a decoding resource pool, and obtains a first real-time load of the pre-filling resource pool and a second real-time load of the decoding resource pool; when the first real-time load and the second real-time load both exceed the preset load, the pre-filling request processed by the node in the pre-filling resource pool is scheduled to the decoding resource pool, so that the nodes in the decoding resource pool process the pre-filling request and the decoding request at the same time, so that the server cluster can make full use of the graphics processor of each node in the decoding resource pool when executing the model inference task, avoid wasting computing power resources, improve the utilization rate of hardware resources in the server cluster, and improve the response speed of large model reasoning.

[0238] In some embodiments, the scheduling module is used to:

[0239] determining pre-fill requests to be scheduled;

[0240] The pre-filling request to be scheduled is segmented to obtain multiple pre-filling request fragments;

[0241] Concatenate the pre-filled request fragment with the decoded request to obtain a hybrid inference request;

[0242] Schedule hybrid inference requests to nodes in the decoding resource pool.

[0243] In some embodiments, the scheduling module is further configured to:

[0244] Establishing a first request processing queue, a second request processing queue, and a third request processing queue in the pre-filled resource pool;

[0245] Based on the length of the text sequence of each request processing queue, the processing priority of each request processing queue is determined; the processing priority is inversely proportional to the length of the text sequence;

[0246] Adding each pre-filling request to each request processing queue based on the text sequence length of each received pre-filling request;

[0247] Get the current pre-filled request from each request processing queue in turn;

[0248] Schedules the current prefill request to a node in the prefill resource pool.

[0249] In some embodiments, the scheduling module is further configured to:

[0250] Establishing a first preempted queue in the pre-filled resource pool;

[0251] Get the processing time of the current pre-fill request;

[0252] When the processing time of the current pre-filling request is longer than the first preset time, suspend processing of the current pre-filling request, and obtain the next pre-filling request from the first request processing queue, the second request processing queue, and the third request processing queue;

[0253] Add the current pre-filled request to the first preempted queue and start the pause processing timer;

[0254] When the suspension processing duration of the current pre-filling request is greater than the second preset duration, determining a target request processing queue corresponding to the current pre-filling request based on the remaining text sequence length of the current pre-filling request;

[0255] Add the current pre-populated request to the target request processing queue.

[0256] In some embodiments, the scheduling module is further configured to:

[0257] Determine the first word delay time setting value;

[0258] Search the historical operation data of the server cluster, and when the first word delay time is less than or equal to the first word delay time setting value, obtain the batch size corresponding to each node in the pre-filling resource pool when processing the pre-filling request;

[0259] The maximum value of the batch sizes corresponding to each node when processing the pre-filling request is determined as the batch size setting value in the pre-filling resource pool.

[0260] In some embodiments, the scheduling module is further configured to:

[0261] Establishing a fourth request processing queue and a fifth request processing queue in the decoding resource pool;

[0262] Adding each decoding request to a fourth request processing queue based on the request time of each received decoding request;

[0263] Based on the request time of each pre-filling request scheduled from the pre-filling resource pool, each decoding request is added to a fifth request processing queue.

[0264] In some embodiments, the scheduling module is further configured to:

[0265] Establishing a second intercepted queue in the decoding resource pool;

[0266] Get the processing time of the current decoding request;

[0267] When the processing time of the current decoding request is longer than the third preset time, suspending the processing of the current decoding request and obtaining the next decoding request from the fourth request processing queue;

[0268] Add the current decoding request to the second intercepted queue and start the pause processing timer;

[0269] When the pause processing duration of the current decoding request is greater than the fourth preset duration, the current decoding request is added to the fourth request processing queue.

[0270] In some embodiments, the scheduling module is further configured to:

[0271] Determine the word unit output interval setting value;

[0272] Search the historical operation data of the server cluster, and when the word unit output interval is less than or equal to the word unit output interval setting value, obtain the batch size corresponding to each node in the decoding resource pool when processing the decoding request;

[0273] The maximum value of the batch sizes corresponding to each node when processing a decoding request is determined as a batch size setting value in the decoding resource pool.

[0274] In some embodiments, the server cluster further includes a hybrid resource pool, and the scheduling module is further configured to:

[0275] When the first real-time load exceeds the preset load, the prefilling requests processed by the nodes in the prefilling resource pool are scheduled to the hybrid resource pool, so that the nodes in the hybrid resource pool process the prefilling requests.

[0276] In some embodiments, the scheduling module is further configured to:

[0277] When the second real-time load exceeds the preset load, the decoding requests processed by the nodes in the decoding resource pool are scheduled to the hybrid resource pool, so that the nodes in the hybrid resource pool process the decoding requests.

[0278] In some embodiments, the scheduling module is further configured to:

[0279] Determine the processing priority of each inference request based on the request time of each received inference request; the inference request includes a pre-filling request scheduled from a pre-filling resource pool and / or a decoding request scheduled from a decoding resource pool;

[0280] Based on the processing priority of each inference request, each inference request is scheduled to a node in the hybrid resource pool.

[0281] In some embodiments, the scheduling module is further configured to:

[0282] Obtaining the third real-time load of the hybrid resource pool;

[0283] Based on the first real-time load, the second real-time load, and the third real-time load, the number of nodes in the pre-filled resource pool, the decoding resource pool, and the hybrid resource pool is adjusted.

[0284] In some embodiments, the scheduling module is further configured to:

[0285] During the server cluster initialization phase, the inference requests to be processed are determined;

[0286] Determine, based on a text sequence length of an inference request to be processed, a first load estimate of a prefilled resource pool and a second load estimate of a decoding resource pool;

[0287] Based on the first load estimate and the second load estimate, each node in the server cluster is allocated to a pre-filling resource pool and a decoding resource pool.

[0288] In some embodiments, the scheduling module is further configured to:

[0289] Receive the current inference request;

[0290] When the current inference request is a pre-filling request, the task queue length of each node in the pre-filling resource pool is obtained;

[0291] Schedule the current inference request to the node corresponding to the minimum task queue length.

[0292] In some embodiments, the scheduling module is further configured to:

[0293] Receive the current inference request;

[0294] When the current inference request is a decoding request, the task queue length of each node in the decoding resource pool is obtained;

[0295] Schedule the current inference request to the node corresponding to the minimum task queue length.

[0296] In some embodiments, the scheduling module is further configured to:

[0297] When each prefilling request processed by a node in the prefilling resource pool is completed, the key value cache and the first word element corresponding to each prefilling request are transmitted to the node corresponding to the decoding request corresponding to each prefilling request in the decoding resource pool.

[0298] Figure 4 It is a schematic diagram of the structure of the server cluster provided by this application, such as Figure 4 As shown, the server cluster 400 includes multiple nodes 410 and a model reasoning scheduling device 420.

[0299] Each node is connected to a model reasoning scheduling device for executing model reasoning; the model reasoning scheduling device is used to execute the model reasoning scheduling method in the above embodiment.

[0300] The server cluster provided in the embodiment of the present application is provided with a model inference scheduling device, which divides the nodes into a pre-filling resource pool and a decoding resource pool, and obtains a first real-time load of the pre-filling resource pool and a second real-time load of the decoding resource pool; when the first real-time load and the second real-time load both exceed the preset load, the pre-filling request processed by the node in the pre-filling resource pool is scheduled to the decoding resource pool, so that the nodes in the decoding resource pool process the pre-filling request and the decoding request at the same time, so that the server cluster can make full use of the graphics processor of each node in the decoding resource pool when executing the model inference task, avoid wasting computing power resources, improve the utilization rate of hardware resources in the server cluster, and improve the response speed of large model inference.

[0301] Figure 5 This is the model reasoning architecture diagram of the server cluster provided by this application, such as Figure 5 As shown, the server cluster includes a prefill resource pool 510, a hybrid resource pool 520 and a decoding resource pool 530. Each resource pool includes multiple nodes. The server cluster also has three schedulers, namely a machine-level prefill scheduler 540 (machine-level Prefill scheduler, MLPS), a machine-level decode scheduler 550 (machine-level Decode scheduler, MLDS) and a cluster-level scheduler 560 (cluster-level scheduler, CLS).

[0302] The cluster-level scheduler runs in the cluster and is responsible for resource pool management and routing incoming inference requests.

[0303] The machine-level pre-fill scheduler runs in each node in the pre-fill resource pool and is responsible for monitoring device resource utilization, maintaining the request queue to be processed, determining the batch for each iteration, and reporting relevant status to the cluster-level scheduler. The request queue includes the first request processing queue, the second request processing queue, the third request processing queue, and the first preempted queue.

[0304] The machine-level decoding scheduler runs in each node in the decoding resource pool and is responsible for monitoring device resource utilization, maintaining the request queue to be processed, determining the batch for each iteration, and reporting relevant status to the cluster-level scheduler. The request queue includes the fourth request processing queue, the fifth request processing queue, and the second preempted queue.

[0305] Figure 6 is a schematic diagram of the structure of the electronic device provided by this application, such as Figure 6 As shown, the electronic device may include: a processor (Processor) 610, a communication interface (Communications Interface) 620, a memory (Memory) 630 and a communication bus (Communications Bus) 640, wherein the processor 610, the communication interface 620, and the memory 630 communicate with each other through the communication bus 640. The processor 610 may call the logic command in the memory 630 to execute the method described in the above embodiment, for example:

[0306] Obtain a first real-time load of a pre-filling resource pool and a second real-time load of a decoding resource pool; when both the first real-time load and the second real-time load exceed a preset load, schedule the pre-filling requests processed by the nodes in the pre-filling resource pool to the decoding resource pool, so that the nodes in the decoding resource pool process the pre-filling requests and the decoding requests at the same time.

[0307] In addition, the logical commands in the above-mentioned memory can be implemented in the form of software functional units and can be stored in a computer-readable storage medium when sold or used as an independent product. Based on this understanding, the technical solution of the present application can be essentially or partly embodied in the form of a software product that contributes to the prior art. The computer software product is stored in a storage medium and includes several commands to enable a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc., which can store program code.

[0308] The processor in the electronic device provided in the embodiment of the present application can call the logic instructions in the memory to implement the above method. Its specific implementation method is consistent with the implementation method of the aforementioned method and can achieve the same beneficial effects, which will not be repeated here.

[0309] An embodiment of the present application also provides a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the methods provided in the above embodiments are implemented.

[0310] Its specific implementation is consistent with the aforementioned method implementation and can achieve the same beneficial effects, so it will not be repeated here.

[0311] An embodiment of the present application provides a computer program product, including a computer program, which implements the above method when executed by a processor.

[0312] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0313] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0314] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit it. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A model reasoning scheduling method, characterized in that: Applied to a server cluster, the server cluster includes a pre-filled resource pool and a decoding resource pool, the method includes: Acquire a first real-time load of the pre-filled resource pool and a second real-time load of the decoding resource pool; When both the first real-time load and the second real-time load exceed a preset load, scheduling the prefilling request processed by the prefilling resource pool to the decoding resource pool, so that the decoding resource pool processes the prefilling request and the decoding request at the same time; The step of scheduling the prefilling request processed by the prefilling resource pool to the decoding resource pool includes: determining pre-fill requests to be scheduled; Segmenting the pre-filling request to be scheduled to obtain a plurality of pre-filling request segments; splicing the pre-filled request fragment with the decoded request to obtain a hybrid inference request; The hybrid inference request is dispatched to a node in the decoding resource pool.

2. The model reasoning scheduling method according to claim 1, characterized in that: The method further comprises: Establishing a first request processing queue, a second request processing queue and a third request processing queue in the pre-filled resource pool; Determining the processing priority of each request processing queue based on the text sequence length of each request processing queue; the processing priority is in inverse proportion to the text sequence length; Adding each pre-filling request to each request processing queue based on the text sequence length of each received pre-filling request; Get the current pre-filled request from each request processing queue in turn; The current pre-fill request is dispatched to a node in the pre-fill resource pool.

3. The model reasoning scheduling method according to claim 2, characterized in that: The method further comprises: Establishing a first preempted queue in the pre-filled resource pool; Obtaining the processing time of the current pre-fill request; When the processing time of the current pre-filling request is longer than the first preset time, suspending the processing of the current pre-filling request, and obtaining the next pre-filling request from the first request processing queue, the second request processing queue, and the third request processing queue; Add the current pre-filled request to the first preempted queue and start pausing processing timing; In a case where the suspension processing duration of the current pre-filling request is greater than a second preset duration, determining a target request processing queue corresponding to the current pre-filling request based on a remaining text sequence length of the current pre-filling request; The current pre-filled request is added to the target request processing queue.

4. The model reasoning scheduling method according to claim 2, characterized in that: The method further comprises: Determine the first word delay time setting value; Searching the historical operation data of the server cluster, and obtaining the batch size corresponding to each node in the pre-filling resource pool when processing the pre-filling request when the first word delay time is less than or equal to the first word delay time setting value; The maximum value of the batch sizes corresponding to each node when processing the pre-filling request is determined as the batch size setting value in the pre-filling resource pool.

5. The model reasoning scheduling method according to claim 1, characterized in that: The method further comprises: Establishing a fourth request processing queue and a fifth request processing queue in the decoding resource pool; adding each decoding request to the fourth request processing queue based on the request time of each received decoding request; Based on the request time of each pre-filling request scheduled from the pre-filling resource pool, each decoding request is added to the fifth request processing queue.

6. The model reasoning scheduling method according to claim 5, characterized in that: The method further comprises: Establishing a second intercepted queue in the decoding resource pool; Get the processing time of the current decoding request; When the processing time of the current decoding request is longer than the third preset time, suspending the processing of the current decoding request and obtaining the next decoding request from the fourth request processing queue; Add the current decoding request to the second intercepted queue and start pausing processing timing; When the pause processing time of the current decoding request is greater than a fourth preset time, the current decoding request is added to the fourth request processing queue.

7. The model reasoning scheduling method according to claim 5, characterized in that: The method further comprises: Determine the word unit output interval setting value; Searching the historical operation data of the server cluster, and obtaining the batch size corresponding to each node in the decoding resource pool when processing a decoding request when the word unit output interval is less than or equal to the word unit output interval setting value; The maximum value of the batch sizes corresponding to each node when processing a decoding request is determined as a batch size setting value in the decoding resource pool.

8. The model reasoning scheduling method according to claim 1, characterized in that: The server cluster further includes a hybrid resource pool, and the method further includes: When the first real-time load exceeds a preset load, the prefilling requests processed by the nodes in the prefilling resource pool are scheduled to the hybrid resource pool so that the nodes in the hybrid resource pool process the prefilling requests.

9. The model reasoning scheduling method according to claim 8, characterized in that: The method further comprises: When the second real-time load exceeds a preset load, the decoding requests processed by the nodes in the decoding resource pool are scheduled to the hybrid resource pool, so that the nodes in the hybrid resource pool process the decoding requests.

10. The model reasoning scheduling method according to claim 9, characterized in that: The method further comprises: Determining a processing priority of each inference request based on a request time of each received inference request; the inference request includes a pre-filling request scheduled from the pre-filling resource pool and / or a decoding request scheduled from the decoding resource pool; Based on the processing priority of each inference request, each inference request is scheduled to a node in the hybrid resource pool.

11. The model reasoning scheduling method according to claim 8, characterized in that: The method further comprises: Acquire a third real-time load of the hybrid resource pool; Based on the first real-time load, the second real-time load, and the third real-time load, the number of nodes in the pre-filled resource pool, the decoding resource pool, and the hybrid resource pool are adjusted.

12. The model reasoning scheduling method according to claim 1, characterized in that: The method further comprises: In the server cluster initialization phase, determining an inference request to be processed; Determining a first load estimate of the pre-filled resource pool and a second load estimate of the decoding resource pool based on a text sequence length of the to-be-processed inference request; Based on the first load estimate and the second load estimate, each node in the server cluster is allocated to the pre-filled resource pool and the decoding resource pool.

13. The model reasoning scheduling method according to claim 1, characterized in that: The method further comprises: Receive the current inference request; In a case where the current inference request is a pre-filling request, obtaining the task queue length of each node in the pre-filling resource pool; The current reasoning request is dispatched to a node corresponding to a minimum task queue length.

14. The model reasoning scheduling method according to claim 1, characterized in that: The method further comprises: Receive the current inference request; When the current inference request is a decoding request, obtaining the task queue length of each node in the decoding resource pool; The current reasoning request is dispatched to a node corresponding to a minimum task queue length.

15. The model reasoning scheduling method according to claim 1, characterized in that: The method further comprises: When each prefilling request processed by a node in the prefilling resource pool is completed, the key value cache and the first word element corresponding to each prefilling request are transmitted to the node corresponding to the decoding request corresponding to each prefilling request in the decoding resource pool.

16. A model reasoning scheduling device, characterized in that: Applied to a server cluster, the server cluster includes a pre-filled resource pool and a decoding resource pool, and the device includes: An acquisition module, used to acquire a first real-time load of the pre-filled resource pool and a second real-time load of the decoding resource pool; a scheduling module, configured to schedule the prefilling request processed by the prefilling resource pool to the decoding resource pool when both the first real-time load and the second real-time load exceed a preset load, so that the decoding resource pool processes the prefilling request and the decoding request at the same time; The scheduling module is used to: determining pre-fill requests to be scheduled; Segmenting the pre-filling request to be scheduled to obtain a plurality of pre-filling request segments; splicing the pre-filled request fragment with the decoded request to obtain a hybrid inference request; The hybrid inference request is dispatched to a node in the decoding resource pool.

17. A server cluster, characterized in that: It includes multiple nodes and a model reasoning scheduling device; The node is connected to the model reasoning scheduling device and is used to perform model reasoning; The model reasoning scheduling device is used to execute the model reasoning scheduling method described in any one of claims 1 to 15.

18. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the model reasoning scheduling method as described in any one of claims 1 to 15 is implemented.

19. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the model reasoning scheduling method as described in any one of claims 1 to 15 is implemented.

Images