Load-aware scheduling method of inference system and inference system

Abstract

One or more embodiments of the invention provide a load-aware scheduling method of an inference system and the inference system. The method is applied to the inference system comprising a global scheduler and an inference engine. The inference engine comprises a Prefill engine and a Decode engine; the Prefill engine and the Decode engine respectively comprise at least one calculation instance; computing resources of the computing instances comprise GPUs carried on the computing nodes where the computing instances are located; the method comprises the following steps: a global scheduler obtains a target reasoning request to be executed, determines a first calculation instance in a Prefill engine of which the GPU load meets a first preset condition based on maintained GPU load information, and sends the target reasoning request to the first calculation instance to execute reasoning calculation in a Prefill stage; and a first local scheduler corresponding to the first calculation instance in the Prefill engine determines a second calculation instance in the Decode engine with the GPU load meeting a second preset condition based on the maintained GPU load information, and sends an inference calculation result of the Prefill stage to the second calculation instance to execute inference calculation of the Decode stage.

Description

Technical Field

[0001] One or more embodiments of the present application relate to the field of artificial intelligence technology, and in particular, to a load-aware scheduling method for an inference system and an inference system. Background Art

[0002] An inference system is a computer program that uses logical rules and known facts to draw new conclusions or decisions. An inference system is an important part of the field of artificial intelligence, mainly used to simulate the human decision-making process. It draws conclusions based on a set of defined knowledge bases and inference engines. An inference system can execute the inference request it obtains and output the corresponding inference results.

[0003] A typical reasoning system usually consists of the following parts: Knowledge Base, Inference Engine, User Interface, and Explanation Facility. The knowledge base includes all the facts and rules known to the storage system. These facts can be about the state of the world, object attributes, etc., while the rules are logical expressions that describe how to draw new conclusions from known facts. The inference engine is the core component of the reasoning system. It is responsible for performing logical operations in the reasoning process, that is, drawing new conclusions or decisions from a given knowledge base; the inference engine uses a series of rules and known facts to derive new knowledge to help the system solve problems or make decisions. The user interface allows users to interact with the system, input queries, or observe the results of the reasoning process. The explanation mechanism is used to explain how the system draws specific conclusions, which is very important for transparency and trust.

[0004] Inference engines are usually deployed on computing clusters when large-scale data, high-concurrency requests, or high-performance computing are required. By deploying inference engines on computing clusters, higher computing power, better fault tolerance, and more flexible resource management can be achieved. However, this introduces the issue of how to schedule inference requests on cluster-level inference engines. Summary of the invention

[0005] One or more embodiments of the present application provide the following technical solutions:

[0006] The present application provides a load-aware scheduling method for an inference system, which is applied to an inference system; the inference system includes a global scheduler and an inference engine; the inference engine includes a Prefill engine and a Decode engine; the Prefill engine is deployed on at least one first computing instance, and the Decode engine is deployed on at least one second computing instance; the computing resources of the first and second computing instances include GPUs mounted on the computing nodes where they are located; the global scheduler maintains GPU load information of each first computing instance that is dynamically updated; the inference system also includes a first local scheduler; the first local scheduler maintains GPU load information of each second computing instance that is dynamically updated;

[0007] The method comprises:

[0008] The global scheduler obtains a target inference request to be executed;

[0009] The global scheduler determines a first computing instance whose GPU load satisfies a first preset condition based on the GPU load information of each first computing instance maintained, and sends the target inference request to the first computing instance, so that the first computing instance performs the inference calculation of the Prefill stage for the target inference request;

[0010] The first local scheduler determines a second computing instance whose GPU load meets a second preset condition based on the GPU load information of each computing instance in the maintained second computing instance, and sends the inference calculation result of the Prefill stage to the second computing instance, so that the second computing instance performs the inference calculation of the Decode stage based on the inference calculation result of the Prefill stage.

[0011] The present application also provides an inference system, which includes a global scheduler and an inference engine; the inference engine includes a Prefill engine and a Decode engine; the Prefill engine is deployed on at least one first computing instance, and the Decode engine is deployed on at least one second computing instance; the computing resources of the first and second computing instances include GPUs mounted on the computing nodes where they are located; the global scheduler maintains dynamically updated GPU load information of each first computing instance; the inference system also includes a first local scheduler; the first local scheduler maintains dynamically updated GPU load information of each second computing instance;

[0012] The global scheduler is used to:

[0013] Get the target inference request to be executed;

[0014] Based on the maintained GPU load information of each first computing instance, determine a first computing instance whose GPU load meets a first preset condition, and send the target inference request to the first computing instance, so that the first computing instance performs inference calculation of the prefill stage for the target inference request;

[0015] The first local scheduler is used for:

[0016] Based on the GPU load information of each computing instance in the maintained second computing instance, determine the second computing instance whose GPU load meets the second preset condition, and send the inference calculation result of the prefill stage to the second computing instance, so that the second computing instance performs the inference calculation of the Decode stage based on the inference calculation result of the prefill stage.

[0017] The present application also provides an electronic device, including:

[0018] processor;

[0019] a memory for storing processor-executable instructions;

[0020] The processor implements the steps of any of the above methods by running the executable instructions.

[0021] The present application also provides a computer-readable storage medium having computer instructions stored thereon, which implement the steps of any of the above methods when executed by a processor.

[0022] In the above technical solution, the inference engine in the inference system may include a Prefill engine and a Decode engine, the Prefill engine may include at least one first computing instance, the Decode engine may include at least one second computing instance, and the computing resources of each computing instance may include a GPU mounted on the computing node where it is located; the global scheduler in the inference system may maintain the GPU load information of each first computing instance that is dynamically updated, and may, when obtaining the target inference request to be executed, first determine the first computing instance whose GPU load meets the first preset condition based on the GPU load information of each first computing instance that is maintained, and then send the target inference request to the first computing instance to perform the inference calculation of the Prefill stage, so as to realize the scheduling of the Prefill stage in the inference process; the first local scheduler corresponding to the first computing instance in the inference system may maintain the GPU load information of each second computing instance that is dynamically updated, and may, when the first computing instance calculates the inference calculation result of the Prefill stage, first determine the second computing instance whose GPU load meets the second preset condition based on the GPU load information of each second computing instance that is maintained, and then send the inference calculation result of the Prefill stage to the second computing instance to perform the inference calculation of the Decode stage, so as to realize the scheduling of the Decode stage in the inference process.

[0023] By adopting the above method, on the one hand, by adding a global scheduler to the inference system, the global scheduler can schedule the inference request to the computing instance with appropriate GPU load in the inference engine for execution, thereby realizing load-aware scheduling of inference requests on the cluster-level inference engine, thereby optimizing the efficiency of GPU resource utilization, reducing the waiting time and processing delay of inference request execution, and improving the overall throughput of the inference system. On the other hand, according to the different stages of the inference process, the inference request is load-awarely scheduled in stages on the cluster-level inference engine, which can further improve the scheduling effect. In addition, by scheduling the inference request to the inference engine through the global scheduler instead of directly scheduling the inference request in the inference engine, the scheduling method can be decoupled from the specific inference engine, which has certain versatility and scalability. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] The following is a description of the accompanying drawings required for describing the exemplary embodiments, wherein:

[0025] Figure 1 It is a schematic diagram of an inference system shown in an exemplary embodiment of the present application.

[0026] Figure 2 It is a schematic diagram of another reasoning system shown in an exemplary embodiment of the present application.

[0027] Figure 3 It is a flowchart of a load-aware scheduling method for an inference system shown in an exemplary embodiment of the present application.

[0028] Figure 4 It is a structural schematic diagram of a device shown in an exemplary embodiment of the present application.

[0029] Figure 5 It is a block diagram of an inference system shown in an exemplary embodiment of the present application. DETAILED DESCRIPTION

[0030] Here, exemplary embodiments will be described in detail, and examples thereof are shown in the accompanying drawings. When the following description relates to the accompanying drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with one or more embodiments of the present application. On the contrary, they are only examples consistent with some aspects of one or more embodiments of the present application.

[0031] It should be noted that in other embodiments, the steps of the corresponding method are not necessarily performed in the order shown and described in this application. In some other embodiments, the steps included in the method may be more or less than those described in this application. In addition, a single step described in this application may be decomposed into multiple steps for description in other embodiments; and multiple steps described in this application may also be combined into a single step for description in other embodiments.

[0032] In this application, the inference engine in the inference system can be deployed on a computing cluster to meet the needs of processing large-scale data, high-concurrency requests or high-performance computing, thereby achieving higher computing power, better fault tolerance and more flexible resource management. In this type of inference system, load-aware scheduling can be used to schedule inference requests on the cluster-level inference engine.

[0033] Load-aware scheduling is a strategy for allocating tasks in a computing system based on the current load of each node or resource. This scheduling method aims to optimize resource usage efficiency, reduce waiting time and processing delays, and improve the overall throughput of the system.

[0034] Load-aware scheduling is usually used in distributed systems, cloud computing platforms, data centers and other scenarios. Its purpose is to dynamically balance the workload of each node and avoid the situation where some nodes are overloaded while other nodes are idle. By monitoring the load of each node in real time, the load-aware scheduler can make more reasonable task allocation decisions.

[0035] In general, the inference engine can use the CPU as a computing resource, where the CPU can be the CPU installed on the device used to deploy the inference engine. The CPU load indicators usually select the CPU usage and memory usage. That is, for the inference engine deployed on the computing cluster, by calculating the CPU usage and memory usage to represent the CPU load, load-aware scheduling of inference requests can be achieved on such inference engines.

[0036] With the development of artificial intelligence technology, the application of reasoning systems based on large models (for example, large language models Large Language Model) is becoming more and more widespread.

[0037] Large models refer to machine learning models with a large number of parameters, such as various variants of the Transformer architecture, including but not limited to natural language processing models such as GPT and BERT. These models achieve powerful representation learning capabilities through large amounts of training data and complex architectures.

[0038] In a large model-based reasoning system, the large model can be viewed as a large knowledge base that contains information learned from a large amount of data. The large model learns a large number of patterns and features during training, which represent complex relationships in the data. Therefore, to some extent, the information stored inside the large model can be viewed as a form of knowledge.

[0039] The inference engine designed for large models is responsible for specific inference work based on large models, that is, it can manage the loading of large models, execute inference operations on large models, and manage interactions with hardware (for example, CPU, GPU or other accelerators). The inference engine can also include optimization algorithms to improve the speed and efficiency of inference.

[0040] In practical applications, the large model can be deployed separately from the reasoning system, or it can be integrated into the reasoning system and the reasoning engine in the system can be used to call the large model to efficiently perform reasoning tasks.

[0041] Since large models are artificial intelligence applications that require a lot of computing resources, the inference engine usually uses GPU as a computing resource when calling large models to perform inference tasks.

[0042] GPUs are designed for parallel processing and can process multiple data points at the same time. This is particularly useful for deep learning models, as they often need to perform the same operation on large amounts of data. Modern AI models, especially neural network models, require a large number of floating-point operations. GPUs generally perform better than CPUs on floating-point operations, especially when processing large-scale matrix operations, which are common operations in deep learning. GPUs generally have higher memory bandwidth than CPUs, which means they are able to read and write data from memory faster. This is very important for applications that need to frequently access large amounts of data. Many GPU vendors (such as NVIDIA) have optimized their hardware and software stacks for machine learning tasks. For example, they provide hardware specifically designed to accelerate tensor operations, as well as programming models like CUDA to take full advantage of these hardware features. GPUs can increase inference speed. For applications deployed at large scale, using GPUs can reduce latency and increase system responsiveness.

[0043] There are many choices for GPU load indicators, and commonly used indicators include SM Clock (Streaming Multiprocessor Clock), SM Activity (Streaming Multiprocessor Activity), Memory Utilization (Memory Usage), etc. Among them, SM Clock refers to the clock frequency of the streaming multiprocessor on the GPU; the streaming multiprocessor is the computing unit on the GPU that is responsible for performing parallel computing tasks. A higher SM Clock frequency means that the GPU's computing units run faster, which generally leads to higher computing performance, but may also increase power consumption and heat. SM Activity refers to the degree of activity of the streaming multiprocessor on the GPU, that is, what proportion of time these computing units are actually performing tasks over a period of time. If SM Activity is high, it means that the GPU's computing units are actively working most of the time; conversely, if it is low, it may mean that there is a computing bottleneck or a long waiting time. Memory Utilization refers to the usage of the GPU video memory, usually indicating the proportion of the video memory currently in use to the total video memory. Higher Memory Utilization means that more video memory is occupied, which may cause slower data exchange or affect performance due to insufficient video memory. When the video memory is close to full, it may trigger data overflow to the host memory, which will further reduce performance.

[0044] For inference engines deployed on computing clusters, load-aware scheduling of inference requests can be achieved on such inference engines by calculating GPU load indicators to represent GPU load conditions.

[0045] In the technical solution provided by one or more embodiments of the present application, the inference engine in the inference system may include a Prefill engine and a Decode engine, the Prefill engine may include at least one first computing instance, the Decode engine may include at least one second computing instance, and the computing resources of each computing instance may include a GPU mounted on the computing node where it is located; the global scheduler in the inference system may maintain the GPU load information of each first computing instance that is dynamically updated, and may, when a target inference request to be executed is obtained, first determine the first computing instance whose GPU load meets the first preset condition based on the GPU load information of each first computing instance that is maintained, and then send the target inference request to the first computing instance to perform the inference calculation of the Prefill stage, thereby realizing the scheduling of the Prefill stage in the inference process; the first local scheduler corresponding to the first computing instance in the inference system may maintain the GPU load information of each second computing instance that is dynamically updated, and may, when the first computing instance calculates the inference calculation result of the Prefill stage, first determine the second computing instance whose GPU load meets the second preset condition based on the GPU load information of each second computing instance that is maintained, and then send the inference calculation result of the Prefill stage to the second computing instance to perform the inference calculation of the Decode stage, thereby realizing the scheduling of the Decode stage in the inference process.

[0046] By adopting the above method, on the one hand, by adding a global scheduler to the inference system, the global scheduler can schedule the inference request to the computing instance with appropriate GPU load in the inference engine for execution, thereby realizing load-aware scheduling of inference requests on the cluster-level inference engine, thereby optimizing the efficiency of GPU resource utilization, reducing the waiting time and processing delay of inference request execution, and improving the overall throughput of the inference system. On the other hand, according to the different stages of the inference process, the inference request is load-awarely scheduled in stages on the cluster-level inference engine, which can further improve the scheduling effect. In addition, by scheduling the inference request to the inference engine through the global scheduler instead of directly scheduling the inference request in the inference engine, the scheduling method can be decoupled from the specific inference engine, which has certain versatility and scalability.

[0047] Please refer to Figure 1 and Figure 2 , Figure 1 and Figure 2 They are respectively schematic diagrams of an inference system shown by an exemplary embodiment of the present application.

[0048] like Figure 1As shown, the above reasoning system may include a global scheduler and a reasoning engine. The reasoning engine may be deployed on a computing cluster composed of at least one computing node.

[0049] A computing cluster is a system consisting of multiple computers or servers connected through a network to work together. These nodes work together to complete computing tasks, providing stronger computing power and higher availability than a single computer. Computing nodes are one of the core components of a computing cluster, usually referring to computers or servers used to perform computing-intensive tasks. Computing nodes can undertake computing tasks, and the GPUs installed on them can be used as computing resources. Among them, computers or servers can be physical or virtual local computers or local servers, or physical or virtual cloud computers or cloud servers.

[0050] In practical applications, the global scheduler may be deployed on the head node in the computing cluster, where the head node generally refers to the node responsible for the management and scheduling of the computing cluster. It may also be deployed on other computing devices outside the computing cluster, and this application does not impose any restrictions on this.

[0051] In the above reasoning system, the above global scheduler and the above reasoning engine can communicate with each other, for example, the two can directly communicate with each other through HTTP (HyperText Transfer Protocol).

[0052] Specifically, the global scheduler can communicate with the applications and components deployed on each computing node in the computing cluster. In this case, the global scheduler can obtain the inference request from the request queue, and select a computing node with a suitable GPU load according to the load of the GPU on each computing node, and schedule the obtained inference request to the computing node, so as to use the GPU on the computing node to execute the inference request.

[0053] It should be noted that the reasoning process of the reasoning system may include multiple stages in some cases. For example, for a reasoning system based on a large model, the reasoning process of the large model (especially the generative model in natural language processing) may include a Prefill stage and a Decode stage.

[0054] In the reasoning process of a large model, the Prefill stage and the Decode stage refer to different steps in generating text.

[0055] In practical applications, the two stages of Prefill and Decode are closely linked. The Prefill stage provides the initial text basis, while the Decode stage is responsible for gradually expanding on this basis to generate a complete text output. These two stages together determine the quality and coherence of the final generated text.

[0056] Based on the above, if Figure 2 As shown, the computing nodes in the computing cluster can be divided into two categories, the first category is the computing nodes used to perform the inference calculations in the Prefill stage, and the second category is the computing nodes used to perform the inference calculations in the Decode stage. In this case, the inference engine can be divided into a Prefill engine and a Decode engine according to the inference calculations performed; wherein the Prefill engine is the inference engine deployed on the first category of computing nodes, and the Decode engine is the inference engine deployed on the second category of computing nodes.

[0057] In practical applications, the resources of the computing nodes in the above computing cluster (for example, computing resources, storage resources, network resources, etc.) can also be allocated as needed, so that part of these resources are used to perform inference calculations in the Prefill stage, and the inference engine using these resources can be called a Prefill engine; another part of these resources are used to perform inference calculations in the Decode stage, and the inference engine using these resources can be called a Decode engine.

[0058] That is, the Prefill engine and the Decode engine can be deployed on different computing nodes in the computing cluster, or they can be deployed on the same computing node in the computing cluster, but using different parts of the resources on the computing node. This application does not impose any restrictions on this.

[0059] Furthermore, the above-mentioned Prefill engine can be deployed on at least one computing instance (which may be referred to as the first computing instance). Similarly, the above-mentioned Decode engine can also be deployed on at least one computing instance (which may be referred to as the second computing instance). Among them, the computing instance refers to a dedicated resource allocated on demand through virtualization technology in a local data center or cloud computing platform. The resources of these computing instances may include computing resources, storage resources, network resources, etc. (for example: GPU, GPU memory, storage, network interface, etc.), so that each computing instance can use these resources to perform specific computing tasks. Computing instances can be created, started, stopped or deleted according to actual needs, and their resources can also be adjusted according to demand.

[0060] Specifically, at least one first computing instance can be deployed on the computing node or part of the resources used to perform the inference calculation of the Prefill stage, and these first computing instances together constitute the above-mentioned Prefill engine. Similarly, at least one second computing instance can be deployed on the computing node or part of the resources used to perform the inference calculation of the Decode stage, and these second computing instances together constitute the above-mentioned Decode engine.

[0061] In an inference engine for large models, the computing resource of a computing instance may be a GPU of the computing node where it is located.

[0062] In actual applications, a computing node can be equipped with multiple GPUs, and a GPU can be assigned to a computing instance as a computing resource of the computing instance. Alternatively, a computing node can be equipped with only one GPU, and the computing resources assigned to a computing instance can be part of the computing unit, memory, etc. of the GPU.

[0063] In the inference engine for large models, each computing instance can call the large model to perform specific inference work, that is, the large model can be loaded into each computing instance, and each computing instance performs the inference operation of the large model. Specifically, the large model can be read from a local or cloud storage medium (e.g., local disk, cloud storage, etc.) to the GPU memory of each computing instance to perform inference based on the large model on the GPU computing resources of each computing instance.

[0064] In actual applications, the Model Runtime framework can be installed in the computing instance, and after the large model is read into the computing instance, the model service is configured according to the Model Runtime framework and the large model, so that the computing instance can be used as an independently run model service instance. Among them, the model runtime refers to the environment and framework used to perform model reasoning after the model is deployed. This environment usually includes a series of steps such as model loading, input data processing, model execution, and output result processing, covering the entire life cycle of the model from loading to execution and then to unloading.

[0065] By deploying computing instances on computing nodes, on the one hand, the resources of computing nodes can be fully utilized to accelerate the execution of computing tasks, thereby improving the computing power of the inference engine; on the other hand, different computing instances are independent of each other, which can ensure the stability and security of computing tasks, and can achieve resource isolation and management, which is conducive to the effective use of resources and avoid resource requisition; on the third hand, for computing tasks that need to be processed in parallel, parallel computing can be used to speed up the completion time of computing tasks; on the fourth hand, elastic scaling can be achieved. When a large number of computing tasks need to be processed, computing instances can be added, and when the number of computing tasks that need to be processed decreases, some computing instances can be terminated to release some resources and realize flexible allocation of resources.

[0066] In order to facilitate the load-aware scheduling of the inference request in the Prefill phase, the global scheduler can maintain the GPU load information of each first computing instance, where the GPU load information can indicate the GPU load situation. In this case, the global scheduler can obtain the inference request from the request queue, and select the first computing instance with the appropriate GPU load situation according to the maintained GPU load information of each first computing instance, and schedule the obtained inference request to the first computing instance, so as to use the GPU as the computing resource of the first computing instance to perform the inference calculation in the Prefill phase for the inference request.

[0067] It should be noted that the GPU load information of each first computing instance maintained by the above-mentioned global scheduler can be dynamically updated. For example, after a certain inference request is scheduled to a certain first computing instance and the first computing instance starts to execute the inference request, GPU load information can be generated according to the current GPU load of the first computing instance, and the generated GPU load information is reported to the global scheduler, and the global scheduler updates the GPU load information of the first computing instance previously maintained based on the GPU load information.

[0068] For the above-mentioned Prefill engine, in order to facilitate the maintenance, management and scheduling of the first computing instance corresponding to the Prefill engine, a local scheduler (Local Scheduler) can also be deployed on each computing node or each part of the resources where the Prefill engine is located, which is called the first local scheduler.

[0069] It should be noted that each first local scheduler is specifically used to maintain, manage and schedule the first computing instance on the computing node or part of the resources where the first local scheduler is located.

[0070] The above-mentioned global scheduler can communicate with each first local scheduler. In this case, the global scheduler can obtain the inference request from the request queue, and select the first computing instance with the appropriate GPU load according to the GPU load information of each first computing instance maintained, and send the obtained inference request to the first local scheduler deployed on the computing node where the first computing instance is located, and the first local scheduler further schedules the inference request to the first computing instance, so as to use the GPU as the computing resource of the first computing instance to perform the inference calculation of the Prefill stage for the inference request.

[0071] Similarly, for the above-mentioned Decode engine instance, in order to facilitate the maintenance, management and scheduling of the second computing instance corresponding to the Decode engine, a local scheduler, called the second local scheduler, can also be deployed on each computing node or each part of the resources where the Decode engine is located.

[0072] It should be noted that each second local scheduler is specifically used to maintain, manage and schedule the second computing instance on the computing node or part of the resources where the second local scheduler is located.

[0073] The first local scheduler can communicate with each second local scheduler. In this case, the first local scheduler can select a second computing instance with a suitable GPU load condition based on the GPU load information of each second computing instance maintained, and send the inference calculation result of the Prefill stage obtained by the corresponding first computing instance to the second local scheduler deployed on the computing node where the second computing instance is located, and the second local scheduler further schedules the inference calculation result of the Prefill stage to the second computing instance, so as to use the GPU as the computing resource of the second computing instance to perform inference calculation of the Decode stage based on the inference calculation result of the Prefill stage.

[0074] In practical applications, in order to ensure the reliability and correctness of the scheduling of inference requests or inference calculation results in the Prefill stage, each local scheduler can maintain a sequential queue (Sequential Queue), which can be a first-in-first-out (FIFO) sequence of inference requests or inference calculation results in the Prefill stage, that is, the inference requests or inference calculation results in the Prefill stage in the sequential queue are processed in the order in which they enter the sequential queue.

[0075] In addition, for the first local scheduler and each first computing instance deployed on the same computing node, the first local scheduler can monitor the GPU load of the first computing instance, generate corresponding GPU load information, and report the generated GPU load information to the above-mentioned global scheduler, so that the global scheduler can maintain the dynamically updated GPU load information of each first computing instance.

[0076] Similarly, for the second local scheduler and each second computing instance deployed on the same computing node, the second local scheduler can monitor the GPU load of the second computing instance, generate corresponding GPU load information, and report the generated GPU load information to each first local scheduler, so that each first local scheduler can maintain the dynamically updated GPU load information of each second computing instance.

[0077] Furthermore, in actual applications, the above-mentioned global scheduler may include two components, namely a request routing (Request Router) component responsible for routing inference requests, and a load management (Load Manager) component responsible for obtaining and maintaining GPU load information of all first computing instances.

[0078] For each first local scheduler, the first local scheduler may include three components, namely, a request routing component responsible for routing the inference calculation results of the Prefill stage obtained by the corresponding first computing instance, a load management component responsible for calculating the GPU load information of the corresponding first computing instance, obtaining and maintaining the GPU load information of the corresponding first computing instance, and a monitor component responsible for monitoring the operation-related information of the corresponding first computing instance (for example: GPU memory utilization, GPU memory bandwidth utilization, the number of inference requests waiting in the sequential queue and the number of running inference requests, etc.).

[0079] For each second local scheduler, the second local scheduler may include two components, namely a load management component responsible for calculating the GPU load information of the corresponding second computing instance, and a monitor component responsible for monitoring the operation-related information of the corresponding second computing instance.

[0080] For each computing instance, the computing instance may contain two components, namely, a cache engine component for caching the inference results to avoid repeated calculations, and an executor component responsible for calling the large model to execute the inference task corresponding to the inference request.

[0081] Please Figure 1 and Figure 2Based on the reference Figure 3 , Figure 3 It is a flowchart of a load-aware scheduling method for an inference system shown in an exemplary embodiment of the present application.

[0082] In this embodiment, the load-aware scheduling method of the above-mentioned inference system can be applied to Figure 1 and Figure 2 The global scheduler shown.

[0083] In some embodiments, the above-mentioned reasoning system may specifically be a reasoning system based on a large model. Accordingly, the reasoning process of the reasoning system may include two stages, wherein the Prefill stage is the Prefill stage in the reasoning process of the large model, and the Decode stage is the Decode stage in the reasoning process of the large model.

[0084] like Figure 3 As shown, the load-aware scheduling method of the above reasoning system may include the following steps:

[0085] Step 302: The global scheduler obtains a target inference request to be executed.

[0086] In this embodiment, the global scheduler may first obtain an inference request to be executed (which may be referred to as a target inference request). For example, the global scheduler may sequentially obtain the inference request from a request queue for temporarily storing all inference requests to be executed as the target inference request.

[0087] Step 304: The global scheduler determines a first computing instance whose GPU load meets a first preset condition based on the GPU load information maintained for each first computing instance, and sends the target inference request to the first computing instance so that the first computing instance performs the inference calculation of the Prefill stage for the target inference request.

[0088] In this embodiment, as described above, the global scheduler can maintain GPU load information of each first computing instance, wherein the GPU load information can indicate the load condition of the GPU. In this case, the global scheduler can determine the first computing instance (which can be referred to as the first target computing instance) whose GPU load meets the first preset condition based on the maintained GPU load information of each first computing instance.

[0089] In some embodiments, the first computing instance whose GPU load satisfies the first preset condition may be the first computing instance whose CPU load is the smallest.

[0090] When the global scheduler determines the first target computing instance, it can send the target inference request to the first target computing instance through the communication connection with the inference engine, so that the first target computing instance can perform the inference calculation of the Prefill stage for the target inference request. That is, the global scheduler can schedule the inference request to be executed to the first computing instance whose GPU load meets the first preset condition to perform the inference calculation of the Prefill stage, thereby realizing the load-aware scheduling of the inference request of the Prefill stage on the cluster-level inference engine.

[0091] Step 306: The first local scheduler determines a second computing instance whose GPU load meets a second preset condition based on the GPU load information of each computing instance in the maintained second computing instance, and sends the inference calculation result of the Prefill stage to the second computing instance, so that the second computing instance performs the inference calculation of the Decode stage based on the inference calculation result of the Prefill stage.

[0092] In this embodiment, as described above, each first local scheduler can maintain GPU load information of each second computing instance, wherein the GPU load information can indicate the load condition of the GPU. In this case, the first local scheduler deployed on the computing node where the first target computing instance is located can determine the second computing instance (which can be referred to as the second target computing instance) whose GPU load meets the second preset condition based on the GPU load information of each second computing instance maintained.

[0093] In some embodiments, the second computing instance whose GPU load satisfies the second preset condition may be the second computing instance whose CPU load is the smallest.

[0094] When the first local scheduler determines the second target computing instance, it can send the inference calculation result of the Prefill stage to the second target computing instance through the communication connection with the Decode engine, so that the second target computing instance can perform the inference calculation of the Decode stage based on the inference calculation result of the Prefill stage. That is, for the first local scheduler corresponding to the first computing instance for performing the inference calculation of the Prefill stage, it can schedule the inference calculation result of the Prefill stage to the second computing instance whose GPU load meets the second preset condition to perform the inference calculation of the Decode stage, thereby realizing the load-aware scheduling of the Decode stage for the inference request on the cluster-level inference engine.

[0095] By splitting the Prefill stage and the Decode stage in the inference process and performing load-aware scheduling in the Prefill stage and the Decode stage respectively, the scheduling effect can be further improved, thereby optimizing resource utilization efficiency, reducing waiting time and processing delays, and improving the overall throughput of the system.

[0096] In actual applications, when maintaining the GPU load information of each first computing instance, the above-mentioned global scheduler can store the instance information of each first computing instance in association with the GPU load information, so that the first target computing instance can be located according to the instance information of the first target computing instance whose determined GPU load meets the first preset condition, and the above-mentioned target inference request can be sent to the first target computing instance.

[0097] Similarly, when maintaining the GPU load information of each second computing instance, the above-mentioned first local scheduler can store the instance information of each second computing instance in association with the GPU load information, so that the second target computing instance can be located based on the instance information of the second target computing instance whose determined GPU load meets the second preset condition, and the inference calculation results of the Prefill stage can be sent to the second target computing instance.

[0098] In some embodiments, as described above, in addition to including at least one computing instance, the above-mentioned inference engine may also include a local scheduler. In this case, when sending the above-mentioned target inference request to the above-mentioned first target computing instance, the target inference request can be specifically sent to the first local scheduler deployed on the computing node where the first target computing instance is located, so that the first local scheduler sends the target inference request to the first target computing instance. When sending the inference calculation result of the Prefill stage to the above-mentioned second target computing instance, the inference calculation result of the Prefill stage can be specifically sent to the second local scheduler deployed on the computing node where the second target computing instance is located, so that the second local scheduler sends the inference calculation result of the Prefill stage to the second target computing instance.

[0099] In some embodiments, in order to make the dynamic update of the GPU load information of each first computing instance maintained by the above-mentioned global scheduler easy to implement, accurate and reliable, each first local scheduler can periodically count the GPU load information of the corresponding each first computing instance according to a preset time period (for example: generate GPU load information based on the current GPU load status of the first computing instance), and report the counted GPU load information to the global scheduler, so that the global scheduler can update the GPU load information of each first computing instance maintained based on the GPU load information reported by each first local scheduler.

[0100] Similarly, in order to make the dynamic update of the GPU load information of each second computing instance maintained by the above-mentioned first local scheduler easy to implement, accurate and reliable, each second local scheduler can periodically count the GPU load information of the corresponding second computing instance according to a preset time period, and report the counted GPU load information to each first local scheduler, so that each first local scheduler can update the GPU load information of each maintained second computing instance based on the GPU load information reported by each second local scheduler.

[0101] In some embodiments, for the above-mentioned first computing instance, its GPU load information can be specifically expressed as GPU resource utilization. Among them, GPU resource utilization can be an indicator for indicating the GPU load, and can specifically be an indicator calculated by the first local scheduler based on the streaming multiprocessor activity (i.e., SM Activity) and the streaming multiprocessor clock frequency (i.e., SM Clock) of the first computing instance. At this time, when determining the above-mentioned first target computing instance with the smallest GPU load based on the GPU load information of each first computing instance maintained, the first target computing instance with the smallest GPU resource utilization can be specifically determined based on the GPU load information of each first computing instance maintained.

[0102] The inference request in the prefill phase is a computationally intensive task, and the GPU utilization is mainly based on the SM activity indicator selection. SM is the core module responsible for performing computational tasks in the GPU, so it directly affects the speed at which the GPU executes instructions. When the GPU load is high, the SM clock frequency will decrease.

[0103] In some embodiments, the GPU resource utilization can be calculated based on the streaming multiprocessor activity and the streaming multiprocessor clock frequency of the computing instance by the following formula:

[0104] U GPU_I =A SM ×f;

[0105]

[0106] Among them, U GPU_I Indicates the GPU resource utilization of the computing instance, A SM represents the streaming multiprocessor activity of the computing instance, f represents the compound coefficient, γ represents the preset first control coefficient, δ represents the preset second control coefficient, and t ′ represents the preset second threshold, C SM Indicates the streaming multiprocessor clock frequency of the compute instance.

[0107] Through the above-mentioned GPU resource utilization, the above-mentioned GPU memory utilization and the above-mentioned GPU memory bandwidth utilization can be combined for load-aware scheduling. In actual application scenarios, SM Clock is negatively correlated with GPU memory utilization. In order to avoid a significant drop in GPU performance due to an excessively high SM Clock, a threshold (i.e., the above-mentioned second threshold) can be set for SM Clock. When the SM Clock value is lower than the second threshold, the SM Clock tends to be increased, resulting in greater GPU resource utilization; when the SM Clock value exceeds the second threshold, the GPU memory utilization tends to be increased, resulting in less GPU resource utilization.

[0108] The above γ and δ are control coefficients that can be obtained through experiments and are used to control the sensitivity of the SM Clock to the response.

[0109] The load size of the computing instance is positively correlated with the GPU resource utilization. That is, the smaller the GPU resource utilization of the computing instance, the smaller the load of the computing instance; the larger the GPU resource utilization of the computing instance, the larger the load of the computing instance.

[0110] In some embodiments, for the above-mentioned second computing instance, its GPU load information can be specifically expressed as instance resource utilization. Among them, the instance resource utilization can also be an indicator for indicating the GPU load, which can be specifically an indicator calculated by the second local scheduler based on the GPU memory utilization and GPU memory bandwidth utilization of the second computing instance. Among them, the GPU memory utilization can be obtained by reading the hardware information of the GPU, and the GPU memory bandwidth utilization can be obtained through the Data Center GPU Manager (DCGM) service provided by NVIDIA. At this time, when determining the above-mentioned second target computing instance with the smallest GPU load based on the GPU load information of each second computing instance maintained, the second target computing instance with the smallest instance resource utilization can be specifically determined based on the GPU load information of each second computing instance maintained.

[0111] In practical applications, the above GPU memory utilization can be the utilization of memory blocks in the GPU. Block refers to a group of threads that can be organized together for easy management and communication. All threads in a block can collaborate to perform certain operations, and shared memory can be used to facilitate such collaboration. Each block can access a specific amount of shared memory and other resources. Therefore, the number of blocks and the number of threads in each block together determine the total amount of resources used.

[0112] It should be noted that when scheduling a certain inference request, if the inference request is an inference request for the Prefill stage in the inference process of a large model, the above-mentioned Block usage rate may specifically be the number of Blocks required for the Prompt of the large model; if the inference request is an inference request for the Decode stage in the inference process of a large model, the above-mentioned Block usage rate may specifically be the actual GPU memory utilization.

[0113] The inference request in the decode stage is a task related to GPU memory, so the GPU memory-related indicators are given priority when considering the load indicators. In addition, the higher the GPU memory bandwidth utilization, the more frequent the data exchange between the GPU memory and the main memory, especially when unloading from the GPU or reading back the data of the main memory to the GPU. In this case, the weight of the GPU memory bandwidth utilization indicator can be increased during the scheduling process.

[0114] In some embodiments, the instance resource utilization can be calculated based on the GPU memory utilization and GPU memory bandwidth utilization of the computing instance by the following formula:

[0115]

[0116] U Composite =U GPU ×a;

[0117]

[0118] Among them, U Instance Indicates the instance resource utilization of the computing instance, M Instance Indicates the total GPU memory size of the compute instance, U Composite represents the composite load of the computing instance, Speed ​​represents the number of tokens generated by the computing instance in each iteration, and U GPU represents the GPU memory utilization, a represents the composite coefficient, α represents the preset third control coefficient, β represents the preset fourth control coefficient, α>β, t represents the preset third threshold, U BW Indicates GPU memory bandwidth utilization.

[0119] Through the above-mentioned composite load, the above-mentioned GPU memory utilization and the above-mentioned GPU memory bandwidth utilization can be combined for load-aware scheduling. In actual application scenarios, the GPU memory bandwidth utilization is negatively correlated with the GPU memory utilization. In order to prevent the data access performance between the GPU memory and the main memory from dropping sharply due to excessive GPU memory bandwidth utilization, a threshold (i.e., the above-mentioned third threshold) can be set for the GPU memory bandwidth utilization. When the GPU memory bandwidth utilization is lower than the threshold, the GPU memory bandwidth utilization tends to be increased, thereby obtaining a larger composite load; when the GPU memory bandwidth utilization exceeds the threshold, the GPU memory utilization tends to be increased while the GPU memory bandwidth utilization is reduced, thereby obtaining a smaller composite load. In this way, the GPU memory utilization can be improved while maintaining a low GPU memory bandwidth utilization.

[0120] The above α and β are control coefficients that can be obtained experimentally and are used to control the response sensitivity to a. The value of α should be greater than the value of β, indicating that it is desirable to increase the GPU memory bandwidth utilization when the GPU memory bandwidth utilization is below a threshold.

[0121] The above Speed ​​can represent the number of new tokens generated by all inference requests in each iteration (TokensGenerated Per Iteration). Therefore, It can be expressed as how many iterations are needed to consume the remaining GPU memory of the computing instance according to the current token generation rate. It can be seen that the smaller the instance resource utilization of the computing instance, the more iterations the remaining GPU memory of the computing instance can support, and the smaller the load of the computing instance; the larger the instance resource utilization of the computing instance, the fewer iterations the remaining GPU memory of the computing instance can support, and the larger the load of the computing instance. In other words, the load size of the computing instance is positively correlated with the instance resource utilization.

[0122] It should be noted that the reasoning process of a large model refers to the process of using a trained large machine learning model to generate corresponding output results based on given input data (i.e., the data contained in the reasoning request). In the field of natural language processing, especially in large language models, the reasoning process usually involves generating a continuous text sequence based on a given input prompt, and generating a token for each iteration. The "token" here usually refers to a series of small units into which text is segmented in natural language processing, such as words or characters, depending on the word segmentation method of the model. Specifically, a piece of text can be provided to the model as input, which is called a Prompt; the model receives the Prompt and converts it into a form that can be processed internally, usually by converting each word into a corresponding digital representation (for example, WordEmbedding); based on the current Prompt, the model predicts the next most likely Token through complex calculations (for example, multi-layer neural network operations); the model adds the predicted Token to the existing Prompt to form a new Prompt; the above steps can be repeated multiple times, and each repetition is called an iteration. In each iteration, the model predicts the next Token based on the latest Prompt until the set stop condition is reached, such as a certain number of Tokens are generated or a specific end mark is encountered.

[0123] In some embodiments, for the second computing instance, its GPU load information may also include the GPU memory utilization. When the first local scheduler determines the second target computing instance with the lowest instance resource utilization based on the GPU load information of each second computing instance maintained, it may first determine the computing instance (referred to as the candidate computing instance) with a GPU memory utilization less than a preset threshold (referred to as the first threshold) based on the GPU load information of each second computing instance maintained, and then sort the candidate computing instances according to the instance resource utilization to determine the second target computing instance with the lowest instance resource utilization from the candidate computing instances. In this way, the number of comparisons of the GPU loads of different second computing instances when the first local scheduler determines the second target computing instance with the lowest GPU load based on the GPU load information of each second computing instance maintained can be reduced, thereby improving the selection efficiency of the second target computing instance and thus improving the scheduling efficiency.

[0124] In some embodiments, when a computing instance is in the process of being terminated, the instance resource utilization of the computing instance may be set to infinity, thereby avoiding scheduling an inference request to the computing instance in the process of being terminated.

[0125] In the above technical solution, the inference engine in the inference system may include a Prefill engine and a Decode engine, the Prefill engine may include at least one first computing instance, the Decode engine may include at least one second computing instance, and the computing resources of each computing instance may include a GPU mounted on the computing node where it is located; the global scheduler in the inference system may maintain the GPU load information of each first computing instance that is dynamically updated, and may, when obtaining the target inference request to be executed, first determine the first computing instance whose GPU load meets the first preset condition based on the GPU load information of each first computing instance that is maintained, and then send the target inference request to the first computing instance to perform the inference calculation of the Prefill stage, so as to realize the scheduling of the Prefill stage in the inference process; the first local scheduler corresponding to the first computing instance in the inference system may maintain the GPU load information of each second computing instance that is dynamically updated, and may, when the first computing instance calculates the inference calculation result of the Prefill stage, first determine the second computing instance whose GPU load meets the second preset condition based on the GPU load information of each second computing instance that is maintained, and then send the inference calculation result of the Prefill stage to the second computing instance to perform the inference calculation of the Decode stage, so as to realize the scheduling of the Decode stage in the inference process.

[0126] By adopting the above method, on the one hand, by adding a global scheduler to the inference system, the global scheduler can schedule the inference request to the computing instance with appropriate GPU load in the inference engine for execution, thereby realizing load-aware scheduling of inference requests on the cluster-level inference engine, thereby optimizing the efficiency of GPU resource utilization, reducing the waiting time and processing delay of inference request execution, and improving the overall throughput of the inference system. On the other hand, according to the different stages of the inference process, the inference request is load-awarely scheduled in stages on the cluster-level inference engine, which can further improve the scheduling effect. In addition, by scheduling the inference request to the inference engine through the global scheduler instead of directly scheduling the inference request in the inference engine, the scheduling method can be decoupled from the specific inference engine, which has certain versatility and scalability.

[0127] Corresponding to the aforementioned method embodiments, the present application also provides device embodiments.

[0128] Please refer to Figure 4 , Figure 4It is a structural diagram of a device shown in an exemplary embodiment of the present application. At the hardware level, the device includes a processor 402, an internal bus 404, a network interface 406, a memory 408 and a non-volatile memory 410, and of course may also include other required hardware. One or more embodiments of the present application can be implemented based on software, such as the processor 402 reading the corresponding computer program from the non-volatile memory 410 into the memory 408 and then running it. Of course, in addition to the software implementation, one or more embodiments of the present application do not exclude other implementations, such as logic devices or a combination of software and hardware, etc., that is to say, the execution subject of the following processing flow is not limited to each logic module, but can also be hardware or logic devices.

[0129] Please refer to Figure 5 , Figure 5 It is a block diagram of an inference system shown in an exemplary embodiment of the present application.

[0130] The above reasoning system can be applied to Figure 4 The device shown is used to implement the technical solution of the present application. The inference system includes a global scheduler 502 and an inference engine 504; the inference engine includes a Prefill engine and a Decode engine; the Prefill engine is deployed on at least one first computing instance, and the Decode engine is deployed on at least one second computing instance; the computing resources of the first and second computing instances include the GPUs mounted on the computing nodes where they are located; the global scheduler maintains the dynamically updated GPU load information of each first computing instance; the inference system also includes a first local scheduler; the first local scheduler maintains the dynamically updated GPU load information of each second computing instance;

[0131] The global scheduler is used to:

[0132] Get the target inference request to be executed;

[0133] Based on the maintained GPU load information of each first computing instance, determine a first computing instance whose GPU load meets a first preset condition, and send the target inference request to the first computing instance, so that the first computing instance performs inference calculation of the prefill stage for the target inference request;

[0134] The first local scheduler is used for:

[0135] Based on the GPU load information of each computing instance in the maintained second computing instance, determine the second computing instance whose GPU load meets the second preset condition, and send the inference calculation result of the prefill stage to the second computing instance, so that the second computing instance performs the inference calculation of the Decode stage based on the inference calculation result of the prefill stage.

[0136] In some embodiments, the first preset condition is that the GPU load in the first computing instance is minimal; and / or,

[0137] The second preset condition is that the GPU load in the second computing instance is the smallest.

[0138] In some embodiments, the Decode engine further includes a second local scheduler;

[0139] The sending the target inference request to the first computing instance includes:

[0140] Sending the target inference request to the first local scheduler, so that the first local scheduler sends the target inference request to the first computing instance;

[0141] The sending the inference calculation result of the Prefill phase to the second computing instance includes:

[0142] The inference calculation result of the Prefill stage is sent to the second local scheduler, so that the second local scheduler sends the inference calculation result of the Prefill stage to the second computing instance.

[0143] In some embodiments, the global scheduler is further configured to:

[0144] Obtaining GPU load information of each first computing instance periodically sent by the first local scheduler according to a preset first time period, and updating the maintained GPU load information of each first computing instance based on the obtained GPU load information;

[0145] The first local scheduler is further configured to:

[0146] Obtain GPU load information of each second computing instance periodically sent by the second local scheduler according to a preset second time period, and update the maintained GPU load information of each second computing instance based on the acquired GPU load information.

[0147] In some embodiments, the GPU load information of the first computing instance includes GPU resource utilization; wherein the GPU resource utilization is an indicator for indicating the GPU load calculated by the first local scheduler based on the streaming multiprocessor activity and the streaming multiprocessor clock frequency of the first computing instance;

[0148] The determining, based on the maintained GPU load information of each first computing instance, a first computing instance whose GPU load satisfies a first preset condition includes:

[0149] Based on the maintained GPU load information of each first computing instance, a first computing instance having the lowest GPU resource utilization is determined.

[0150] In some embodiments, the GPU load information of the second computing instance includes instance resource utilization; wherein the instance resource utilization is an indicator for indicating the GPU load calculated by the second local scheduler based on the GPU memory utilization and GPU memory bandwidth utilization of the second computing instance;

[0151] The determining, based on the maintained GPU load information of each second computing instance, a second computing instance whose GPU load satisfies a second preset condition includes:

[0152] Based on the maintained GPU load information of each second computing instance, a second computing instance having the lowest instance resource utilization is determined.

[0153] In some embodiments, the GPU load information of the second computing instance further includes GPU memory utilization;

[0154] The determining, based on the maintained GPU load information of each second computing instance, the second computing instance with the lowest instance resource utilization rate comprises:

[0155] Determine, based on the maintained GPU load information of each second computing instance, a candidate computing instance whose GPU memory utilization is less than a preset first threshold;

[0156] The candidate computing instances are sorted according to the instance resource utilization rates to determine a second computing instance with the lowest instance resource utilization rate from the candidate computing instances.

[0157] In some embodiments, the GPU resource utilization is calculated based on the streaming multiprocessor activity and the streaming multiprocessor clock frequency of the computing instance by the following formula:

[0158] U GPU_I =A SM ×f;

[0159]

[0160] Among them, U GPU_I Indicates the GPU resource utilization of the computing instance, A SM represents the streaming multiprocessor activity of the computing instance, f represents the compound coefficient, γ represents the preset first control coefficient, δ represents the preset second control coefficient, and t ′ represents the preset second threshold, C SM Indicates the streaming multiprocessor clock frequency of the compute instance.

[0161] In some embodiments, the instance resource utilization is calculated based on the GPU memory utilization and GPU memory bandwidth utilization of the computing instance by the following formula:

[0162]

[0163] U Composite =U GPU ×a;

[0164]

[0165] Among them, U Instance Indicates the instance resource utilization of the computing instance, M Instance Indicates the total GPU memory size of the compute instance, U Composite represents the composite load of the computing instance, Speed ​​represents the number of tokens generated by the computing instance in each iteration, and U GPU represents the GPU memory utilization of the computing instance, a represents the composite coefficient, α represents the preset third control coefficient, β represents the preset fourth control coefficient, α>β, t represents the preset third threshold, U BW Indicates the GPU memory bandwidth utilization of the compute instance.

[0166] In some embodiments, the instance resource utilization of a computing instance is set to infinity if the computing instance is in the process of terminating.

[0167] For the device embodiment, it basically corresponds to the method embodiment, so the relevant parts can refer to the partial description of the method embodiment. The device embodiment described above is only schematic, wherein the modules described as separate components may or may not be physically separated, and the components displayed as modules may or may not be physical modules, that is, they may be located in one place, or they may be distributed on multiple network modules. Some or all of the modules may be selected according to actual needs to achieve the purpose of the technical solution of the present application.

[0168] The systems, devices, modules or units described in the above embodiments may be implemented by computer chips or entities, or by products with certain functions. A typical implementation device is a computer, which may be in the form of a personal computer, a laptop computer, a cellular phone, a camera phone, a smart phone, a personal digital assistant, a media player, a navigation device, an email transceiver, a game console, a tablet computer, a wearable device or a combination of any of these devices.

[0169] In a typical configuration, a computer includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0170] The memory may include non-permanent storage in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. The memory is an example of a computer-readable medium.

[0171] Computer-readable media include permanent and non-permanent, removable and non-removable media that can store information by any method or technology. Information can be computer-readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, read-only compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, disk storage, quantum memory, graphene-based storage media or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer-readable media does not include temporary computer-readable media (transitory media), such as modulated data signals and carrier waves.

[0172] It should be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of further restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0173] The above specific embodiments of the present application are described. Other embodiments are within the scope of the present application. In some cases, the actions or steps recorded in the present application can be performed in an order different from that in the embodiments and still achieve the desired results. In addition, the process depicted in the accompanying drawings does not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0174] The terms used in one or more embodiments of the present application are only for the purpose of describing specific embodiments, and are not intended to limit one or more embodiments of the present application. The singular forms of "a", "said" and "the" are also intended to include plural forms, unless the context clearly indicates other meanings. The term "and / or" refers to and includes any or all possible combinations of one or more associated listed items.

[0175] The description of the terms "one embodiment", "some embodiments", "example", "specific example" or "an implementation method" etc. used in one or more embodiments of the present application means that the specific features or characteristics described in conjunction with the embodiment are included in at least one embodiment of the present application. The schematic description of these terms is not necessarily for the same embodiment. Moreover, the specific features or characteristics described can be combined in a suitable manner in one or more embodiments of the present application. In addition, different embodiments and specific features or characteristics in different embodiments can be combined without contradicting each other.

[0176] It should be understood that, although the terms first, second, third, etc. may be used to describe various information in one or more embodiments of the present application, these information should not be limited to these terms. These terms are only used to distinguish the same type of information from each other. For example, without departing from the scope of one or more embodiments of the present application, the first information may also be referred to as the second information, and similarly, the second information may also be referred to as the first information. Depending on the context, the word "if" as used herein may be interpreted as "at the time of" or "when" or "in response to determination".

[0177] The above description is merely a preferred embodiment of one or more embodiments of the present application and is not intended to limit one or more embodiments of the present application. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of one or more embodiments of the present application shall be included in the scope of protection of one or more embodiments of the present application.

[0178] The user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of relevant countries and regions, and provide corresponding operation entrances for users to choose to authorize or refuse.

Claims

1. A load-aware scheduling method for an inference system, applied to an inference system; The inference system includes a global scheduler and an inference engine; the inference engine includes a Prefill engine and a Decode engine; The Prefill engine is deployed on at least one first computing instance, and the Decode engine is deployed on at least one second computing instance; the computing resources of the first and second computing instances include GPUs mounted on the computing nodes where they are located; the global scheduler maintains dynamically updated GPU load information of each first computing instance; the inference system also includes a first local scheduler; the first local scheduler maintains dynamically updated GPU load information of each second computing instance; The method comprises: The global scheduler obtains a target inference request to be executed; The global scheduler determines a first computing instance whose GPU load satisfies a first preset condition based on the GPU load information of each first computing instance maintained, and sends the target inference request to the first computing instance, so that the first computing instance performs the inference calculation of the Prefill stage for the target inference request; The first local scheduler determines a second computing instance whose GPU load meets a second preset condition based on the GPU load information of each computing instance in the maintained second computing instance, and sends the inference calculation result of the Prefill stage to the second computing instance, so that the second computing instance performs the inference calculation of the Decode stage based on the inference calculation result of the Prefill stage.

2. The method according to claim 1, wherein the first preset condition is that the GPU load in the first computing instance is minimal; and / or, The second preset condition is that the GPU load in the second computing instance is the smallest.

3. The method according to claim 1, wherein the reasoning system further comprises a second local scheduler; The sending the target inference request to the first computing instance includes: Sending the target inference request to the first local scheduler, so that the first local scheduler sends the target inference request to the first computing instance; The sending the inference calculation result of the Prefill phase to the second computing instance includes: The inference calculation result of the Prefill stage is sent to the second local scheduler, so that the second local scheduler sends the inference calculation result of the Prefill stage to the second computing instance.

4. The method according to claim 3, further comprising: The global scheduler obtains GPU load information of each first computing instance periodically sent by the first local scheduler according to a preset first time period, and updates the maintained GPU load information of each first computing instance based on the obtained GPU load information; The first local scheduler obtains GPU load information of each second computing instance periodically sent by the second local scheduler according to a preset second time period, and updates the maintained GPU load information of each second computing instance based on the obtained GPU load information.

5. According to the method of claim 1, the GPU load information of the first computing instance includes GPU resource utilization; wherein, The GPU resource utilization is an indicator for indicating GPU load calculated by the first local scheduler based on the streaming multiprocessor activity and the streaming multiprocessor clock frequency of the first computing instance; The determining, based on the maintained GPU load information of each first computing instance, a first computing instance whose GPU load satisfies a first preset condition includes: Based on the maintained GPU load information of each first computing instance, a first computing instance having the lowest GPU resource utilization is determined.

6. According to the method of claim 3, the GPU load information of the second computing instance includes instance resource utilization; wherein, The instance resource utilization is an indicator for indicating GPU load calculated by the second local scheduler based on the GPU memory utilization and GPU memory bandwidth utilization of the second computing instance; The determining, based on the maintained GPU load information of each second computing instance, a second computing instance whose GPU load satisfies a second preset condition includes: Based on the maintained GPU load information of each second computing instance, a second computing instance having the lowest instance resource utilization is determined.

7. The method according to claim 6, wherein the GPU load information of the second computing instance further includes GPU memory utilization; The determining, based on the maintained GPU load information of each second computing instance, the second computing instance with the lowest instance resource utilization rate comprises: Determine, based on the maintained GPU load information of each second computing instance, a candidate computing instance whose GPU memory utilization is less than a preset first threshold; The candidate computing instances are sorted according to the instance resource utilization rates to determine a second computing instance with the lowest instance resource utilization rate from the candidate computing instances.

8. The method according to claim 5, wherein the GPU resource utilization is calculated based on the streaming multiprocessor activity and the streaming multiprocessor clock frequency of the computing instance by the following formula: U GPU_I =A SM ×f; in, U GPU_I Indicates the GPU resource utilization of the computing instance, A SM represents the streaming multiprocessor activity of the computing instance, f represents the compound coefficient, γ represents the preset first control coefficient, δ represents the preset second control coefficient, t′ represents the preset second threshold, C SM Indicates the streaming multiprocessor clock frequency of the compute instance.

9. The method according to claim 6, wherein the instance resource utilization is calculated based on the GPU memory utilization and GPU memory bandwidth utilization of the computing instance by the following formula: IN Composite =U GPU ×a; in, U Instance Indicates the instance resource utilization of the computing instance, M Instance Indicates the total GPU memory size of the compute instance, U Composite represents the composite load of the computing instance, Speed ​​represents the number of tokens generated by the computing instance in each iteration, and U GPU represents the GPU memory utilization of the computing instance, a represents the composite coefficient, α represents the preset third control coefficient, β represents the preset fourth control coefficient, α>β, t represents the preset third threshold, U BW Indicates the GPU memory bandwidth utilization of the compute instance. 10 . The method according to claim 9 , wherein the instance resource utilization of the computing instance is set to infinity when the computing instance is in the process of being terminated.

11. An inference system, comprising a global scheduler and an inference engine; the inference engine comprises a Prefill engine and a Decode engine; the Prefill engine is deployed on at least one first computing instance, and the Decode engine is deployed on at least one second computing instance; the computing resources of the first and second computing instances comprise GPUs mounted on the computing nodes where they are located; the global scheduler maintains GPU load information of each first computing instance that is dynamically updated; the inference system further comprises a first local scheduler; the first local scheduler maintains GPU load information of each second computing instance that is dynamically updated; The global scheduler is used to: Get the target inference request to be executed; Based on the maintained GPU load information of each first computing instance, determine a first computing instance whose GPU load meets a first preset condition, and send the target inference request to the first computing instance, so that the first computing instance performs inference calculation of the prefill stage for the target inference request; The first local scheduler is used for: Based on the GPU load information of each computing instance in the maintained second computing instance, determine the second computing instance whose GPU load meets the second preset condition, and send the inference calculation result of the prefill stage to the second computing instance, so that the second computing instance performs the inference calculation of the Decode stage based on the inference calculation result of the prefill stage.

12. An electronic device comprising: processor; a memory for storing processor-executable instructions; The processor implements the method according to any one of claims 1 to 10 by running the executable instructions.

13. A computer-readable storage medium having computer instructions stored thereon, wherein when the instructions are executed by a processor, the method according to any one of claims 1 to 10 is implemented.

Images