Load-aware scheduling method for inference systems and inference systems

Abstract

This application provides a load-aware scheduling method and an inference system according to one or more embodiments. The method is applied to an inference system including a global scheduler and an inference engine. The inference engine includes a prefill engine and a decode engine. The prefill engine and the decode engine each include at least one computing instance. The computing resources of the computing instance include the GPU mounted on the computing node where it resides. The method includes: the global scheduler acquiring a target inference request to be executed, and based on the maintained GPU load information, determining a first computing instance in the prefill engine whose GPU load meets a first preset condition, and sending the target inference request to the first computing instance to perform inference computation in the prefill stage; a first local scheduler in the prefill engine corresponding to the first computing instance, based on the maintained GPU load information, determining a second computing instance in the decode engine whose GPU load meets a second preset condition, and sending the inference computation result of the prefill stage to the second computing instance to perform inference computation in the decode stage.

Description

Technical Field

[0001] One or more embodiments of this application relate to the field of artificial intelligence technology, and in particular to a load-aware scheduling method and an inference system for an inference system. Background Technology

[0002] An inference system is a computer program that uses logical rules and known facts to arrive at new conclusions or decisions. It is an important component of artificial intelligence, primarily used to simulate human decision-making processes. It derives conclusions based on a defined knowledge base and inference engine. An inference system can execute its received inference requests and output the corresponding inference results.

[0003] A typical reasoning system usually consists of the following components: a knowledge base, an inference engine, a user interface, and an explanation facility. The knowledge base stores all the facts and rules known to the system. These facts can be about the state of the world, object attributes, etc., while the rules are logical expressions describing how to derive new conclusions from known facts. The inference engine is the core component of the reasoning system; it is responsible for performing the logical operations in the reasoning process, that is, deriving new conclusions or decisions from the given knowledge base. The inference engine uses a set of rules and known facts to deduce new knowledge, thereby helping the system solve problems or make decisions. The user interface allows users to interact with the system, input queries, or observe the results of the reasoning process. The explanation facility explains how the system arrives at specific conclusions, which is crucial for transparency and trustworthiness.

[0004] In scenarios requiring the processing of large-scale data, high concurrency requests, or high-performance computing, inference engines are typically deployed on compute clusters. Deploying inference engines on compute clusters enables higher computing power, better fault tolerance, and more flexible resource management. However, this raises the question of how to schedule inference requests at the cluster-level inference engine. Summary of the Invention

[0005] One or more embodiments of this application provide the following technical solutions:

[0006] This application provides a load-aware scheduling method for an inference system, applied to an inference system; the inference system includes a global scheduler and an inference engine; the inference engine includes a prefill engine and a decode engine; the prefill engine is deployed on at least one first computing instance, and the decode engine is deployed on at least one second computing instance; the computing resources of the first and second computing instances include the GPUs mounted on their respective computing nodes; the global scheduler maintains dynamically updated GPU load information for each of the first computing instances; the inference system also includes a first local scheduler; the first local scheduler maintains dynamically updated GPU load information for each of the second computing instances;

[0007] The method includes:

[0008] The global scheduler acquires the target inference request to be executed;

[0009] The global scheduler determines the first computing instance whose GPU load meets the first preset condition based on the GPU load information of each first computing instance it maintains, and sends the target inference request to the first computing instance so that the first computing instance can perform inference computation in the Prefill phase for the target inference request.

[0010] The first local scheduler determines the second computing instance whose GPU load meets the second preset condition based on the GPU load information of each computing instance in the maintained second computing instance, and sends the inference computing results of the Prefill stage to the second computing instance so that the second computing instance can perform the inference computing of the Decode stage based on the inference computing results of the Prefill stage.

[0011] This application also provides an inference system, which includes a global scheduler and an inference engine; the inference engine includes a prefill engine and a decode engine; the prefill engine is deployed on at least one first computing instance, and the decode engine is deployed on at least one second computing instance; the computing resources of the first and second computing instances include GPUs mounted on their respective computing nodes; the global scheduler maintains dynamically updated GPU load information for each of the first computing instances; the inference system also includes a first local scheduler; the first local scheduler maintains dynamically updated GPU load information for each of the second computing instances.

[0012] The global scheduler is used for:

[0013] Obtain the target inference request to be executed;

[0014] Based on the GPU load information of each of the maintained first computing instances, the first computing instance whose GPU load meets the first preset condition is determined, and the target inference request is sent to the first computing instance so that the first computing instance can perform inference computation in the prefill stage for the target inference request.

[0015] The first local scheduler is used for:

[0016] Based on the GPU load information of each computing instance in the maintained second computing instance, a second computing instance whose GPU load meets the second preset condition is determined, and the inference computing results of the prefill stage are sent to the second computing instance so that the second computing instance can perform the inference computing of the Decode stage based on the inference computing results of the prefill stage.

[0017] This application also provides an electronic device, including:

[0018] processor;

[0019] Memory used to store processor-executable instructions;

[0020] The processor executes the executable instructions to implement the steps of the method as described in any of the preceding descriptions.

[0021] This application also provides a computer-readable storage medium having computer instructions stored thereon, which, when executed by a processor, implement the steps of the method as described in any of the preceding claims.

[0022] In the above technical solution, the inference engine in the inference system can include a Prefill engine and a Decode engine. The Prefill engine can include at least one first computing instance, and the Decode engine can include at least one second computing instance. The computing resources of each computing instance can include the GPU mounted on its computing node. The global scheduler in the inference system can maintain dynamically updated GPU load information of each first computing instance. When a target inference request to be executed is obtained, it can first determine the first computing instance whose GPU load meets the first preset condition based on the maintained GPU load information of each first computing instance, and then send the target inference request to the first computing instance to perform the inference computation of the Prefill stage, thereby realizing the scheduling of the Prefill stage in the inference process. The first local scheduler corresponding to the first computing instance in the inference system can maintain dynamically updated GPU load information of each second computing instance. When the first computing instance calculates the inference computation result of the Prefill stage, it can first determine the second computing instance whose GPU load meets the second preset condition based on the maintained GPU load information of each second computing instance, and then send the inference computation result of the Prefill stage to the second computing instance to perform the inference computation of the Decode stage, thereby realizing the scheduling of the Decode stage in the inference process.

[0023] By adopting the above approach, on the one hand, by adding a global scheduler to the inference system, inference requests can be scheduled to compute instances with appropriate GPU loads within the inference engine for execution. This achieves load-aware scheduling of inference requests at the cluster-level inference engine, thereby optimizing GPU resource utilization efficiency, reducing waiting time and processing latency for inference request execution, and improving the overall throughput of the inference system. On the other hand, performing phased load-aware scheduling of inference requests at the cluster-level inference engine based on different stages of the inference process can further improve scheduling effectiveness. Furthermore, scheduling inference requests to the inference engine via a global scheduler, rather than scheduling them directly within the inference engine, decouples the scheduling method from specific inference engines, providing a degree of versatility and scalability. Attached Figure Description

[0024] The accompanying drawings used in the description of the exemplary embodiments will now be explained, wherein:

[0025] Figure 1 This is a schematic diagram of an exemplary embodiment of the present application illustrating a reasoning system.

[0026] Figure 2 This is a schematic diagram of another inference system illustrated in an exemplary embodiment of this application.

[0027] Figure 3 This is a flowchart illustrating a load-aware scheduling method for an inference system according to an exemplary embodiment of this application.

[0028] Figure 4 This is a schematic diagram of the structure of a device shown in an exemplary embodiment of this application.

[0029] Figure 5 This is a block diagram illustrating an exemplary embodiment of the present application of an inference system. Detailed Implementation

[0030] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with one or more embodiments of this application. Rather, they are merely examples consistent with some aspects of one or more embodiments of this application.

[0031] It should be noted that the steps of the corresponding methods are not necessarily performed in the order shown and described in this application in other embodiments. In some other embodiments, the methods may include more or fewer steps than those described in this application. Furthermore, a single step described in this application may be broken down into multiple steps in other embodiments; and multiple steps described in this application may be combined into a single step in other embodiments.

[0032] In this application, the inference engine in the inference system can be deployed on a computing cluster to meet the needs of processing large-scale data, high-concurrency requests, or high-performance computing, thereby achieving higher computing power, better fault tolerance, and more flexible resource management. In this type of inference system, load-aware scheduling can be used to schedule inference requests at the cluster-level inference engine.

[0033] Load-aware scheduling is a strategy in computing systems that allocates tasks based on the current load of each node or resource. This scheduling method aims to optimize resource utilization efficiency, reduce waiting time and processing latency, and improve the overall throughput of the system.

[0034] Load-aware scheduling is commonly used in distributed systems, cloud computing platforms, and data centers. Its purpose is to dynamically balance the workload of each node, preventing some nodes from being overloaded while others are idle. By monitoring the load of each node in real time, the load-aware scheduler can make more reasonable task allocation decisions.

[0035] In general, inference engines can use CPUs as computing resources, where the CPU can be the CPU installed on the device where the inference engine is deployed. CPU load metrics typically include CPU utilization and memory utilization. That is, for inference engines deployed on computing clusters, calculating CPU utilization and memory utilization to represent CPU load allows for load-aware scheduling of inference requests on such engines.

[0036] With the development of artificial intelligence technology, reasoning systems based on large models (e.g., large language models) are being used more and more widely.

[0037] Large models refer to machine learning models with a large number of parameters, such as various variants of the Transformer architecture, including but not limited to natural language processing models like GPT and BERT. These models achieve powerful representation learning capabilities through large amounts of training data and complex architectures.

[0038] In large model-based inference systems, the large model can be viewed as a vast knowledge base containing information learned from massive amounts of data. During training, the large model learns numerous patterns and features that represent complex relationships within the data. Therefore, to some extent, the information stored within the large model can be considered a form of knowledge.

[0039] The inference engine, designed for large models, is responsible for performing specific inference tasks based on these models. This includes managing the loading of large models, executing inference operations on them, and managing interactions with hardware (e.g., CPUs, GPUs, or other accelerators). The inference engine can also include optimization algorithms to improve the speed and efficiency of inference.

[0040] In practical applications, large models can be deployed separately from the inference system, or they can be integrated into the inference system and the inference engine within it can be used to call the large model to perform inference tasks efficiently.

[0041] Since large models are artificial intelligence applications that require a lot of computing resources, inference engines typically use GPUs as computing resources when calling large models to perform inference tasks.

[0042] GPUs are designed for parallel processing, capable of handling multiple data points simultaneously. This is particularly useful for deep learning models, which often need to perform the same operations on large amounts of data. Modern AI models, especially neural network models, require a significant amount of floating-point operations. GPUs typically outperform CPUs in floating-point operations, especially when handling large-scale matrix operations, a common operation in deep learning. GPUs generally have higher memory bandwidth than CPUs, meaning they can read and write data from memory much faster. This is crucial for applications that require frequent access to large amounts of data. Many GPU vendors (e.g., NVIDIA) have optimized their hardware and software stacks for machine learning tasks. For example, they offer hardware specifically designed to accelerate tensor operations, and programming models like CUDA to fully leverage these hardware features. GPUs can improve inference speed. For applications deployed at scale, using GPUs can reduce latency and improve system responsiveness.

[0043] There are many GPU load metrics to choose from, commonly including SM Clock (Streaming Multiprocessor Clock), SM Activity (Streaming Multiprocessor Activity), and Memory Utilization (Memory Usage). SM Clock refers to the clock frequency of the streaming multiprocessors on the GPU; streaming multiprocessors are computing units on the GPU responsible for performing parallel computing tasks. A higher SM Clock frequency means the GPU's computing units run faster, which usually leads to higher computing performance, but may also increase power consumption and heat. SM Activity refers to the activity level of the streaming multiprocessors on the GPU, i.e., what percentage of a given period of time these computing units are actually executing tasks. A high SM Activity indicates that the GPU's computing units are actively working most of the time; conversely, a low SM Activity may indicate a computing bottleneck or long waiting times. Memory Utilization refers to the GPU's memory usage, typically representing the percentage of total memory currently in use. A high Memory Utilization means more memory is being used, which may slow down data exchange or impact performance due to insufficient memory. When the memory is near full capacity, it may trigger data overflow into host memory, further degrading performance.

[0044] For inference engines deployed on computing clusters, load-aware scheduling of inference requests can be achieved by calculating GPU load metrics to represent GPU load.

[0045] In the technical solutions provided by one or more embodiments of this application, the inference engine in the inference system may include a Prefill engine and a Decode engine. The Prefill engine may include at least one first computing instance, and the Decode engine may include at least one second computing instance. The computing resources of each computing instance may include the GPU mounted on the computing node where it resides. The global scheduler in the inference system can maintain dynamically updated GPU load information of each first computing instance. When a target inference request to be executed is obtained, it can first determine the first computing instance whose GPU load meets a first preset condition based on the maintained GPU load information of each first computing instance, and then send the target inference request to the first computing instance to perform the inference computation of the Prefill stage, thereby realizing the scheduling of the Prefill stage in the inference process. The first local scheduler corresponding to the first computing instance in the inference system can maintain dynamically updated GPU load information of each second computing instance. When the first computing instance calculates the inference computation result of the Prefill stage, it can first determine the second computing instance whose GPU load meets a second preset condition based on the maintained GPU load information of each second computing instance, and then send the inference computation result of the Prefill stage to the second computing instance to perform the inference computation of the Decode stage, thereby realizing the scheduling of the Decode stage in the inference process.

[0046] By adopting the above approach, on the one hand, by adding a global scheduler to the inference system, inference requests can be scheduled to compute instances with appropriate GPU loads within the inference engine for execution. This achieves load-aware scheduling of inference requests at the cluster-level inference engine, thereby optimizing GPU resource utilization efficiency, reducing waiting time and processing latency for inference request execution, and improving the overall throughput of the inference system. On the other hand, performing phased load-aware scheduling of inference requests at the cluster-level inference engine based on different stages of the inference process can further improve scheduling effectiveness. Furthermore, scheduling inference requests to the inference engine via a global scheduler, rather than scheduling them directly within the inference engine, decouples the scheduling method from specific inference engines, providing a degree of versatility and scalability.

[0047] Please refer to Figure 1 and Figure 2 , Figure 1 and Figure 2 These are schematic diagrams illustrating an exemplary embodiment of the present application of an inference system.

[0048] like Figure 1As shown, the aforementioned inference system may include a global scheduler and an inference engine. The inference engine can be deployed on a computing cluster consisting of at least one computing node.

[0049] A computing cluster is a system in which multiple computers or servers are connected via a network to work collaboratively. These nodes work together to complete computing tasks, providing greater computing power and higher availability than a single computer. A computing node is one of the core components of a computing cluster, typically referring to a computer or server used to perform computationally intensive tasks. Computing nodes can undertake computing tasks, and the GPUs mounted on them can serve as computing resources. The computers or servers can be physical or virtual local computers or servers, or physical or virtual cloud computers or servers.

[0050] In practical applications, the global scheduler described above can be deployed on the head node of the computing cluster. The head node usually refers to the node responsible for the management and scheduling of the computing cluster. It can also be deployed on other computing devices outside the computing cluster. This application does not limit this.

[0051] In the aforementioned inference system, the global scheduler and the inference engine can communicate with each other, for example, they can communicate directly via the HTTP (HyperText Transfer Protocol) protocol.

[0052] Specifically, the global scheduler can communicate with the applications and components deployed on each compute node in the computing cluster. In this case, the global scheduler can obtain inference requests from the request queue, and select a compute node with a suitable GPU load based on the load of the GPUs on each compute node, scheduling the obtained inference requests to that compute node so that the GPUs on that compute node can be used to execute the inference requests.

[0053] It should be noted that the reasoning process of an inference system may include multiple stages in some cases. For example, for a large model-based inference system, the inference process of the large model (especially a generative model in natural language processing) may include a prefill stage and a decode stage.

[0054] In the reasoning process of a large model, the Prefill and Decode stages refer to different steps in generating text.

[0055] In practical applications, the Prefill and Decode stages are closely linked. The Prefill stage provides the initial text foundation, while the Decode stage is responsible for progressively expanding upon this foundation to generate the complete text output. Together, these two stages determine the quality and coherence of the final generated text.

[0056] Based on the above, such as Figure 2 As shown, the computing nodes in the aforementioned computing cluster can be divided into two categories: the first category consists of computing nodes used to perform inference computation in the Prefill phase, and the second category consists of computing nodes used to perform inference computation in the Decode phase. In this case, based on the inference computation performed, the aforementioned inference engines can be divided into Prefill engines and Decode engines; where the Prefill engine is the inference engine deployed on the first type of computing nodes, and the Decode engine is the inference engine deployed on the second type of computing nodes.

[0057] In practical applications, the resources (e.g., computing resources, storage resources, network resources, etc.) of the computing nodes in the above computing cluster can be allocated as needed, so that a portion of these resources are used to perform inference computation in the Prefill stage, and the inference engine using this portion of resources can be called the Prefill engine; another portion of these resources are used to perform inference computation in the Decode stage, and the inference engine using this portion of resources can be called the Decode engine.

[0058] That is, the Prefill engine and the Decode engine can be deployed on different computing nodes in the computing cluster, or they can be deployed on the same computing node in the computing cluster, but they can use different parts of the resources on the computing node. This application does not restrict this.

[0059] Furthermore, the aforementioned Prefill engine can be deployed on at least one compute instance (referred to as the first compute instance). Similarly, the aforementioned Decode engine can also be deployed on at least one compute instance (referred to as the second compute instance). A compute instance refers to dedicated resources allocated on demand through virtualization technology within a local data center or cloud computing platform. These compute instance resources can include computing resources, storage resources, network resources, etc. (e.g., GPUs, GPU memory, storage, network interfaces, etc.), enabling each compute instance to use these resources to execute specific computing tasks. Compute instances can be created, started, stopped, or deleted as needed, and their resources can also be adjusted according to requirements.

[0060] Specifically, at least one first computing instance can be deployed on the computing nodes or some resources used to perform inference computation in the Prefill phase, and these first computing instances together constitute the aforementioned Prefill engine. Similarly, at least one second computing instance can be deployed on the computing nodes or some resources used to perform inference computation in the Decode phase, and these second computing instances together constitute the aforementioned Decode engine.

[0061] In inference engines designed for large models, the computational resources for a computation instance can be the GPU of the compute node it resides in.

[0062] In practical applications, a computing node can host multiple GPUs, and a single GPU can be allocated to a computing instance as its computing resource. Alternatively, a computing node can host only one GPU, and the computing resources allocated to a computing instance can be a portion of that GPU's computing units, memory, etc.

[0063] In a large-model inference engine, each computing instance can invoke the large model to perform specific inference tasks. That is, the large model can be loaded into various computing instances, and each instance can execute the inference operations based on the large model. Specifically, the large model can be read from local or cloud storage media (e.g., local disk, cloud storage) into the GPU memory of each computing instance, so that inference can be performed on the GPU computing resources of each instance based on the large model.

[0064] In practical applications, a model runtime framework can be installed in a computing instance. After a large model is loaded into the computing instance, a model service can be configured based on the model runtime framework and the large model, enabling the computing instance to function as an independently runnable model service instance. The model runtime refers to the environment and framework used to execute model inference after model deployment. This environment typically includes a series of steps such as model loading, input data processing, model execution, and output result processing, covering the entire lifecycle of the model from loading to execution to unloading.

[0065] By deploying computing instances on computing nodes, firstly, the resources of the computing nodes can be fully utilized to accelerate the execution of computing tasks, thereby improving the computing power of the inference engine; secondly, the independence between different computing instances ensures the stability and security of computing tasks, and enables resource isolation and management, which helps to effectively utilize resources and avoid resource monopolization; thirdly, for computing tasks that require parallel processing, parallel computing can be used to accelerate the completion time of computing tasks; and fourthly, elastic scaling can be achieved, adding computing instances when a large number of computing tasks need to be processed, and terminating some computing instances when the number of computing tasks to be processed decreases, thereby releasing some resources and achieving flexible resource allocation.

[0066] To facilitate load-aware scheduling of inference requests during the Prefill phase, the global scheduler can maintain GPU load information for each first compute instance, indicating the GPU's load status. In this case, the global scheduler can retrieve inference requests from the request queue and, based on the maintained GPU load information for each first compute instance, select a first compute instance with a suitable GPU load. The retrieved inference request is then scheduled to that first compute instance, utilizing its GPU as a computing resource to perform inference computation during the Prefill phase.

[0067] It should be noted that the GPU load information of each first compute instance maintained by the global scheduler can be dynamically updated. For example, after an inference request is scheduled to a first compute instance and the first compute instance begins to execute the inference request, GPU load information can be generated based on the current GPU load of the first compute instance, and the generated GPU load information can be reported to the global scheduler. The global scheduler then updates the previously maintained GPU load information of the first compute instance based on this GPU load information.

[0068] For the aforementioned Prefill engine, in order to facilitate the maintenance, management and scheduling of the first computing instance corresponding to the Prefill engine, a local scheduler can also be deployed on each computing node or part of the resources where the Prefill engine is located, which is called the first local scheduler.

[0069] It should be noted that each first local scheduler is specifically used to maintain, manage, and schedule the first computing instance on the computing node or part of the resources where the first local scheduler is located.

[0070] The global scheduler described above can communicate with each of the first local schedulers. In this case, the global scheduler can obtain inference requests from the request queue and, based on the GPU load information of each first compute instance it maintains, select a first compute instance with a suitable GPU load. The global scheduler then sends the obtained inference request to the first local scheduler deployed on the compute node where the first compute instance is located. The first local scheduler further schedules the inference request to the first compute instance so that the GPU, which serves as the computing resource of the first compute instance, can be used to perform inference computation in the prefill phase for the inference request.

[0071] Similarly, for the Decode engine instance mentioned above, in order to facilitate the maintenance, management and scheduling of the second computing instance corresponding to the Decode engine, a local scheduler can also be deployed on each computing node or part of the resources where the Decode engine is located, which is called the second local scheduler.

[0072] It should be noted that each second local scheduler is specifically used to maintain, manage, and schedule the second computing instance on the computing node or part of the resources where the second local scheduler is located.

[0073] The first local scheduler can communicate with each of the second local schedulers. In this case, the first local scheduler can select a second computing instance with a suitable GPU load based on the GPU load information of each second computing instance it maintains, and send the inference computation results of the Prefill stage obtained by the corresponding first computing instance to the second local scheduler deployed on the computing node where the second computing instance is located. The second local scheduler then schedules the inference computation results of the Prefill stage to the second computing instance, so that the GPU, which is the computing resource of the second computing instance, can be used to perform inference computation of the Decode stage based on the inference computation results of the Prefill stage.

[0074] In practical applications, to ensure the reliability and correctness of scheduling inference requests or inference computation results in the Prefill phase, each local scheduler can maintain a Sequential Queue. This Sequential Queue can be a First-In-First-Out (FIFO) sequence of inference requests or inference computation results in the Prefill phase, meaning that inference requests or inference computation results in the Sequential Queue are processed in the order they entered the Sequential Queue.

[0075] In addition, for the first local scheduler and each first computing instance deployed on the same computing node, the first local scheduler can monitor the GPU load of the first computing instance, generate corresponding GPU load information, and report the generated GPU load information to the global scheduler, so that the global scheduler can maintain dynamically updated GPU load information of each first computing instance.

[0076] Similarly, for the second local scheduler and each second computing instance deployed on the same computing node, the second local scheduler can monitor the GPU load of the second computing instance, generate corresponding GPU load information, and report the generated GPU load information to each first local scheduler, so that each first local scheduler can maintain dynamically updated GPU load information of each second computing instance.

[0077] Furthermore, in practical applications, the aforementioned global scheduler can contain two components: a request router component responsible for routing inference requests, and a load manager component responsible for acquiring and maintaining GPU load information for all first compute instances.

[0078] Each first local scheduler may contain three components: a request routing component responsible for routing the inference computation results of the Prefill phase obtained from the corresponding first compute instance; a load management component responsible for calculating the GPU load information of the corresponding first compute instance, obtaining and maintaining the GPU load information of the corresponding first compute instance; and a monitor component responsible for monitoring the operation-related information of the corresponding first compute instance (e.g., GPU memory utilization, GPU memory bandwidth utilization, number of inference requests waiting in the sequential queue, and number of running inference requests).

[0079] Each second local scheduler may contain two components: a load management component responsible for calculating the GPU load information of the corresponding second computing instance, and a monitor component responsible for monitoring the running-related information of the corresponding second computing instance.

[0080] Each computing instance can contain two components: a cache engine component for caching inference results and avoiding redundant computation, and an executor component responsible for calling the large model to execute the inference task corresponding to the inference request.

[0081] Please Figure 1 and Figure 2Based on, refer to Figure 3 , Figure 3 This is a flowchart illustrating a load-aware scheduling method for an inference system according to an exemplary embodiment of this application.

[0082] In this embodiment, the load-aware scheduling method of the inference system described above can be applied to, for example... Figure 1 and Figure 2 The global scheduler shown.

[0083] In some embodiments, the aforementioned inference system may specifically be a large-model-based inference system. Accordingly, the inference process of such inference system may include two stages, wherein the Prefill stage is the Prefill stage in the inference process of the large model, and the Decode stage is the Decode stage in the inference process of the large model.

[0084] like Figure 3 As shown, the load-aware scheduling method of the above-mentioned inference system may include the following steps:

[0085] Step 302: The global scheduler obtains the target inference request to be executed.

[0086] In this embodiment, the global scheduler can first obtain the inference requests to be executed (which may be referred to as target inference requests). For example, the global scheduler can obtain the inference requests in sequence from the request queue used to temporarily store all inference requests to be executed, as target inference requests.

[0087] Step 304: The global scheduler determines the first computing instance whose GPU load meets the first preset condition based on the GPU load information of each first computing instance it maintains, and sends the target inference request to the first computing instance so that the first computing instance can perform inference computation in the Prefill phase for the target inference request.

[0088] In this embodiment, as described above, the global scheduler can maintain GPU load information for each first computing instance, whereby the GPU load information can indicate the GPU load status. In this case, the global scheduler can determine the first computing instance (which may be referred to as the first target computing instance) whose GPU load meets the first preset condition based on the maintained GPU load information of each first computing instance.

[0089] In some embodiments, the first computing instance whose GPU load meets the first preset condition may be the first computing instance with the minimum CPU load.

[0090] Once the global scheduler has identified the first target computing instance, it can send the target inference request to the first target computing instance via the communication connection with the inference engine. The first target computing instance then performs the prefill phase inference computation for the target inference request. In other words, the global scheduler can schedule the inference request to be executed to the first computing instance whose GPU load meets a first preset condition for prefill phase inference computation. This enables load-aware scheduling of inference requests during the prefill phase at the cluster-level inference engine.

[0091] Step 306: The first local scheduler determines the second computing instance whose GPU load meets the second preset condition based on the GPU load information of each computing instance in the maintained second computing instance, and sends the inference computing results of the Prefill stage to the second computing instance so that the second computing instance can perform the inference computing of the Decode stage based on the inference computing results of the Prefill stage.

[0092] In this embodiment, as described above, each first local scheduler can maintain GPU load information for each second computing instance, whereby the GPU load information can indicate the GPU load status. In this case, the first local scheduler deployed on the computing node where the first target computing instance is located can determine the second computing instance (which may be referred to as the second target computing instance) whose GPU load meets the second preset condition based on the maintained GPU load information of each second computing instance.

[0093] In some embodiments, the second computing instance whose GPU load meets the second preset condition may be the second computing instance with the lowest CPU load.

[0094] Once the first local scheduler has identified the second target computing instance, it can send the inference results from the Prefill phase to the second target computing instance via a communication connection with the Decode engine. The second target computing instance then performs the inference calculations for the Decode phase based on these results. In other words, the first local scheduler corresponding to the first computing instance used to perform the inference calculations for the Prefill phase can schedule the inference results from the Prefill phase to the second computing instance whose GPU load meets the second preset condition for inference calculations in the Decode phase. This enables load-aware scheduling of inference requests for the Decode phase at the cluster-level inference engine.

[0095] By separating the Prefill and Decode phases in the inference process and performing load-aware scheduling for the Prefill and Decode phases respectively, the scheduling effect can be further improved, thereby optimizing resource utilization efficiency, reducing waiting time and processing latency, and increasing the overall throughput of the system.

[0096] In practical applications, when maintaining the GPU load information of each first computing instance, the global scheduler can associate and store the instance information of each first computing instance with the GPU load information. Thus, based on the instance information of the first target computing instance whose GPU load meets the first preset condition, the scheduler can locate the first target computing instance and send the target inference request to the first target computing instance.

[0097] Similarly, when maintaining the GPU load information of each second computing instance, the first local scheduler can associate and store the instance information of each second computing instance with the GPU load information. In this way, it can locate the second target computing instance based on the instance information of the second target computing instance whose GPU load meets the second preset condition, and send the inference calculation results of the Prefill stage to the second target computing instance.

[0098] In some embodiments, as described above, the inference engine may include at least one compute instance and a local scheduler. In this case, when sending the target inference request to the first target compute instance, the target inference request may specifically be sent to a first local scheduler deployed on the compute node where the first target compute instance resides, so that the first local scheduler sends the target inference request to the first target compute instance. When sending the inference computation result of the Prefill stage to the second target compute instance, the inference computation result of the Prefill stage may specifically be sent to a second local scheduler deployed on the compute node where the second target compute instance resides, so that the second local scheduler sends the inference computation result of the Prefill stage to the second target compute instance.

[0099] In some embodiments, in order to make the dynamic updating of the GPU load information of each first computing instance maintained by the global scheduler easy to implement and accurate and reliable, each first local scheduler can periodically collect the GPU load information of the corresponding first computing instance according to a preset time period (for example, generate GPU load information based on the current GPU load of the first computing instance), and report the collected GPU load information to the global scheduler, so that the global scheduler can update the GPU load information of each first computing instance it maintains based on the GPU load information reported by each first local scheduler.

[0100] Similarly, in order to make the dynamic updating of the GPU load information of each second computing instance maintained by the first local scheduler easy to implement and accurate and reliable, each second local scheduler can periodically collect the GPU load information of the corresponding second computing instance according to a preset time period, and report the collected GPU load information to each first local scheduler, so that each first local scheduler can update the GPU load information of each second computing instance it maintains based on the GPU load information reported by each second local scheduler.

[0101] In some embodiments, for the first computing instance described above, its GPU load information can specifically be represented as GPU resource utilization. GPU resource utilization can be an indicator of GPU load, specifically calculated by the first local scheduler based on the streaming multiprocessor activity (SM Activity) and streaming multiprocessor clock frequency (SM Clock) of the first computing instance. In this case, when determining the first target computing instance with the lowest GPU load based on the maintained GPU load information of each of the first computing instances, the first target computing instance with the lowest GPU resource utilization can be determined specifically based on the maintained GPU load information of each of the first computing instances.

[0102] Inference requests during the Prefill phase are computationally intensive tasks, and GPU utilization is primarily based on the SM activity indicator selection. The SM is the core module responsible for performing computational tasks within the GPU, thus directly impacting the speed at which the GPU executes instructions. When the GPU load is high, the SM clock frequency decreases.

[0103] In some embodiments, GPU resource utilization can be calculated based on the streaming multiprocessor activity and streaming multiprocessor clock frequency of the computing instance using the following formula:

[0104] U GPU_I =A SM ×f;

[0105]

[0106] Among them, U GPU_I A represents the GPU resource utilization of a computing instance. SM The streaming multiprocessor activity of the computing instance is represented by f, the composite coefficient, γ, the preset first control coefficient, δ, and t. ′ C represents the preset second threshold. SM Indicates the streaming multiprocessor clock frequency of the compute instance.

[0107] The GPU resource utilization rate described above can be combined with the GPU memory utilization rate and the GPU memory bandwidth utilization rate for load-aware scheduling. In practical applications, SM Clock and GPU memory utilization are negatively correlated. To avoid excessively high SM Clock values ​​leading to a significant drop in GPU performance, a threshold (the second threshold mentioned above) can be set for SM Clock. When the SM Clock value is below the second threshold, it tends to increase, resulting in higher GPU resource utilization; when the SM Clock value exceeds the second threshold, it tends to increase GPU memory utilization, resulting in lower GPU resource utilization.

[0108] The γ and δ mentioned above are control coefficients that can be obtained experimentally and are used to control the sensitivity of the SM Clock to the response.

[0109] The load on a compute instance is positively correlated with GPU resource utilization. That is, the lower the GPU resource utilization of a compute instance, the lower its load; and the higher the GPU resource utilization, the higher its load.

[0110] In some embodiments, for the aforementioned second computing instance, its GPU load information can specifically be represented as instance resource utilization. Instance resource utilization can also be an indicator of GPU load, specifically calculated by the second local scheduler based on the GPU memory utilization and GPU memory bandwidth utilization of the second computing instance. GPU memory utilization can be obtained by reading the GPU's hardware information, and GPU memory bandwidth utilization can be obtained through the NVIDIA Data Center GPU Manager (DCGM) service. In this case, when determining the second target computing instance with the lowest GPU load based on the maintained GPU load information of each second computing instance, specifically, the second target computing instance with the lowest instance resource utilization can be determined based on the maintained GPU load information of each second computing instance.

[0111] In practical applications, the GPU memory utilization rate mentioned above can be interpreted as the utilization rate of memory blocks within the GPU. A block refers to a group of threads that can be organized together for easier management and communication. All threads within a block can collaborate to perform certain operations, and shared memory can be used to facilitate this collaboration. Each block can access a specific amount of shared memory and other resources. Therefore, the number of blocks and the number of threads within each block together determine the total amount of resources used.

[0112] It should be noted that when scheduling an inference request, if the inference request is a Prefill stage in the inference process of a large model, the aforementioned Block utilization rate can specifically be the number of Blocks required by the Prompt of the large model; if the inference request is a Decode stage in the inference process of a large model, the aforementioned Block utilization rate can specifically be the actual GPU memory utilization rate.

[0113] Inference requests during the Decode phase are a GPU memory-dependent task, therefore, GPU memory-related metrics should be prioritized when considering load metrics. Furthermore, higher GPU memory bandwidth utilization indicates more frequent data exchange between GPU memory and main memory, especially when data is offloaded from the GPU or read back from main memory to the GPU. In this case, the weight of the GPU memory bandwidth utilization metric can be increased during scheduling.

[0114] In some embodiments, instance resource utilization can be calculated based on the GPU memory utilization and GPU memory bandwidth utilization of the computing instance using the following formula:

[0115]

[0116] U Composite =U GPU ×a;

[0117]

[0118] Among them, U Instance M represents the instance resource utilization rate of a compute instance. Instance U represents the total GPU memory size of the compute instance. Composite U represents the composite load of the computation instance, Speed ​​represents the number of tokens generated by the computation instance in each iteration, and U represents the composite load of the computation instance. GPU This represents GPU memory utilization, 'a' represents the composite coefficient, 'α' represents the preset third control coefficient, 'β' represents the preset fourth control coefficient, 'α>β', 't' represents the preset third threshold, and U... BW This indicates the GPU memory bandwidth utilization rate.

[0119] The aforementioned composite load allows for the combination of GPU memory utilization and GPU memory bandwidth utilization for load-aware scheduling. In practical applications, GPU memory bandwidth utilization is negatively correlated with GPU memory utilization. To prevent a sharp decline in data access performance between GPU memory and main memory due to excessively high GPU memory bandwidth utilization, a threshold (the third threshold mentioned above) can be set for GPU memory bandwidth utilization. When GPU memory bandwidth utilization is below the threshold, the system tends to increase it, resulting in a larger composite load; conversely, when GPU memory bandwidth utilization exceeds the threshold, the system tends to increase GPU memory utilization while simultaneously decreasing it, resulting in a smaller composite load. This approach allows for improving GPU memory utilization while maintaining a lower GPU memory bandwidth utilization.

[0120] The α and β mentioned above are control coefficients that can be obtained experimentally and are used to control the sensitivity of the response to α. The value of α should be greater than the value of β, indicating that it is desirable to increase the GPU memory bandwidth utilization when the GPU memory bandwidth utilization is below a threshold.

[0121] The Speed ​​mentioned above can represent the number of new tokens generated per iteration across all inference requests. Therefore, This indicates how many iterations are needed to exhaust the remaining GPU memory of a compute instance at the current token generation rate. Therefore, a lower instance resource utilization rate means more iterations the remaining GPU memory can support, resulting in a lower compute instance load; conversely, a higher instance resource utilization rate means fewer iterations the remaining GPU memory can support, resulting in a higher compute instance load. In other words, the compute instance load is positively correlated with instance resource utilization rate.

[0122] It's important to clarify that the inference process of large models refers to the process of using a pre-trained large-scale machine learning model to generate corresponding output results based on given input data (i.e., the data contained in the inference request). In the field of natural language processing, especially in large language models, the inference process typically involves generating a continuous sequence of text based on a given input prompt, generating a token in each iteration. Here, a "token" usually refers to a series of small units into which text is segmented in natural language processing, such as words or characters, depending on the model's segmentation method. Specifically, a text, called a Prompt, can be provided to the model as input. The model receives the Prompt and transforms it into a form that it can process internally, typically converting each word into its corresponding numerical representation (e.g., WordEmbedding). Based on the current Prompt, the model uses complex calculations (e.g., multi-layer neural network operations) to predict the next most likely token. The model adds the predicted token to the existing Prompt to form a new Prompt. The aforementioned steps can be repeated multiple times, with each repetition called an iteration. In each iteration, the model predicts the next token based on the latest Prompt until a set stopping condition is met, such as generating a certain number of tokens or encountering a specific end marker.

[0123] In some embodiments, the GPU load information of the second computing instance may further include the GPU memory utilization rate. When the first local scheduler determines the second target computing instance with the lowest instance resource utilization rate based on the GPU load information of each maintained second computing instance, it can first determine computing instances (candidate computing instances) with GPU memory utilization rates lower than a preset threshold (referred to as the first threshold) based on the GPU load information of each maintained second computing instance. Subsequently, the candidate computing instances can be sorted according to their instance resource utilization rates to determine the second target computing instance with the lowest instance resource utilization rate from among the candidate computing instances. This reduces the number of times the first local scheduler compares the GPU load of different second computing instances when determining the second target computing instance with the lowest GPU load based on the GPU load information of each maintained second computing instance, improving the selection efficiency of the second target computing instance and thus improving scheduling efficiency.

[0124] In some embodiments, when a compute instance is in the process of termination, the instance resource utilization of the compute instance can be set to infinity, thereby avoiding scheduling inference requests to compute instances in the process of termination.

[0125] In the above technical solution, the inference engine in the inference system can include a Prefill engine and a Decode engine. The Prefill engine can include at least one first computing instance, and the Decode engine can include at least one second computing instance. The computing resources of each computing instance can include the GPU mounted on its computing node. The global scheduler in the inference system can maintain dynamically updated GPU load information of each first computing instance. When a target inference request to be executed is obtained, it can first determine the first computing instance whose GPU load meets the first preset condition based on the maintained GPU load information of each first computing instance, and then send the target inference request to the first computing instance to perform the inference computation of the Prefill stage, thereby realizing the scheduling of the Prefill stage in the inference process. The first local scheduler corresponding to the first computing instance in the inference system can maintain dynamically updated GPU load information of each second computing instance. When the first computing instance calculates the inference computation result of the Prefill stage, it can first determine the second computing instance whose GPU load meets the second preset condition based on the maintained GPU load information of each second computing instance, and then send the inference computation result of the Prefill stage to the second computing instance to perform the inference computation of the Decode stage, thereby realizing the scheduling of the Decode stage in the inference process.

[0126] By adopting the above approach, on the one hand, by adding a global scheduler to the inference system, inference requests can be scheduled to compute instances with appropriate GPU loads within the inference engine for execution. This achieves load-aware scheduling of inference requests at the cluster-level inference engine, thereby optimizing GPU resource utilization efficiency, reducing waiting time and processing latency for inference request execution, and improving the overall throughput of the inference system. On the other hand, performing phased load-aware scheduling of inference requests at the cluster-level inference engine based on different stages of the inference process can further improve scheduling effectiveness. Furthermore, scheduling inference requests to the inference engine via a global scheduler, rather than scheduling them directly within the inference engine, decouples the scheduling method from specific inference engines, providing a degree of versatility and scalability.

[0127] Corresponding to the embodiments of the methods described above, this application also provides embodiments of the apparatus.

[0128] Please refer to Figure 4 , Figure 4This is a schematic diagram illustrating the structure of a device according to an exemplary embodiment of this application. At the hardware level, the device includes a processor 402, an internal bus 404, a network interface 406, memory 408, and non-volatile memory 410, and may also include other necessary hardware. One or more embodiments of this application can be implemented in software, for example, the processor 402 reads the corresponding computer program from the non-volatile memory 410 into memory 408 and then runs it. Of course, besides software implementation, one or more embodiments of this application do not exclude other implementation methods, such as logic devices or a combination of hardware and software, etc. That is to say, the execution entity of the following processing flow is not limited to individual logic modules, but can also be hardware or logic devices.

[0129] Please refer to Figure 5 , Figure 5 This is a block diagram illustrating an exemplary embodiment of the present application of an inference system.

[0130] The above reasoning system can be applied to Figure 4 The device shown is used to implement the technical solution of this application. The inference system includes a global scheduler 502 and an inference engine 504; the inference engine includes a prefill engine and a decode engine; the prefill engine is deployed on at least one first computing instance, and the decode engine is deployed on at least one second computing instance; the computing resources of the first and second computing instances include the GPUs mounted on their respective computing nodes; the global scheduler maintains dynamically updated GPU load information for each first computing instance; the inference system also includes a first local scheduler; the first local scheduler maintains dynamically updated GPU load information for each second computing instance.

[0131] The global scheduler is used for:

[0132] Obtain the target inference request to be executed;

[0133] Based on the GPU load information of each of the maintained first computing instances, the first computing instance whose GPU load meets the first preset condition is determined, and the target inference request is sent to the first computing instance so that the first computing instance can perform inference computation in the prefill stage for the target inference request.

[0134] The first local scheduler is used for:

[0135] Based on the GPU load information of each computing instance in the maintained second computing instance, a second computing instance whose GPU load meets the second preset condition is determined, and the inference computing results of the prefill stage are sent to the second computing instance so that the second computing instance can perform the inference computing of the Decode stage based on the inference computing results of the prefill stage.

[0136] In some embodiments, the first preset condition is that the GPU load in the first computing instance is minimal; and / or,

[0137] The second preset condition is that the GPU load in the second computing instance is minimal.

[0138] In some embodiments, the Decode engine further includes a second local scheduler;

[0139] Sending the target inference request to the first computing instance includes:

[0140] The target inference request is sent to the first local scheduler, so that the first local scheduler can send the target inference request to the first computing instance;

[0141] Sending the inference calculation results of the Prefill stage to the second calculation instance includes:

[0142] The inference computation results of the Prefill phase are sent to the second local scheduler, so that the second local scheduler can send the inference computation results of the Prefill phase to the second computing instance.

[0143] In some embodiments, the global scheduler is further configured to:

[0144] The system obtains the GPU load information of each first computing instance periodically sent by the first local scheduler according to a preset first time period, and updates the GPU load information of each first computing instance based on the obtained GPU load information.

[0145] The first local scheduler is also used for:

[0146] The system obtains the GPU load information of each second computing instance periodically sent by the second local scheduler according to a preset second time period, and updates the GPU load information of each second computing instance based on the obtained GPU load information.

[0147] In some embodiments, the GPU load information of the first computing instance includes GPU resource utilization; wherein the GPU resource utilization is an indicator used to indicate GPU load calculated by the first local scheduler based on the streaming multiprocessor activity and streaming multiprocessor clock frequency of the first computing instance.

[0148] The step of determining the first computing instance whose GPU load meets the first preset condition based on the maintained GPU load information of each first computing instance includes:

[0149] Based on the GPU load information of each maintained first computing instance, the first computing instance with the lowest GPU resource utilization is determined.

[0150] In some embodiments, the GPU load information of the second computing instance includes instance resource utilization; wherein the instance resource utilization is an indicator used to indicate GPU load calculated by the second local scheduler based on the GPU memory utilization and GPU memory bandwidth utilization of the second computing instance.

[0151] The step of determining the second computing instance whose GPU load meets the second preset condition based on the maintained GPU load information of each second computing instance includes:

[0152] Based on the GPU load information of each maintained second computing instance, the second computing instance with the lowest instance resource utilization is determined.

[0153] In some embodiments, the GPU load information of the second computing instance also includes GPU memory utilization;

[0154] The process of determining the second computing instance with the lowest instance resource utilization based on the maintained GPU load information of each second computing instance includes:

[0155] Based on the GPU load information of each of the maintained second computing instances, candidate computing instances whose GPU memory utilization is less than a preset first threshold are determined.

[0156] The candidate computing instances are sorted according to the instance resource utilization rate to determine the second computing instance with the lowest instance resource utilization rate from the candidate computing instances.

[0157] In some embodiments, GPU resource utilization is calculated based on the streaming multiprocessor activity and streaming multiprocessor clock frequency of the computing instance using the following formula:

[0158] U GPU_I =A SM ×f;

[0159]

[0160] Among them, U GPU_I A represents the GPU resource utilization of a computing instance. SM The streaming multiprocessor activity of the computing instance is represented by f, the composite coefficient, γ, the preset first control coefficient, δ, and t. ′ C represents the preset second threshold. SM Indicates the streaming multiprocessor clock frequency of the compute instance.

[0161] In some embodiments, instance resource utilization is calculated based on the GPU memory utilization and GPU memory bandwidth utilization of the computing instance using the following formula:

[0162]

[0163] U Composite =U GPU ×a;

[0164]

[0165] Among them, U Instance M represents the instance resource utilization rate of a compute instance. Instance U represents the total GPU memory size of the compute instance. Composite U represents the composite load of the computation instance, Speed ​​represents the number of tokens generated by the computation instance in each iteration, and U represents the composite load of the computation instance. GPU This represents the GPU memory utilization of the computing instance, where 'a' represents the composite coefficient, α represents the preset third control coefficient, β represents the preset fourth control coefficient, α>β, t represents the preset third threshold, and U BW This represents the GPU memory bandwidth utilization of the compute instance.

[0166] In some embodiments, the instance resource utilization of a compute instance is set to infinity when the compute instance is in the process of terminating.

[0167] For the device embodiments, they basically correspond to the method embodiments; therefore, relevant details can be found in the descriptions of the method embodiments. The device embodiments described above are merely illustrative. The modules described as separate components may or may not be physically separate. The components shown as modules may or may not be physical modules; that is, they may be located in one place or distributed across multiple network modules. Some or all of the modules can be selected to achieve the technical objectives of this application according to actual needs.

[0168] The systems, devices, modules, or units described in the above embodiments can be implemented by computer chips or physical entities, or by products with certain functions. A typical implementation device is a computer, which can take the form of a personal computer, laptop computer, cellular phone, camera phone, smartphone, personal digital assistant, media player, navigation device, email sending and receiving device, game console, tablet computer, wearable device, or any combination of these devices.

[0169] In a typical configuration, a computer includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0170] Memory may include non-persistent storage in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media.

[0171] Computer-readable media, including both permanent and non-permanent, removable and non-removable media, can store information using any method or technology. Information can be computer-readable instructions, data structures, program modules, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, disk storage, quantum memory, graphene-based storage media or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transient computer-readable media, such as modulated data signals and carrier waves.

[0172] It should be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0173] The foregoing has described specific embodiments of this application. Other embodiments are within the scope of this application. In some cases, the actions or steps described in this application may be performed in a different order than those shown in the embodiments and still achieve the desired results. Furthermore, the processes depicted in the accompanying drawings do not necessarily require a specific or sequential order to achieve the desired results. In some implementations, multitasking and parallel processing are also possible or may be advantageous.

[0174] The terminology used in one or more embodiments of this application is for the purpose of describing particular embodiments only and is not intended to limit the scope of one or more embodiments of this application. The singular forms “a,” “the,” and “the” are also intended to include the plural forms unless the context clearly indicates otherwise. The term “and / or” refers to and includes any or all possible combinations of one or more associated listed items.

[0175] The terms "an embodiment," "some embodiments," "example," "specific example," or "one implementation," as used in one or more embodiments of this application, refer to specific features or characteristics described in connection with that embodiment, which are included in at least one embodiment of this application. Illustrative descriptions of these terms do not necessarily refer to the same embodiment. Furthermore, the described specific features or characteristics may be combined in a suitable manner in one or more embodiments of this application. In addition, different embodiments and specific features or characteristics from different embodiments may be combined without contradiction.

[0176] It should be understood that although the terms first, second, third, etc., may be used to describe various information in one or more embodiments of this application, such information should not be limited to these terms. These terms are only used to distinguish information of the same type from one another. For example, without departing from the scope of one or more embodiments of this application, first information may also be referred to as second information, and similarly, second information may also be referred to as first information. Depending on the context, the word "if" as used herein may be interpreted as "when," "when," or "in response to a determination."

[0177] The above description is merely a preferred embodiment of one or more embodiments of this application and is not intended to limit the scope of one or more embodiments of this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of one or more embodiments of this application should be included within the protection scope of one or more embodiments of this application.

[0178] The user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties. Furthermore, the collection, use and processing of the relevant data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and corresponding operation entry points are provided for users to choose to authorize or refuse.

Claims

1. A load-aware scheduling method for inference systems, applied to inference systems; The inference system includes a global scheduler and an inference engine; the inference engine includes a prefill engine and a decode engine. The Prefill engine is deployed on at least one first compute instance, and the Decode engine is deployed on at least one second compute instance; the computing resources of the first and second compute instances include the GPUs mounted on their respective compute nodes; the global scheduler maintains dynamically updated GPU load information for each first compute instance; the inference system also includes a first local scheduler; the first local scheduler maintains dynamically updated GPU load information for each second compute instance. The method includes: The global scheduler acquires the target inference request to be executed; The global scheduler determines the first computing instance whose GPU load meets the first preset condition based on the GPU load information of each first computing instance it maintains, and sends the target inference request to the first computing instance so that the first computing instance can perform inference computation in the Prefill phase for the target inference request. The first local scheduler determines the second computing instance whose GPU load meets the second preset condition based on the GPU load information of each computing instance in the maintained second computing instance, and sends the inference computing results of the Prefill stage to the second computing instance so that the second computing instance can perform the inference computing of the Decode stage based on the inference computing results of the Prefill stage.

2. The method according to claim 1, wherein the first preset condition is that the GPU load in the first computing instance is minimal; and / or, The second preset condition is that the GPU load in the second computing instance is minimal.

3. The method according to claim 1, wherein the inference system further comprises a second local scheduler; Sending the target inference request to the first computing instance includes: The target inference request is sent to the first local scheduler, so that the first local scheduler can send the target inference request to the first computing instance; Sending the inference calculation results of the Prefill stage to the second calculation instance includes: The inference computation results of the Prefill phase are sent to the second local scheduler, so that the second local scheduler can send the inference computation results of the Prefill phase to the second computing instance.

4. The method according to claim 3, further comprising: The global scheduler obtains the GPU load information of each first computing instance periodically sent by the first local scheduler according to a preset first time period, and updates the GPU load information of each first computing instance maintained based on the obtained GPU load information. The first local scheduler obtains the GPU load information of each second computing instance periodically sent by the second local scheduler according to a preset second time period, and updates the GPU load information of each second computing instance maintained based on the obtained GPU load information.

5. The method according to claim 1, wherein the GPU load information of the first computing instance includes GPU resource utilization; wherein, The GPU resource utilization rate is an indicator used to indicate GPU load, calculated by the first local scheduler based on the streaming multiprocessor activity and streaming multiprocessor clock frequency of the first computing instance. The step of determining the first computing instance whose GPU load meets the first preset condition based on the maintained GPU load information of each first computing instance includes: Based on the GPU load information of each of the maintained first computing instances, the first computing instance with the lowest GPU resource utilization is determined.

6. The method according to claim 3, wherein the GPU load information of the second computing instance includes instance resource utilization; wherein, The instance resource utilization rate is an indicator used to indicate GPU load, calculated by the second local scheduler based on the GPU memory utilization and GPU memory bandwidth utilization of the second computing instance. The step of determining the second computing instance whose GPU load meets the second preset condition based on the maintained GPU load information of each second computing instance includes: Based on the GPU load information of each maintained second computing instance, the second computing instance with the lowest instance resource utilization is determined.

7. The method according to claim 6, wherein the GPU load information of the second computing instance further includes GPU memory utilization; The process of determining the second computing instance with the lowest instance resource utilization based on the maintained GPU load information of each second computing instance includes: Based on the GPU load information of each of the maintained second computing instances, candidate computing instances whose GPU memory utilization is less than a preset first threshold are determined. The candidate computing instances are sorted according to the instance resource utilization rate to determine the second computing instance with the lowest instance resource utilization rate from the candidate computing instances.

8. The method according to claim 5, wherein GPU resource utilization is calculated based on the streaming multiprocessor activity and streaming multiprocessor clock frequency of the computing instance using the following formula: U GPU_I =A SM ×f; in, U GPU_I A represents the GPU resource utilization of a computing instance. SM The streaming multiprocessor activity of the computing instance is represented by f, the composite coefficient, γ, the preset first control coefficient, δ, the preset second control coefficient, t′, and the preset second threshold. SM Indicates the streaming multiprocessor clock frequency of the compute instance.

9. The method according to claim 6, wherein the instance resource utilization is calculated based on the GPU memory utilization and GPU memory bandwidth utilization of the computing instance using the following formula: IN Composite =U GPU ×a; in, U Instance M represents the instance resource utilization rate of a compute instance. Instance U represents the total GPU memory size of the compute instance. Composite U represents the composite load of the computation instance, Speed ​​represents the number of tokens generated by the computation instance in each iteration, and U represents the composite load of the computation instance. GPU This represents the GPU memory utilization of the computing instance, where 'a' represents the composite coefficient, 'α' represents the preset third control coefficient, 'β' represents the preset fourth control coefficient, 'α > β', and 't' represents the preset third threshold. BW This represents the GPU memory bandwidth utilization of the compute instance.

10. The method of claim 9, wherein the instance resource utilization of the computing instance is set to infinity when the computing instance is in the process of termination.

11. An inference system, the inference system comprising a global scheduler and an inference engine; the inference engine comprising a prefill engine and a decode engine; the prefill engine being deployed on at least one first computing instance, and the decode engine being deployed on at least one second computing instance; the computing resources of the first and second computing instances comprising GPUs mounted on their respective computing nodes; the global scheduler maintaining dynamically updated GPU load information for each of the first computing instances; the inference system further comprising a first local scheduler; the first local scheduler maintaining dynamically updated GPU load information for each of the second computing instances; The global scheduler is used for: Obtain the target inference request to be executed; Based on the GPU load information of each of the maintained first computing instances, the first computing instance whose GPU load meets the first preset condition is determined, and the target inference request is sent to the first computing instance so that the first computing instance can perform inference computation in the prefill stage for the target inference request. The first local scheduler is used for: Based on the GPU load information of each computing instance in the maintained second computing instance, a second computing instance whose GPU load meets the second preset condition is determined, and the inference computing results of the prefill stage are sent to the second computing instance so that the second computing instance can perform the inference computing of the Decode stage based on the inference computing results of the prefill stage.

12. An electronic device, comprising: processor; Memory used to store processor-executable instructions; The processor implements the method as described in any one of claims 1 to 10 by executing the executable instructions.

13. A computer-readable storage medium having stored thereon computer instructions that, when executed by a processor, implement the method as described in any one of claims 1 to 10.

Images