Load-aware scheduling method of inference system and inference system

Abstract

One or more embodiments of the invention provide a load-aware scheduling method of an inference system and the inference system. The method is applied to a global scheduler in the inference system. The inference system further comprises an inference engine. The inference engine comprises at least one computing instance deployed on each computing node in a computing cluster; computing resources of the computing examples comprise GPUs carried on the computing nodes where the computing examples are located; the global scheduler maintains dynamically updated GPU load information of each calculation instance; the method comprises the steps of obtaining a to-be-executed target reasoning request; based on the maintained GPU load information of each calculation instance, determining a target calculation instance of which the GPU load meets a preset condition; and sending the target reasoning request to the target calculation instance, so that the target calculation instance executes the target reasoning request.

Description

Technical Field

[0001] One or more embodiments of the present application relate to the field of artificial intelligence technology, and in particular, to a load-aware scheduling method for an inference system and an inference system. Background Art

[0002] An inference system is a computer program that uses logical rules and known facts to draw new conclusions or decisions. An inference system is an important part of the field of artificial intelligence, mainly used to simulate the human decision-making process. It derives conclusions based on a set of defined knowledge bases and inference engines. An inference system can execute the inference request it obtains and output the corresponding inference results.

[0003] A typical reasoning system usually consists of the following parts: Knowledge Base, Inference Engine, User Interface, and Explanation Facility. The knowledge base includes all the facts and rules known to the storage system. These facts can be about the state of the world, object properties, etc., while the rules are logical expressions that describe how to draw new conclusions from known facts. The inference engine is the core component of the reasoning system. It is responsible for performing logical operations in the reasoning process, that is, drawing new conclusions or decisions from a given knowledge base; the inference engine uses a series of rules and known facts to derive new knowledge to help the system solve problems or make decisions. The user interface allows users to interact with the system, input queries, or observe the results of the reasoning process. The explanation mechanism is used to explain how the system draws specific conclusions, which is very important for transparency and trust.

[0004] Inference engines are usually deployed on computing clusters when large-scale data, high-concurrency requests, or high-performance computing are required. By deploying inference engines on computing clusters, higher computing power, better fault tolerance, and more flexible resource management can be achieved. However, this introduces the issue of how to schedule inference requests on cluster-level inference engines. Summary of the invention

[0005] One or more embodiments of the present application provide the following technical solutions: The present application provides a load-aware scheduling method for an inference system, which is applied to a global scheduler in an inference system; the inference system further includes an inference engine; the inference engine includes at least one computing instance deployed on each computing node in a computing cluster; the computing resources of the computing instance include a GPU mounted on the computing node; the global scheduler maintains GPU load information of each computing instance that is dynamically updated; The method comprises: Get the target inference request to be executed; Based on the GPU load information of each computing instance maintained, determine a target computing instance whose GPU load meets a preset condition; The target reasoning request is sent to the target computing instance so that the target computing instance executes the target reasoning request.

[0006] The present application also provides an inference system, the inference system comprising a global scheduler and an inference engine; the inference engine comprises at least one computing instance deployed on each computing node in a computing cluster; the computing resources of the computing instance comprise a GPU mounted on the computing node; the global scheduler maintains dynamically updated GPU load information of each computing instance; The global scheduler is used to: Get the target inference request to be executed; Based on the GPU load information of each computing instance maintained, determine a target computing instance whose GPU load meets a preset condition; The target reasoning request is sent to the target computing instance so that the target computing instance executes the target reasoning request.

[0007] The present application also provides an electronic device, including: processor; a memory for storing processor-executable instructions; The processor implements the steps of any of the above methods by running the executable instructions.

[0008] The present application also provides a computer-readable storage medium having computer instructions stored thereon, which implement the steps of any of the above methods when executed by a processor.

[0009] In the above technical scheme, the inference engine in the inference system may include at least one computing instance deployed on each computing node in the computing cluster. The computing resources of each computing instance may include a GPU mounted on the node. The global scheduler in the inference system can maintain dynamically updated GPU load information of each computing instance, and when a target inference request to be executed is obtained, it can first determine the target computing instance whose GPU load meets the preset conditions based on the GPU load information of each computing instance maintained, and then send the target inference request to the target computing instance so that the target computing instance can execute the target inference request, thereby realizing scheduling for the target inference request.

[0010] By adopting the above method, by adding a global scheduler to the inference system, the global scheduler can schedule the inference request to the computing instance with appropriate GPU load in the inference engine for execution, realizing load-aware scheduling of inference requests on the cluster-level inference engine, thereby optimizing the efficiency of GPU resource utilization, reducing the waiting time and processing delay of inference request execution, and improving the overall throughput of the inference system. In addition, by scheduling the inference request to the inference engine through the global scheduler instead of directly scheduling the inference request in the inference engine, the scheduling method can be decoupled from the specific inference engine, which has certain versatility and scalability. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] The following is a description of the accompanying drawings required for describing the exemplary embodiments, wherein: Figure 1 It is a schematic diagram of an inference system shown in an exemplary embodiment of the present application.

[0012] Figure 2 It is a schematic diagram of another reasoning system shown in an exemplary embodiment of the present application.

[0013] Figure 3 It is a flowchart of a load-aware scheduling method for an inference system shown in an exemplary embodiment of the present application.

[0014] Figure 4 It is a structural schematic diagram of a device shown in an exemplary embodiment of the present application.

[0015] Figure 5 It is a block diagram of an inference system shown in an exemplary embodiment of the present application. DETAILED DESCRIPTION

[0016] Here, exemplary embodiments will be described in detail, and examples thereof are shown in the accompanying drawings. When the following description relates to the accompanying drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with one or more embodiments of the present application. On the contrary, they are only examples consistent with some aspects of one or more embodiments of the present application.

[0017] It should be noted that in other embodiments, the steps of the corresponding method are not necessarily performed in the order shown and described in this application. In some other embodiments, the steps included in the method may be more or less than those described in this application. In addition, a single step described in this application may be decomposed into multiple steps for description in other embodiments; and multiple steps described in this application may also be combined into a single step for description in other embodiments.

[0018] In this application, the inference engine in the inference system can be deployed on a computing cluster to meet the needs of processing large-scale data, high-concurrency requests or high-performance computing, thereby achieving higher computing power, better fault tolerance and more flexible resource management. In this type of inference system, load-aware scheduling can be used to schedule inference requests on the cluster-level inference engine.

[0019] Load-aware scheduling is a strategy for allocating tasks in a computing system based on the current load of each node or resource. This scheduling method aims to optimize resource usage efficiency, reduce waiting time and processing delays, and improve the overall throughput of the system.

[0020] Load-aware scheduling is usually used in distributed systems, cloud computing platforms, data centers and other scenarios. Its purpose is to dynamically balance the workload of each node and avoid the situation where some nodes are overloaded while other nodes are idle. By monitoring the load of each node in real time, the load-aware scheduler can make more reasonable task allocation decisions.

[0021] In general, the inference engine can use the CPU as a computing resource, where the CPU can be the CPU installed on the device used to deploy the inference engine. The CPU load indicators usually select the CPU usage and memory usage. That is, for the inference engine deployed on the computing cluster, by calculating the CPU usage and memory usage to represent the CPU load, load-aware scheduling of inference requests can be achieved on such inference engines.

[0022] With the development of artificial intelligence technology, reasoning systems based on large models (such as Large Language Model) are becoming more and more widely used.

[0023] Large models refer to machine learning models with a large number of parameters, such as various variants of the Transformer architecture, including but not limited to natural language processing models such as GPT and BERT. These models achieve powerful representation learning capabilities through large amounts of training data and complex architectures.

[0024] In a large model-based reasoning system, the large model can be viewed as a large knowledge base that contains information learned from a large amount of data. The large model learns a large number of patterns and features during training, which represent complex relationships in the data. Therefore, to some extent, the information stored inside the large model can be viewed as a form of knowledge.

[0025] The inference engine designed for large models is responsible for calling the large model to perform specific inference work, that is, it can manage the loading of large models, execute inference operations of large models, and manage the interaction with hardware (for example: CPU, GPU or other accelerators). The inference engine can also include optimization algorithms to improve the speed and efficiency of inference.

[0026] In practical applications, the large model can be deployed separately from the reasoning system, or it can be integrated into the reasoning system and the reasoning engine in the system can be used to call the large model to efficiently perform reasoning tasks.

[0027] Since large models are artificial intelligence applications that require a lot of computing resources, the inference engine usually uses GPU as a computing resource when calling large models to perform inference tasks.

[0028] GPUs are designed for parallel processing and can process multiple data points at the same time. This is particularly useful for deep learning models, as they often need to perform the same operation on large amounts of data. Modern AI models, especially neural network models, require a large number of floating-point operations. GPUs generally perform better than CPUs on floating-point operations, especially when processing large-scale matrix operations, which are common operations in deep learning. GPUs generally have higher memory bandwidth than CPUs, which means they are able to read and write data from memory faster. This is very important for applications that need to frequently access large amounts of data. Many GPU vendors (such as NVIDIA) have optimized their hardware and software stacks for machine learning tasks. For example, they provide hardware specifically designed to accelerate tensor operations, as well as programming models like CUDA to take full advantage of these hardware features. GPUs can increase inference speed. For applications deployed at large scale, using GPUs can reduce latency and increase system responsiveness.

[0029] There are many choices for GPU load indicators. Commonly used indicators include SM Clock (Streaming Multiprocessor Clock), SM Activity (Streaming Multiprocessor Activity), Memory Utilization (Memory Usage), etc. Among them, SM Clock refers to the clock frequency of the streaming multiprocessor on the GPU; the streaming multiprocessor is the computing unit on the GPU that is responsible for performing parallel computing tasks. A higher SM Clock frequency means that the GPU's computing units run faster, which generally leads to higher computing performance, but may also increase power consumption and heat. SM Activity refers to the degree of activity of the streaming multiprocessor on the GPU, that is, what proportion of time these computing units are actually performing tasks over a period of time. If SM Activity is high, it means that the GPU's computing units are actively working most of the time; conversely, if it is low, it may mean that there is a computing bottleneck or a long waiting time. Memory Utilization refers to the usage of the GPU video memory, usually indicating the proportion of the video memory currently in use to the total video memory. Higher Memory Utilization means that more video memory is occupied, which may cause slower data exchange or affect performance due to insufficient video memory. When the video memory is close to full, it may trigger data overflow to the host memory, which will further reduce performance.

[0030] For inference engines deployed on computing clusters, load-aware scheduling of inference requests can be achieved on such inference engines by calculating GPU load indicators to represent GPU load conditions.

[0031] In the technical solution provided by one or more embodiments of the present application, the inference engine in the inference system may include at least one computing instance deployed on each computing node in the computing cluster, and the computing resources of each computing instance may include a GPU mounted on the node. The global scheduler in the inference system can maintain dynamically updated GPU load information of each computing instance, and when a target inference request to be executed is obtained, it can first determine the target computing instance whose GPU load meets preset conditions based on the GPU load information of each computing instance maintained, and then send the target inference request to the target computing instance so that the target computing instance executes the target inference request, thereby realizing scheduling for the target inference request.

[0032] By adopting the above method, by adding a global scheduler to the inference system, the global scheduler can schedule the inference request to the computing instance with appropriate GPU load in the inference engine for execution, realizing load-aware scheduling of inference requests on the cluster-level inference engine, thereby optimizing the efficiency of GPU resource utilization, reducing the waiting time and processing delay of inference request execution, and improving the overall throughput of the inference system. In addition, by scheduling the inference request to the inference engine through the global scheduler instead of directly scheduling the inference request in the inference engine, the scheduling method can be decoupled from the specific inference engine, which has certain versatility and scalability.

[0033] Please refer to Figure 1 and Figure 2 , Figure 1 and Figure 2 They are respectively schematic diagrams of an inference system shown by an exemplary embodiment of the present application.

[0034] like Figure 1 As shown, the above reasoning system may include a global scheduler and a reasoning engine. The reasoning engine may be deployed on a computing cluster (Cluster) composed of at least one computing node (Node).

[0035] A computing cluster is a system consisting of multiple computers or servers connected through a network to work together. These nodes work together to complete computing tasks, providing stronger computing power and higher availability than a single computer. Computing nodes are one of the core components of a computing cluster, usually referring to computers or servers used to perform computing-intensive tasks. Computing nodes can undertake computing tasks, and the GPUs installed on them can be used as computing resources. Among them, computers or servers can be physical or virtual local computers or local servers, or physical or virtual cloud computers or cloud servers.

[0036] In practical applications, the global scheduler may be deployed on the head node in the computing cluster, where the head node generally refers to the node responsible for the management and scheduling of the computing cluster. It may also be deployed on other computing devices outside the computing cluster, and this application does not impose any restrictions on this.

[0037] In the above reasoning system, the above global scheduler and the above reasoning engine can communicate with each other, for example, the two can directly communicate with each other through HTTP (HyperText Transfer Protocol).

[0038] Specifically, the global scheduler can communicate with the applications and components deployed on each computing node in the computing cluster. In this case, the global scheduler can obtain the inference request from the request queue, and select a computing node with a suitable GPU load according to the load of the GPU on each computing node, and schedule the obtained inference request to the computing node, so as to use the GPU on the computing node to execute the inference request.

[0039] like Figure 2 As shown, for each computing node in the above computing cluster, at least one computing instance can be deployed on the computing node. Among them, the computing instance refers to a dedicated resource allocated on demand through virtualization technology in a local data center or cloud computing platform. The resources of these computing instances may include computing resources, storage resources, network resources, etc. (for example: GPU, GPU memory, storage, network interface, etc.), so that each computing instance can use these resources to perform specific computing tasks. Computing instances can be created, started, stopped or deleted according to actual needs, and their resources can also be adjusted according to demand.

[0040] In an inference engine for large models, the computing resource of a computing instance may be a GPU of the computing node where it is located.

[0041] In actual applications, a computing node can be equipped with multiple GPUs, and a GPU can be assigned to a computing instance as a computing resource of the computing instance. Alternatively, a computing node can be equipped with only one GPU, and the computing resources assigned to a computing instance can be part of the computing unit, memory, etc. of the GPU.

[0042] In the inference engine for large models, each computing instance can call the large model to perform specific inference work, that is, the large model can be loaded into each computing instance, and each computing instance performs the inference operation of the large model. Specifically, the large model can be read from a local or cloud storage medium (e.g., local disk, cloud storage, etc.) to the GPU memory of each computing instance to perform inference based on the large model on the GPU computing resources of each computing instance.

[0043] In actual applications, the Model Runtime framework can be installed in the computing instance, and after the large model is read into the computing instance, the model service is configured according to the Model Runtime framework and the large model, so that the computing instance can be used as an independently run model service instance. Among them, the model runtime refers to the environment and framework used to perform model reasoning after the model is deployed. This environment usually includes a series of steps such as model loading, input data processing, model execution, and output result processing, covering the entire life cycle of the model from loading to execution and then unloading.

[0044] By deploying computing instances on computing nodes, on the one hand, the resources of computing nodes can be fully utilized to accelerate the execution of computing tasks, thereby improving the computing power of the inference engine; on the other hand, different computing instances are independent of each other, which can ensure the stability and security of computing tasks, and can achieve resource isolation and management, which is conducive to the effective use of resources and avoid resource requisition; on the third hand, for computing tasks that need to be processed in parallel, parallel computing can be used to speed up the completion time of computing tasks; on the fourth hand, elastic scaling can be achieved. When a large number of computing tasks need to be processed, computing instances can be added, and when the number of computing tasks that need to be processed decreases, some computing instances can be terminated to release some resources and realize flexible allocation of resources.

[0045] In order to facilitate load-aware scheduling of inference requests, the global scheduler may maintain GPU load information of each computing instance, wherein the GPU load information may indicate the load condition of the GPU. In this case, the global scheduler may obtain an inference request from the request queue, and select a computing instance with a suitable GPU load condition based on the maintained GPU load information of each computing instance, and schedule the obtained inference request to the computing instance, so as to use the GPU as the computing resource of the computing instance to execute the inference request.

[0046] It should be noted that the GPU load information of each computing instance maintained by the above-mentioned global scheduler can be dynamically updated. For example, after a certain inference request is scheduled to a certain computing instance and the computing instance starts to execute the inference request, GPU load information can be generated according to the current GPU load of the computing instance, and the generated GPU load information is reported to the global scheduler, and the global scheduler updates the GPU load information of the computing instance previously maintained based on the GPU load information.

[0047] For each computing node in the computing cluster, in order to facilitate maintenance, management and scheduling of computing instances deployed on the computing node, a local scheduler (LocalScheduler) may also be deployed on the computing node.

[0048] The global scheduler can communicate with the local schedulers deployed on each computing node in the computing cluster. In this case, the global scheduler can obtain the inference request from the request queue, and select the computing instance with the appropriate GPU load according to the GPU load information of each computing instance maintained, and send the obtained inference request to the local scheduler deployed on the computing node where the computing instance is located, and the local scheduler further schedules the inference request to the computing instance to execute the inference request using the GPU as the computing resource of the computing instance.

[0049] In practical applications, in order to ensure the reliability and correctness of inference request scheduling, each local scheduler can maintain a sequential queue, which can be a first-in-first-out (FIFO) sequence of inference requests. That is, the inference requests in the sequential queue are processed in the order in which they enter the sequential queue.

[0050] In addition, for each computing node in the above-mentioned computing cluster, the local scheduler deployed on the computing node can monitor the GPU load of each computing instance deployed on the computing node, generate corresponding GPU load information, and report the generated GPU load information to the above-mentioned global scheduler, so that the global scheduler can maintain the dynamically updated GPU load information of each computing instance.

[0051] Furthermore, in actual applications, the above-mentioned global scheduler may include two components, namely, a request router component responsible for routing inference requests, and a load manager component responsible for obtaining and maintaining GPU load information of all computing instances.

[0052] For each local scheduler, the local scheduler may include two components: a load management component responsible for calculating the GPU load information of all computing instances deployed on the computing node, and a monitor component responsible for monitoring the operation-related information of all computing instances deployed on the computing node (for example, GPU memory utilization, GPU memory bandwidth utilization, the number of inference requests waiting in the sequential queue and the number of running inference requests, etc.).

[0053] For each computing instance, the computing instance may contain two components: a cache engine component for caching the inference results to avoid repeated calculations, and an executor component responsible for calling the large model to execute the inference task corresponding to the inference request.

[0054] Please Figure 1 and Figure 2 Based on the reference Figure 3 , Figure 3 It is a flowchart of a load-aware scheduling method for an inference system shown in an exemplary embodiment of the present application.

[0055] In this embodiment, the load-aware scheduling method of the above-mentioned inference system can be applied to Figure 1 The global scheduler shown.

[0056] like Figure 3 As shown, the load-aware scheduling method of the above reasoning system may include the following steps: Step 302: Obtain a target inference request to be executed.

[0057] In this embodiment, the global scheduler may first obtain an inference request to be executed (which may be referred to as a target inference request). For example, the global scheduler may sequentially obtain the inference request from a request queue for temporarily storing all inference requests to be executed as the target inference request.

[0058] Step 304: Based on the maintained GPU load information of each computing instance, determine a target computing instance whose GPU load meets a preset condition.

[0059] In this embodiment, as described above, the global scheduler can maintain GPU load information of each computing instance, wherein the GPU load information can indicate the load of the GPU. In this case, the global scheduler can determine a computing instance (which can be referred to as a target computing instance) whose GPU load meets a preset condition based on the maintained GPU load information of each computing instance.

[0060] In some embodiments, the computing instance whose GPU load satisfies a preset condition may be the computing instance whose CPU load is the smallest.

[0061] Step 306: Send the target reasoning request to the target computing instance so that the target computing instance executes the target reasoning request.

[0062] In this embodiment, once the above-mentioned global scheduler has determined the above-mentioned target computing instance, it can send the above-mentioned target reasoning request to the target computing instance through the communication connection with the above-mentioned inference engine, so that the target reasoning request can be executed by the target computing instance. That is, the global scheduler can schedule the target reasoning request to the target computing instance whose GPU load meets the preset conditions, thereby realizing load-aware scheduling of reasoning requests on the cluster-level inference engine, thereby optimizing resource utilization efficiency, reducing waiting time and processing delay, and improving the overall throughput of the system.

[0063] In actual applications, when maintaining the GPU load information of each computing instance, the above-mentioned global scheduler can store the instance information of each computing instance in association with the GPU load information, so that the target computing instance can be located based on the instance information of the target computing instance whose GPU load meets the preset conditions, and the above-mentioned target inference request can be sent to the target computing instance.

[0064] In some embodiments, as described above, the above-mentioned inference engine may include, in addition to at least one computing instance deployed on each computing node in the above-mentioned computing cluster, a local scheduler deployed on each computing node in the computing cluster. Accordingly, when sending the above-mentioned target inference request to the above-mentioned target computing instance, the target inference request may be sent to the target local scheduler, so that the target local scheduler sends the target inference request to the target computing instance; wherein the target local scheduler is specifically a local scheduler deployed on the computing node where the target computing instance is located.

[0065] In some embodiments, in order to make the dynamic update of the GPU load information of each computing instance maintained by the above-mentioned global scheduler easy to implement, accurate and reliable, for each computing node in the above-mentioned computing cluster, the local scheduler deployed on the computing node can periodically count the GPU load information of each computing instance deployed on the computing node according to a preset time period (for example: generate GPU load information based on the current GPU load status of the computing instance), and report the counted GPU load information to the global scheduler, so that the global scheduler can update the GPU load information of each computing instance deployed on the computing node maintained based on the GPU load information reported by the local scheduler.

[0066] In some embodiments, the above-mentioned GPU load information can be specifically expressed as instance resource utilization. It should be noted that, for each computing instance, the instance resource utilization of the computing instance can be an indicator for indicating the GPU load, which can be specifically an indicator calculated by a local scheduler deployed on the computing node where the computing instance is located based on the GPU memory utilization and GPU memory bandwidth utilization of the computing instance. Among them, the GPU memory utilization can be obtained by reading the hardware information of the GPU, and the GPU memory bandwidth utilization can be obtained through the Data Center GPUManager (DCGM) service provided by NVIDIA. At this time, when determining the target computing instance whose GPU load meets the preset conditions based on the GPU load information of each computing instance maintained, the target computing instance with the smallest instance resource utilization can be determined based on each computing instance maintained.

[0067] In practical applications, the above GPU memory utilization can be the utilization of memory blocks in the GPU. Block refers to a group of threads that can be organized together for easy management and communication. All threads in a block can collaborate to perform certain operations, and shared memory can be used to facilitate such collaboration. Each block can access a specific amount of shared memory and other resources. Therefore, the number of blocks and the number of threads in each block together determine the total amount of resources used.

[0068] It should be noted that when scheduling an inference request, if the inference request is an inference request for the Prefill stage in the inference process of a large model, the above-mentioned Block usage rate may specifically be the number of Blocks required for the Prompt of the large model; if the inference request is an inference request for the Decode stage in the inference process of a large model, the above-mentioned Block usage rate may specifically be the actual GPU memory utilization.

[0069] In the reasoning process of large models (especially generative models in natural language processing), the Prefill stage and the Decode stage refer to different steps when generating text.

[0070] In the Prefill phase, the model usually generates an initial text fragment that contains the starting part of the generated sequence. The goal of the Prefill phase is to provide a good start for subsequent generation. This phase may involve using some predefined strategies or algorithms to select the most appropriate starting vocabulary or phrases. For example, when dealing with a conditional generation task, the model may first generate a part of the text as a basis based on the input conditions (such as summary generation, question answering, etc.).

[0071] The main process of generating text in the decoding stage, the model will generate new content word by word or word by word based on the existing text. In each iteration, the model predicts the next most likely word and adds it to the current text sequence. This process continues until the preset end condition is reached (for example, the maximum text length is reached, or the text end marker is generated). Various strategies may be used in the decoding stage to optimize the quality of generation, such as Top-k sampling, Top-p (also known as Nucleus Sampling) sampling and other techniques, which can help the model avoid generating content that is too mundane or meaningless.

[0072] In practical applications, the two stages of Prefill and Decode are closely linked. The Prefill stage provides the initial text basis, while the Decode stage is responsible for gradually expanding on this basis to generate a complete text output. These two stages together determine the quality and coherence of the final generated text.

[0073] In some embodiments, the instance resource utilization can be calculated based on the GPU memory utilization and GPU memory bandwidth utilization of the computing instance by the following formula: ; ; ; in, Indicates the instance resource utilization of the computing instance. Indicates the total GPU memory size of the compute instance. represents the composite load of the compute instance, represents the number of tokens generated by the computation instance in each iteration. Indicates the GPU memory utilization of the compute instance. represents the composite coefficient, represents the preset first control coefficient, represents the preset second control coefficient, , represents a preset second threshold, Indicates the GPU memory bandwidth utilization of the compute instance.

[0074] Through the above composite load, the above GPU memory utilization and the above GPU memory bandwidth utilization can be combined for load-aware scheduling. In actual application scenarios, GPU memory bandwidth utilization is negatively correlated with GPU memory utilization. In order to prevent the data access performance between GPU memory and main memory from dropping sharply due to excessive GPU memory bandwidth utilization, a threshold (i.e., the above second threshold) can be set for GPU memory bandwidth utilization. When the GPU memory bandwidth utilization is lower than the threshold, the GPU memory bandwidth utilization tends to be increased to obtain a larger composite load; when the GPU memory bandwidth utilization exceeds the threshold, the GPU memory utilization tends to be increased while the GPU memory bandwidth utilization is reduced to obtain a smaller composite load. In this way, the GPU memory utilization can be improved while maintaining a low GPU memory bandwidth utilization.

[0075] Above and These are control coefficients that can be obtained through experiments and are used to control response sensitivity. The value should be greater than The value of indicates that you want to increase the GPU memory bandwidth utilization when the GPU memory bandwidth utilization is below the threshold.

[0076] Above It can represent the number of new tokens generated by all inference requests in each iteration (TokensGenerated Per Iteration). Therefore, It can be expressed as how many iterations are needed to consume the remaining GPU memory of the computing instance according to the current token generation rate. It can be seen that the smaller the instance resource utilization of the computing instance, the more iterations the remaining GPU memory of the computing instance can support, and the smaller the load of the computing instance; the larger the instance resource utilization of the computing instance, the fewer iterations the remaining GPU memory of the computing instance can support, and the larger the load of the computing instance. In other words, the load size of the computing instance is positively correlated with the instance resource utilization.

[0077] It should be noted that the reasoning process of a large model refers to the process of using a trained large machine learning model to generate corresponding output results based on given input data (that is, the data contained in the reasoning request). In the field of natural language processing, especially in large language models, the reasoning process usually involves generating a continuous text sequence based on a given input prompt, and generating a token for each iteration. The "token" here usually refers to a series of small units into which text is segmented in natural language processing, such as words or characters, depending on the word segmentation method of the model. Specifically, a piece of text can be provided to the model as input, which is called a Prompt; the model receives the Prompt and converts it into a form that can be processed internally, usually by converting each word into a corresponding digital representation (for example, WordEmbedding); based on the current Prompt, the model predicts the next most likely Token through complex calculations (for example, multi-layer neural network operations); the model adds the predicted Token to the existing Prompt to form a new Prompt; the above steps can be repeated multiple times, and each repetition is called an iteration. In each iteration, the model predicts the next Token based on the latest Prompt until the set stop condition is reached, such as a certain number of Tokens are generated or a specific end mark is encountered.

[0078] In some embodiments, the GPU load information may also include the GPU memory utilization. When the global scheduler determines the target computing instance with the lowest instance resource utilization based on the GPU load information of each computing instance maintained, it may first determine the computing instance (referred to as the candidate computing instance) whose GPU memory utilization is less than a preset threshold (referred to as the first threshold) based on the GPU load information of each computing instance maintained, and then sort the candidate computing instances according to the instance resource utilization to determine the target computing instance with the lowest instance resource utilization from the candidate computing instances. In this way, when the global scheduler determines the target computing instance whose GPU load meets the preset conditions based on the GPU load information of each computing instance maintained, the number of comparisons of the size of the GPU loads of different computing instances can be reduced, thereby improving the selection efficiency of the target computing instance and thus improving the scheduling efficiency.

[0079] In some embodiments, when a computing instance is in the process of being terminated, the instance resource utilization of the computing instance may be set to infinity, thereby avoiding scheduling an inference request to the computing instance that is in the process of being terminated.

[0080] In the above technical scheme, the inference engine in the inference system may include at least one computing instance deployed on each computing node in the computing cluster. The computing resources of each computing instance may include a GPU mounted on the node. The global scheduler in the inference system can maintain dynamically updated GPU load information of each computing instance, and when a target inference request to be executed is obtained, it can first determine the target computing instance whose GPU load meets the preset conditions based on the GPU load information of each computing instance maintained, and then send the target inference request to the target computing instance so that the target computing instance can execute the target inference request, thereby realizing scheduling for the target inference request.

[0081] By adopting the above method, by adding a global scheduler to the inference system, the global scheduler can schedule the inference request to the computing instance with appropriate GPU load in the inference engine for execution, realizing load-aware scheduling of inference requests on the cluster-level inference engine, thereby optimizing the efficiency of GPU resource utilization, reducing the waiting time and processing delay of inference request execution, and improving the overall throughput of the inference system. In addition, by scheduling the inference request to the inference engine through the global scheduler instead of directly scheduling the inference request in the inference engine, the scheduling method can be decoupled from the specific inference engine, which has certain versatility and scalability.

[0082] Corresponding to the aforementioned method embodiments, the present application also provides device embodiments.

[0083] Please refer to Figure 4, Figure 4 It is a structural diagram of a device shown in an exemplary embodiment of the present application. At the hardware level, the device includes a processor 402, an internal bus 404, a network interface 406, a memory 408 and a non-volatile memory 410, and of course may also include other required hardware. One or more embodiments of the present application can be implemented based on software, such as the processor 402 reading the corresponding computer program from the non-volatile memory 410 into the memory 408 and then running it. Of course, in addition to the software implementation, one or more embodiments of the present application do not exclude other implementations, such as logic devices or a combination of software and hardware, etc., that is to say, the execution subject of the following processing flow is not limited to each logic module, but can also be hardware or logic devices.

[0084] Please refer to Figure 5 , Figure 5 It is a block diagram of an inference system shown in an exemplary embodiment of the present application.

[0085] The above reasoning system can be applied to Figure 4 The device shown in the figure is used to implement the technical solution of the present application. The inference system includes a global scheduler 502 and an inference engine 504; the inference engine includes at least one computing instance deployed on each computing node in the computing cluster; the computing resources of the computing instance include the GPU mounted on the computing node; the global scheduler maintains the dynamically updated GPU load information of each computing instance.

[0086] The global scheduler is used to: Get the target inference request to be executed; Based on the GPU load information of each computing instance maintained, determine a target computing instance whose GPU load meets a preset condition; The target reasoning request is sent to the target computing instance so that the target computing instance executes the target reasoning request.

[0087] In some embodiments, the preset condition is that the GPU load in the computing instance is minimal.

[0088] In some embodiments, the inference engine further includes a local scheduler deployed on each computing node in the computing cluster; The sending the target reasoning request to the target computing instance includes: The target inference request is sent to a target local scheduler, so that the target local scheduler sends the target inference request to the target computing instance; wherein the target local scheduler is a local scheduler deployed on a computing node where the target computing instance is located.

[0089] In some embodiments, the global scheduler is further configured to: Obtain GPU load information of each computing instance on each computing node periodically sent by a local scheduler on each computing node according to a preset time period; The GPU load information of each computing instance maintained is updated based on the acquired GPU load information.

[0090] In some embodiments, the GPU load information includes instance resource utilization; wherein the instance resource utilization is an indicator for indicating the GPU load calculated by the local scheduler based on the GPU memory utilization and GPU memory bandwidth utilization of the computing instance; The step of determining a target computing instance whose GPU load satisfies a preset condition based on the maintained GPU load information of each computing instance includes: Based on the maintained GPU load information of each computing instance, a target computing instance with the minimum instance resource utilization is determined.

[0091] In some embodiments, the GPU load information also includes GPU memory utilization; The determining, based on the GPU load information of each computing instance maintained, the target computing instance with the minimum instance resource utilization rate comprises: Determine, based on the maintained GPU load information of each computing instance, a candidate computing instance whose GPU memory utilization is less than a preset first threshold; The candidate computing instances are sorted according to the instance resource utilization rates to determine a target computing instance with the minimum instance resource utilization rate from the candidate computing instances.

[0092] In some embodiments, the instance resource utilization is calculated based on the GPU memory utilization and GPU memory bandwidth utilization of the computing instance by the following formula: ; ; ; in, Indicates the instance resource utilization of the computing instance. Indicates the total GPU memory size of the compute instance. represents the composite load of the compute instance, represents the number of tokens generated by the computation instance in each iteration. Indicates the GPU memory utilization of the compute instance. represents the composite coefficient, represents the preset first control coefficient, represents the preset second control coefficient, , represents a preset second threshold, Indicates the GPU memory bandwidth utilization of the compute instance.

[0093] In some embodiments, the instance resource utilization of a computing instance is set to infinity if the computing instance is in the process of terminating.

[0094] For the device embodiment, it basically corresponds to the method embodiment, so the relevant parts can refer to the partial description of the method embodiment. The device embodiment described above is only schematic, wherein the modules described as separate components may or may not be physically separated, and the components displayed as modules may or may not be physical modules, that is, they may be located in one place, or they may be distributed on multiple network modules. Some or all of the modules may be selected according to actual needs to achieve the purpose of the technical solution of the present application.

[0095] The systems, devices, modules or units described in the above embodiments may be implemented by computer chips or entities, or by products with certain functions. A typical implementation device is a computer, which may be in the form of a personal computer, a laptop computer, a cellular phone, a camera phone, a smart phone, a personal digital assistant, a media player, a navigation device, an email transceiver, a game console, a tablet computer, a wearable device or a combination of any of these devices.

[0096] In a typical configuration, a computer includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0097] The memory may include non-permanent storage in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. The memory is an example of a computer-readable medium.

[0098] Computer readable media include permanent and non-permanent, removable and non-removable media that can store information by any method or technology. Information can be computer-readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassettes, disk storage, quantum memory, graphene-based storage media or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined in this article, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.

[0099] It should be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of further restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0100] The above specific embodiments of the present application are described. Other embodiments are within the scope of the present application. In some cases, the actions or steps recorded in the present application can be performed in an order different from that in the embodiments and still achieve the desired results. In addition, the process depicted in the accompanying drawings does not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0101] The terms used in one or more embodiments of the present application are only for the purpose of describing specific embodiments, and are not intended to limit one or more embodiments of the present application. The singular forms of "a", "said" and "the" are also intended to include plural forms, unless the context clearly indicates other meanings. The term "and / or" refers to and includes any or all possible combinations of one or more associated listed items.

[0102] The description of the terms "one embodiment", "some embodiments", "example", "specific example" or "an implementation method" etc. used in one or more embodiments of the present application means that the specific features or characteristics described in conjunction with the embodiment are included in at least one embodiment of the present application. The schematic description of these terms is not necessarily for the same embodiment. Moreover, the specific features or characteristics described can be combined in a suitable manner in one or more embodiments of the present application. In addition, different embodiments and specific features or characteristics in different embodiments can be combined without contradicting each other.

[0103] It should be understood that, although the terms first, second, third, etc. may be used to describe various information in one or more embodiments of the present application, these information should not be limited to these terms. These terms are only used to distinguish the same type of information from each other. For example, without departing from the scope of one or more embodiments of the present application, the first information may also be referred to as the second information, and similarly, the second information may also be referred to as the first information. Depending on the context, the word "if" as used herein may be interpreted as "at the time of" or "when" or "in response to determination".

[0104] The above description is only a preferred embodiment of one or more embodiments of the present application and is not intended to limit one or more embodiments of the present application. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of one or more embodiments of the present application shall be included in the scope of protection of one or more embodiments of the present application.

[0105] The user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of relevant countries and regions, and provide corresponding operation entrances for users to choose to authorize or refuse.

Claims

1. A load-aware scheduling method for an inference system, applied to a global scheduler in an inference system; the inference system further comprises an inference engine; the inference engine comprises at least one computing instance deployed on each computing node in a computing cluster; the computing resources of the computing instance comprise a GPU mounted on the computing node; ； The global scheduler maintains dynamically updated GPU load information of each computing instance; The method comprises: Get the target inference request to be executed; Based on the GPU load information of each computing instance maintained, determine a target computing instance whose GPU load meets a preset condition; The target reasoning request is sent to the target computing instance so that the target computing instance executes the target reasoning request.

2. According to the method of claim 1, the preset condition is that the GPU load in the computing instance is minimal.

3. The method according to claim 1, wherein the inference engine further comprises a local scheduler deployed on each computing node in the computing cluster; sending the target inference request to the target computing instance, include: The target inference request is sent to a target local scheduler, so that the target local scheduler sends the target inference request to the target computing instance; wherein the target local scheduler is a local scheduler deployed on a computing node where the target computing instance is located.

4. The method according to claim 3, further comprising: include: Obtain GPU load information of each computing instance on each computing node periodically sent by a local scheduler on each computing node according to a preset time period; The GPU load information of each computing instance maintained is updated based on the acquired GPU load information.

5. The method according to claim 3, wherein the GPU load information comprises instance resource utilization; in, The instance resource utilization is an indicator for indicating GPU load calculated by the local scheduler based on the GPU memory utilization and GPU memory bandwidth utilization of the computing instance; The step of determining a target computing instance whose GPU load satisfies a preset condition based on the maintained GPU load information of each computing instance includes: Based on the maintained GPU load information of each computing instance, a target computing instance with the minimum instance resource utilization is determined.

6. The method according to claim 5, wherein the GPU load information further includes GPU memory utilization; The target computing instance with the minimum instance resource utilization is determined based on the GPU load information of each computing instance maintained, include: Determine, based on the maintained GPU load information of each computing instance, a candidate computing instance whose GPU memory utilization is less than a preset first threshold; The candidate computing instances are sorted according to the instance resource utilization rates to determine a target computing instance with the minimum instance resource utilization rate from the candidate computing instances.

7. The method according to claim 5, wherein the instance resource utilization is calculated based on the GPU memory utilization and GPU memory bandwidth utilization of the computing instance by the following formula: ； ； ； in, Indicates the instance resource utilization of the computing instance. Indicates the total GPU memory size of the compute instance. represents the composite load of the compute instance, represents the number of tokens generated by the computation instance in each iteration. Indicates the GPU memory utilization of the compute instance. represents the composite coefficient, represents the preset first control coefficient, represents the preset second control coefficient, , represents a preset second threshold, Indicates the GPU memory bandwidth utilization of the compute instance. 8 . The method according to claim 7 , wherein the instance resource utilization of the computing instance is set to infinity when the computing instance is in the process of being terminated.

9. An inference system, comprising a global scheduler and an inference engine; the inference engine comprises at least one computing instance deployed on each computing node in a computing cluster; the computing resources of the computing instance comprise a GPU mounted on the computing node; the global scheduler maintains dynamically updated GPU load information of each computing instance; The global scheduler is used to: Get the target inference request to be executed; Based on the GPU load information of each computing instance maintained, determine a target computing instance whose GPU load meets a preset condition; The target reasoning request is sent to the target computing instance so that the target computing instance executes the target reasoning request.

10. An electronic device, include: processor; a memory for storing processor-executable instructions; The processor implements the method according to any one of claims 1 to 8 by running the executable instructions.

11. A computer-readable storage medium having computer instructions stored thereon, wherein the instructions are executed by a processor to implement the method according to any one of claims 1 to 8.

Images