GPU resource scheduling and optimizing method oriented to large model reasoning

Abstract

The invention relates to a GPU resource scheduling and optimizing method oriented to large model reasoning. The method comprises the following steps: segmenting a model by adopting a model parallelization strategy to obtain each task to be processed; performing GPU scheduling on each to-be-processed task at a task level according to resource data returned by an instrumented code; and executing the computing kernels and the communication kernels of the to-be-processed tasks of different batches in parallel on the same GPU. By adopting the method, the large model reasoning performance and the resource utilization efficiency can be improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of artificial intelligence, in particular to a GPU resource scheduling and optimization method for large model inference. BACKGROUND

[0002] With the continuous expansion of the scale of large language models (LLM), the inference process faces many challenges. On the one hand, the single query inference processing time is long, which cannot meet the real-time requirements; on the other hand, to meet the inference requirements, excessive computing resources need to be configured, which not only increases the operating cost, but also reduces the resource utilization efficiency. For example, in the inference of large-scale Transformer generation models, the model computation is huge, and the existing inference system cannot achieve high GPU throughput while ensuring low latency.

[0003] In the machine learning inference scene, as the core computing device, the hardware scheduling mechanism of GPU (such as NVIDIA's CUDA runtime scheduling) has significant limitations. First, the hardware scheduler is inefficient, using a strict FIFO strategy, which easily causes HoL (Head of Line) blocking, resulting in insufficient utilization of GPU resources. For example, the number of hardware queues of NVIDIA GPU is limited, and when multiple tasks are concurrent, relying on the sequential submission of kernels will result in less than 18% resource utilization. Second, it lacks flexibility and cannot adjust the scheduling strategy according to application-level goals (such as latency, fairness), making it difficult to cope with diverse workloads. Third, it is a black box, and existing frameworks cannot obtain fine-grained information about GPU kernel scheduling (such as SM resource occupancy, task completion time), making scheduling decisions lack basis.

[0004] To achieve computation parallelization and distributed memory allocation, existing distributed inference systems use two methods: intra-operator parallelism and inter-operator parallelism. These two methods have been extensively studied in throughput-oriented training tasks, and they divide and distribute the target model to different devices to handle larger models, but have different effects on latency and throughput. Intra-operator parallelism partitions each operation across multiple devices, which can reduce the computational latency of a single operation because multiple devices can perform partial computation simultaneously. However, to eliminate data dependencies between devices, frequent collective communication is required. This not only limits the improvement of throughput, but also leads to resource waste, as the computing units will be idle during communication. Inter-operator parallelism divides the target model into disjoint stages, each containing a set of consecutive operations and residing on a separate device, and requests are processed in a pipelined manner. Its advantage is that the communication volume is limited, and the throughput can increase linearly with the number of devices, and the cost of processing a single request is low. However, the disadvantage is that it cannot reduce latency, as each request is processed by a single device in sequence, and additional communication and load imbalance can even slightly increase latency.

[0005] Furthermore, in heterogeneous computing scenarios with multi-tenants and mixed workloads, different tasks have vastly different requirements for computing resources. Traditional inference systems and load balancing strategies struggle to adapt, leading to irrational resource allocation and overall performance degradation. This inability to flexibly schedule applications based on their actual needs leads to a mismatch between scheduling results and application performance targets, impacting overall system performance. Therefore, a new technical solution is urgently needed to address these issues and improve the performance and resource utilization efficiency of large-model inference. Summary of the Invention

[0006] Based on this, it is necessary to provide a GPU resource scheduling and optimization method for large model reasoning that can improve the performance and resource utilization efficiency of large model reasoning to address the above technical problems.

[0007] In a first aspect, the present application provides a GPU resource scheduling and optimization method for large model reasoning, the method comprising:

[0008] Use the model parallelization strategy to split the model and generate tasks to be processed based on the split model;

[0009] Performing GPU scheduling on each of the pending tasks at the task level according to the resource data returned by the instrumented code;

[0010] The computing kernels and communication kernels of the tasks to be processed in different batches are executed in parallel on the same GPU.

[0011] In one embodiment, before executing the computing kernels and communication kernels of the tasks to be processed in different batches in parallel on the same GPU, the method further includes:

[0012] Obtaining model parameters, and generating inputs based on the model parameters;

[0013] Testing each kernel based on each input to obtain performance data corresponding to the input, the performance data including kernel name, input scale parameter, and execution time;

[0014] storing the performance data;

[0015] The step of executing the computing kernels and communication kernels of the tasks to be processed in different batches in parallel on the same GPU includes:

[0016] Based on the stored performance data, the computing kernels and communication kernels of the tasks to be processed in different batches are overlapped according to the execution time, so that the computing kernels and communication kernels of the tasks to be processed in different batches are executed in parallel on the same GPU.

[0017] In one embodiment, the method further comprises:

[0018] In the case where the scale of the kernel function is greater than the preset function scale, the kernel function with the scale greater than the preset function scale is decomposed by different decomposition strategies, and performance data corresponding to the kernel function under each decomposition strategy is obtained.

[0019] The parallel execution of the computing kernels and the communication kernels of different batches of the to-be-processed tasks on the same GPU also includes:

[0020] In the case where the scale of the computing kernel and / or the communication kernel is greater than the preset function scale, the computing kernels and the communication kernels of different batches of the to-be-processed tasks are overlapped according to the execution time based on the performance data corresponding to different decomposition strategies, so that the computing kernels and the communication kernels of different batches of the to-be-processed tasks are executed in parallel on the same GPU.

[0021] In one embodiment, in the case where the scale of the computing kernel and / or the communication kernel is greater than the preset function scale, the computing kernels and the communication kernels of different batches of the to-be-processed tasks are overlapped according to the execution time based on the performance data corresponding to different decomposition strategies, including:

[0022] In the case where the communication kernel with the scale greater than the preset function scale cannot be matched with the computing kernel in the main computing flow, the communication kernel is decomposed according to different decomposition strategies, and performance data corresponding to each decomposition strategy is obtained.

[0023] Based on the performance data corresponding to each decomposition strategy and the execution time of the computing kernel in the main computing flow, the computing kernels and the communication kernels of different batches of the to-be-processed tasks are overlapped according to the execution time.

[0024] In one embodiment, the decomposing the communication kernel according to different decomposition strategies includes:

[0025] The kernel operation function is bound to new parameters using std::bind to generate a new callback function, and a new parameter tuple is constructed to store the operation parameters of the decomposed kernel function, wherein the decomposition strategy is vertical decomposition. In one embodiment, the GPU scheduling of each to-be-processed task at the task level according to the resource data returned by the instrumented code includes:

[0026] The to-be-processed tasks are allocated to a waiting queue.

[0027] In a case where the execution queue is not full, the pending tasks in the waiting queue are dequeued and assigned to the execution queue, each pending task in the execution queue is executed in parallel, the execution queue assigns two cuda streams to each pending task at a task level according to a load request of each GPU, a priority of the pending task, and a memory usage, the cuda streams include a main computing stream and a communication auxiliary computing stream, the main computing stream is used to execute a computing kernel, and the communication auxiliary computing stream is used to execute a communication kernel.

[0028] In one of the embodiments, the method further comprises:

[0029] Simulating inference on each of the pending tasks in the execution queue obtains a plurality of library functions, and the library functions are assembled in sequence to obtain a function list;

[0030] The parallel execution of the computing kernels and the communication kernels of different batches of the pending tasks on the same GPU comprises:

[0031] Scanning the function list, in a case where a current function is a computing kernel, the function is assigned to the main computing stream;

[0032] Scanning the function list of other tasks, and assigning a communication kernel function with a size of the execution time matched with a size of the execution time of the current function to the communication auxiliary computing stream, the main computing stream and the communication auxiliary computing stream are calculated in parallel.

[0033] In one of the embodiments, the main computing stream includes two key marker events; in a case where a first key marker event is triggered, the main computing stream and the communication auxiliary computing stream start respective kernel tasks; in a case where a second key marker event is triggered, the main computing stream and the communication auxiliary computing stream execute respective kernel tasks.

[0034] In one of the embodiments, before the GPU scheduling of each of the pending tasks at a task level according to the resource data returned by the instrumented code, the method further comprises:

[0035] Instrumenting a kernel function in the code to obtain resource data of a system runtime according to the instrumented code.

[0036] In a second aspect, the application further provides a GPU resource scheduling and optimization device for large model inference, the device comprises:

[0037] A receiving module is configured to split a model by using a model parallelization strategy, and generate each pending task based on the split model;

[0038] A task-level scheduling module, configured to schedule the GPU for each of the pending tasks at the task level according to the resource data returned by the instrumented code;

[0039] The kernel-level scheduling module is used to execute the computing kernels and communication kernels of different batches of the tasks to be processed in parallel on the same GPU.

[0040] The above-mentioned GPU resource scheduling and optimization method for large-model inference adopts a model parallelization strategy to segment the model and generates each task to be processed based on the segmented model; according to the resource data returned by the instrumented code, the GPU is scheduled at the task level for each task to be processed; the computing kernels and communication kernels of the tasks to be processed in different batches are executed in parallel on the same GPU. In this way, when multiple batches of tasks are executed at the same time, the computing kernels and communication kernels of different tasks will overlap, which leads to inconsistencies in the inference stages of each batch. Multi-layer calculations are executed in sequence in a pipeline manner between different GPU groups, and the scenario where tasks in different inference stages are performed simultaneously achieves an effect similar to inter-layer parallelism. The two cooperate with each other through "time slicing multiplexing" to convert the communication time wasted in traditional methods into computing time, maximize resource utilization, and thus improve the performance and resource utilization efficiency of large-model inference. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] In order to more clearly illustrate the technical solutions in the embodiments of the present application or related technologies, the following briefly introduces the drawings required for use in the embodiments of the present application or related technical descriptions. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying any creative work.

[0042] Figure 1 1 is a flow chart of a GPU resource scheduling method according to an embodiment;

[0043] Figure 2 A comparison chart of intra-operator parallelism and inter-operator parallelism in an instance;

[0044] Figure 3 This is the overall process of the performance analysis phase in an instance;

[0045] Figure 4 A comparison chart of two kernel decomposition methods: horizontal decomposition and vertical decomposition of the matrix;

[0046] Figure 5 A schematic diagram of the overall process of task-level scheduling in an instance;

[0047] Figure 6 A schematic diagram of kernel-level scheduling in an instance;

[0048] Figure 7 FIG. 1 is a structural block diagram of a GPU resource scheduling device in an embodiment;

[0049] Figure 8 FIG. 2 is an internal structural diagram of a computer device in an embodiment. DETAILED DESCRIPTION

[0050] In order to make the purposes, technical solutions and advantages of the present application clearer, further detailed description will be made to the present application in combination with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application, and are not used to limit the present application.

[0051] It should be noted that the terms "first", "second", etc. used in the present application can be used to describe various elements, but these elements are not limited by these terms. These terms are only used to distinguish the first element from the second element. The terms "include" and "have" and any variations thereof used in the present application are intended to cover non-exclusive inclusion. The term "multiple" used in the present application refers to two and more than two. The term "and / or" used in the present application refers to one of the options, or any combination of multiple options.

[0052] In an embodiment, as shown in Figure 1 A GPU resource scheduling and optimization method for large model inference is provided. The method can be applied to a server, and can also be applied to a terminal and a system including a terminal and a server, and can be realized through the interaction of the terminal and the server. The method includes the following steps:

[0053] S102: The model is segmented using a model parallelization strategy, and each to-be-processed task is generated based on the segmented model.

[0054] The to-be-processed task is determined for large language model (LLM) inference, and the to-be-processed task includes tasks of different batches.

[0055] In the present application, the model is segmented using a model parallelization strategy. For example, in the implementation of the present embodiment, the focus is on the design of a distributed inference system. As shown in Figure 2As shown in the figure, targeting the characteristics of large-scale Transformer generation models, a model parallelization strategy combining intra-layer and inter-layer parallelism is adopted. Intra-layer parallelism splits the input tensors of computational tasks in the Transformer layer, such as matrix multiplication or attention operations, by dimension, allowing different components to be independently computed on different GPUs. This fully leverages the computing power of multiple GPUs and avoids the problem of a single GPU being unable to complete the computation due to memory limitations. Inter-layer parallelism does not follow the traditional approach of splitting each model stage onto different GPUs for separate execution. Instead, when multiple batches of tasks are executed simultaneously, the compute and communication kernels of different tasks overlap, leading to inconsistencies in the inference phase of each batch. Multi-layer computations are pipelined sequentially across different GPU groups, achieving a similar effect to inter-layer parallelism when tasks at different inference phases are executed simultaneously. The two methods work together through "time slicing and reuse," converting communication time wasted in traditional methods into computation time, maximizing resource utilization.

[0056] Each pending task is generated based on the segmented model, that is, a pending task corresponding to a model, that is, a request.

[0057] S104: Perform GPU scheduling on each to-be-processed task at the task level according to the resource data returned by the instrumented code.

[0058] S106: Execute the computing kernels and communication kernels of different batches of tasks to be processed in parallel on the same GPU.

[0059] The scheduling of this application includes two levels: task-level scheduling and kernel-level scheduling. Task-level scheduling is to batch each request and implement inference according to the total resource situation when the request arrives. This level of scheduling can isolate resources through task scheduling in multi-task scenarios, such as running multiple deep learning models at the same time, to avoid mutual interference. Kernel-level scheduling is to schedule the execution order of kernels within each deployed task. This level of scheduling can orderly overlap the operation of each kernel according to the characteristics of the GPU, making full use of the GPU resources to avoid resource waste and time extension caused by resource conflicts.

[0060] That is, for larger operators in the model, such as the matrix multiplication operator, this application will decompose the operator and distribute the decomposed parts to different GPUs, allowing multiple GPUs to each perform a part of the calculation, so that the workload of the operator can be divided into multiple devices and performed simultaneously. For different batches, this application will process multiple batches in parallel. When one batch starts communicating, the computing kernels of other batches are scheduled to the GPU for execution. This strategy is used to achieve overlap of calculation and communication between different batches.

[0061] The kernel scheduling calculation of the application and the communication kernel are reasonably scheduled, so that the two types of kernels can be executed on a GPU at the same time, thereby improving the utilization of the GPU.

[0062] The GPU resource scheduling method divides the model by using a model parallelization strategy, generates each to-be-processed task based on the divided model, performs GPU scheduling on each to-be-processed task at a task level according to resource data returned by the instrumented code, and executes the computing kernels and the communication kernels of different batches of to-be-processed tasks in parallel on the same GPU. In this way, when multiple batches of tasks are executed at the same time, the computing kernels and the communication kernels of different tasks are overlapped, thereby causing inconsistency of inference stages of each batch, and multi-layer calculation is sequentially executed in a pipeline manner between different GPU groups, so that a similar inter-layer parallel effect is achieved in a scenario in which tasks of different inference stages are simultaneously performed. Through time slicing multiplexing, the communication time wasted in the traditional method is converted into calculation time, the resource utilization rate is maximized, and the performance and resource utilization efficiency of large model inference are improved.

[0063] In one of the optional embodiments, before the computing kernels and the communication kernels of different batches of to-be-processed tasks are executed in parallel on the same GPU, the method further includes obtaining model parameters and generating each input based on the model parameters, testing each kernel based on each input to obtain performance data corresponding to the input, the performance data including a kernel name, an input size parameter, and an execution time, storing the performance data, and executing the computing kernels and the communication kernels of different batches of to-be-processed tasks in parallel on the same GPU, including overlapping the computing kernels and the communication kernels of different batches of to-be-processed tasks according to the execution time based on the stored performance data, so that the computing kernels and the communication kernels of different batches of to-be-processed tasks are executed in parallel on the same GPU.

[0064] Before kernel decomposition, a performance analysis stage is further included, that is, before the model is formally run, all size data within the input range in this scenario are inferred to perform performance testing, performance data of each kernel running at each size is collected and saved with an independent ID, and through analysis of the kernel duration, key information can be provided for the scheduling algorithm. In actual operation, different sizes of inputs will affect the execution time of the kernel, and recording these time information helps to more accurately find a matched kernel subset, improves the accuracy and efficiency of scheduling, and further improves the overall performance of the system.

[0065] Specifically, as shown in FIG. 1, Figure 3 the performance analysis stage includes:

[0066] Step S111: Obtain model parameters and simulate to generate each input.

[0067] The obtained parameters are specifically the maximum batchsize and the maximum sequence of the model. According to the two parameters, different inputs of all scales from 0 to the maximum batchsize and from 0 to the maximum sequence can be simulated and generated. Since the execution time of each kernel is to be tested, the initial values of the inputs are random, and the output result is not concerned.

[0068] Step S112: reasoning each input;

[0069] Step S113: testing each kernel multiple times and taking the average execution time, and some special kernels are decomposed and then tested;

[0070] All kernel functions involved in the reasoning process are executed multiple times and the average time is calculated. Some large-scale kernel functions, such as the GEMM kernel, are not only tested for the average execution time of the original kernel, but also decomposed by different decomposition strategies and then tested for the average execution time of each strategy.

[0071] Step S114: saving the test data of each kernel;

[0072] After each kernel is tested, the test data is stored in a file in the order of kernel name, batchsize, sequence size, and execution time for subsequent use.

[0073] In the subsequent internal kernel scheduling mechanism with smaller granularity, the performance data of each kernel in each scenario generated in the previous performance analysis stage is used to roughly estimate the actual execution time of the kernel, and then different types of kernels, i.e. calculation and communication kernels, are reasonably scheduled so that the two types of kernels can be executed on a GPU at the same time, thereby improving the utilization of the GPU.

[0074] In the above embodiment, the execution performance of the kernel functions of the model in all input scales allowed in the scenario, i.e. the average execution time of the kernel functions, is tested according to the actual situation of the scenario, and these performance data are saved in a file for reference by the subsequent scheduling stage.

[0075] In one of the optional embodiments, the method further comprises: in the case where the size of the kernel function is greater than the preset function size, decomposing the kernel function with a size greater than the preset function size by different decomposition strategies, and obtaining the performance data of the kernel function corresponding to each decomposition strategy; and performing the calculation kernel and the communication kernel of the different batches of to-be-processed tasks in parallel on the same GPU, and further comprising: in the case where the size of the calculation kernel and / or the communication kernel is greater than the preset function size, overlapping the calculation kernel and the communication kernel of the different batches of to-be-processed tasks according to the execution time based on the performance data corresponding to the different decomposition strategies, so that the calculation kernel and the communication kernel of the different batches of to-be-processed tasks are executed in parallel on the same GPU.

[0076] In the kernel decomposition mechanism existing in the performance analysis stage and the kernel scheduling stage, the mechanism first tests the execution average time of the various decomposition strategies of the large kernel existing in the model in the performance analysis stage, and then in the actual kernel scheduling stage, if the large kernel greatly exceeds the estimated range when overlapping the communication and the calculation kernel, the estimated time of each decomposition scheme generated in the performance analysis stage is filled, and if there is a decomposition scheme that meets the time requirement, the decomposition scheme is applied to decompose and fill the kernel into the kernel execution queue.

[0077] For some huge kernels in large models, a uniform division strategy is adopted, and the running time of the kernel under each division is recorded in the performance analysis stage. During runtime, the scheduler dynamically determines the final decomposition of the kernel into parts based on the results of the previous analysis, combined with the actual situation such as the overlapable time of current calculation and communication. Different decomposition strategies have a significant impact on performance, and appropriate decomposition strategies can maximize the overall performance and efficiency of the system. A larger decomposition factor can bring better delay and throughput results, because a larger decomposition factor can achieve more accurate decomposition granularity, allowing the scheduler to find kernel subsets with more matched durations.

[0078] In some optional embodiments, in the case where the size of the calculation kernel and / or the communication kernel is greater than the preset function size, the calculation kernel and the communication kernel of the different batches of to-be-processed tasks are overlapped according to the execution time based on the performance data corresponding to the different decomposition strategies, including: in the case where there is a communication kernel with a size greater than the preset function size and the calculation kernel in the main calculation stream cannot be matched, decomposing the communication kernel according to different decomposition strategies and obtaining the performance data corresponding to each decomposition strategy; and overlapping the calculation kernel and the communication kernel of the different batches of to-be-processed tasks according to the execution time based on the performance data corresponding to each decomposition strategy and the execution time of the calculation kernel in the main calculation stream.

[0079] In the above embodiments, the kernel decomposition mechanism exists in the performance analysis stage and the kernel scheduling stage. The mechanism first tests the average execution time of the large kernel in the model under various decomposition strategies in the performance analysis stage, and then in the actual kernel scheduling stage, if the large kernel greatly exceeds the estimated range when overlapping the communication and computation kernels, the estimated time of each decomposition scheme generated in the performance analysis stage is used to fill in, and if there is a decomposition scheme that meets the time requirement, the kernel is decomposed and filled into the kernel execution queue using the decomposition scheme.

[0080] In some optional embodiments, the communication kernel is decomposed according to different decomposition strategies, including: using std::bind to bind the kernel operation function with new parameters, generating a new callback function, and constructing a new parameter tuple to store the decomposed kernel function operation parameters, wherein the decomposition strategy is vertical decomposition. As shown in Figure 4 The kernel decomposition part implemented by the present application performs decomposition operation on a specific large kernel (such as GEMM) in the performance analysis stage. The reason for performing kernel decomposition operation is that if there is a large communication kernel that cannot be matched with the computation kernel in the main computation stream in the kernel scheduling stage, the communication kernel can be decomposed according to different decomposition strategies, and if the kernel can be placed in the communication auxiliary stream under a decomposition strategy, the resources of the GPU can be fully utilized to ensure that the resources are not wasted. Such decomposition strategy can make the kernel-level scheduling more flexible. The specific method is to use std::bind to bind the GEMM function with new parameters, generate a new callback function, and construct a new parameter tuple args_gemm to store the decomposed GEMM operation parameters. In this process, the decomposition operation only changes the third parameter (serial number 2) of GEMM, that is, the number of columns of matrix B (C = A x B), that is, the vertical decomposition of the matrix. The vertical decomposition method is superior to the horizontal decomposition method in performance. The horizontal decomposition method will significantly reduce the computation intensity, and the cumulative execution time of all fine-grained kernels will be significantly longer than the original kernel. In contrast, the vertical decomposition method is much more superior in performance. The first reason is that matrix A is already a thin and long matrix, and horizontal decomposition will make it thinner and longer, resulting in poor data locality. The second reason is that matrix B is larger than matrix A. When decomposing matrix B, the required memory input / output amount is much less.

[0081] In some optional embodiments, according to the resource data returned by the instrumented code, the GPU scheduling is performed at the task level for each to-be-processed task, including: assigning the to-be-processed task to a waiting queue; in the case that the execution queue is not full, dequeuing the to-be-processed task in the waiting queue and assigning it to the execution queue, and each to-be-processed task in the execution queue is executed in parallel, and the execution queue assigns two cuda streams to the to-be-processed task at the task level according to the load request of each GPU, the priority of the to-be-processed task, and the memory usage, the cuda stream includes a main computing stream and a communication auxiliary computing stream, and the main computing stream is used to execute a computing kernel, and the communication auxiliary computing stream is used to execute a communication kernel.

[0082] As shown in FIG. 1, Figure 5 In one embodiment, a global scheduler is designed to achieve efficient scheduling of the entire distributed system. It will comprehensively consider the load of each GPU, the priority of the request, the memory usage and other factors, and dynamically adjust the allocation of tasks. For example, when the load of a certain GPU is high, the scheduler will preferentially allocate new tasks to GPUs with lower load. In addition, the scheduler also monitors the overall performance indicators of the system in real time, and verifies whether the job is suitable for the current GPU resources during the scheduling process, and delays the scheduling if the resources are insufficient. Then, according to the real-time resource utilization, the task queue is sorted, and the job with high resource matching degree is preferentially scheduled. The task-level scheduling of the method first sets two queues, a waiting queue and an execution queue, all incoming tasks are first placed in the waiting queue, if the execution queue is not full, the tasks in the waiting queue are dequeued and enqueued into the execution queue to start executing inference. The execution queue has a task upper limit, all tasks in the execution queue will be executed in parallel, and whenever a new task arrives in the execution queue, the application will allocate two cuda streams to the task from the total stream pool at the task level according to the real-time remaining resources, calculate how many threads the task needs, and update the remaining resource situation.

[0083] In some optional embodiments, before the GPU scheduling is performed at the task level for each to-be-processed task according to the resource data returned by the instrumented code, the method further includes: instrumenting the code of the kernel function in the model, and obtaining the resource data of the system runtime according to the instrumented code.

[0084] During the running of some tasks, the application will run the resource monitoring code instrumented in advance, which will return some resource information such as the usage of SM registers, shared memory, and thread blocks. When the request arrives, the application will schedule the task according to the current resource situation, and calculate the required resources of the task according to the input size and model parameter information, and update the resource remaining situation for reference for subsequent scheduling.

[0085] In some optional embodiments, the method also includes: performing simulated reasoning on each pending task in the execution queue to obtain a number of library functions, and assembling the library functions in sequence to obtain a function list; executing the computing kernels and communication kernels of different batches of pending tasks in parallel on the same GPU, including: scanning the function list, and when the current function is a computing kernel, allocating the function to the main computing flow; scanning the function lists of other tasks, and allocating the communication kernel function whose execution time matches the execution time of the current function to the communication auxiliary computing flow, and the main computing flow and the communication auxiliary computing flow are calculated in parallel.

[0086] When a task enters the execution queue, it undergoes simulated inference. This means that the task is not actually computed by the GPU. This simulated inference process captures the relevant CUDA library functions generated during the entire inference process, packages them, and assembles them into a function list in order. Once the task is fully executed, the application releases the two CUDA streams and returns them to the overall stream pool.

[0087] like Figure 6 As shown, in one embodiment, at the beginning of task execution, the system will create two key CUDA streams for each task, namely the main computing stream (Primary Stream) and the communication auxiliary stream (Sub Stream). The main computing stream mainly undertakes heavy computing tasks. For example, in the large model reasoning process, a large number of matrix multiplication operations (GEMM) and activation function calculations in the Transformer layer are executed in this stream. These computing operations will consume a lot of GPU computing resources, such as the SM unit in the GPU. The communication auxiliary stream focuses on processing cross-GPU data communication tasks, such as the NCCL All-Reduce operation in a multi-GPU environment, which mainly consumes memory bandwidth and communication interface resources. This application does not load the ready-made original model, but adds kernel function classification code to each layer of the model, that is, divides it into two categories: computing and communication. This classification provides a basis for the subsequent reasonable allocation of tasks on different streams. Therefore, during the kernel scheduling phase, the kernel scheduler uses the kernel function list compiled in the previous step, based on the performance analysis phase, to detect the estimated execution time of each kernel. It first scans the kernel function list of a task. When a kernel type conversion occurs, such as switching from a compute kernel to a communication kernel, this series of consecutive compute kernels is assigned to the main compute stream. Simultaneously, the kernel function lists of other tasks are scanned, and communication kernel functions with matching execution times are inserted into the communication auxiliary stream, thereby overlapping the execution of the compute and communication kernels. To prevent performance degradation caused by discrepancies between estimated and actual execution times, kernel overlap between the two streams leaves room for timing errors. For example, a 100-second compute kernel will only overlap an 80-second communication kernel.

[0088] In the above embodiments, the task level and kernel level combined scheduling method using cuda flow is used. First, when a new task arrives, the application will allocate two cuda flows to the task at the task level according to the real-time remaining resources, and update the resource remaining situation. All kernel functions and scales required to execute during the entire inference process of the task are combined into a queue, and the two cuda flows are released back to the flow pool after the task is executed. Then, between the two cuda flows, the kernel function queue composed in the last step is used to detect the estimated time of each kernel execution, and the calculation and communication kernels of different batches of tasks are overlapped according to the estimated time size in a reasonable order. In order to prevent the performance decline caused by the difference between the estimated time and the actual execution time, a certain space is left for the time error between the two flows, for example, 100s of calculation kernels are overlapped with 80s of communication kernels.

[0089] In some optional embodiments, the main computing flow includes two key marker events; in the case of triggering the first key marker event, the main computing flow and the communication auxiliary computing flow start the kernel tasks of each other; in the case of triggering the second key marker event, the main computing flow and the communication auxiliary computing flow execute the kernel tasks of each other.

[0090] In order to ensure the orderly progress of calculation and communication, the application inserts two key markers, EventA and EventB, in the main computing flow using CUDA events. When EventA is triggered, the CPU will inform the main computing flow and the communication auxiliary flow to start the kernel tasks of each other. However, at this time, these kernel tasks will not be executed immediately, but will be in a waiting state. The main computing flow continues to execute, and when it executes to EventB, it establishes a dependency relationship with the communication auxiliary flow based on EventB through cuda capability, thereby triggering the kernel tasks in the two flows to start actual execution. The execution order of the communication auxiliary flow strictly depends on the triggering of EventB, and only when the main computing flow executes to EventB, the communication auxiliary flow will execute the communication kernel tasks therein according to the predetermined order.

[0091] In summary, the application effectively addresses many challenges in large language model inference through a series of closely coordinated and targeted implementation methods. In the design of distributed inference systems, the use of model parallelization, communication optimization, pipeline execution and global scheduling techniques solves the problems of large model computation resource demand, high communication overhead and complex task scheduling, and improves the overall inference efficiency and system adaptability. Kernel decomposition optimizes key kernels, and compared with different decomposition methods, vertical decomposition with better performance is selected to improve GPU resource utilization. These implementation methods work together to achieve a balance between low latency and high GPU throughput without relying on specific GPU manufacturer hardware schedulers, reducing the computing resource cost of service providers, providing strong technical support for the widespread application of large language models in generative AI, and promoting the development of related industries.

[0092] It should be understood that, although each step in the flowchart involved in the above embodiments is displayed in sequence according to the arrow, these steps are not necessarily executed in the order indicated by the arrow. Unless otherwise specified herein, the execution of these steps is not strictly limited in sequence, and these steps can be executed in other orders. Moreover, at least part of the steps in the flowchart involved in the above embodiments can include multiple steps or stages, which are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily sequential, but can be alternately executed with other steps or steps or stages in other steps. It can be understood that the steps in different embodiments can be freely combined as needed, and various non-contradictory schemes formed by the combination are within the scope of protection of the present application.

[0093] Based on the same inventive concept, the embodiments of the present application also provide a GPU resource scheduling and optimization device for large model inference for implementing the above-mentioned GPU resource scheduling and optimization method for large model inference. The implementation scheme for solving problems provided by the device is similar to the implementation scheme described in the above method, so the specific limitations in one or more GPU resource scheduling and optimization device embodiments for large model inference provided below can refer to the limitations of the GPU resource scheduling and optimization method for large model inference in the above text, which will not be repeated here.

[0094] In one exemplary embodiment, as shown in Figure 7 A GPU resource scheduling and optimization device for large model inference is provided, comprising: a segmentation module 701, a task-level scheduling module 702, and a kernel-level scheduling module 703, wherein:

[0095] A segmentation module 701 is used to segment the model using a model parallelization strategy and generate tasks to be processed based on the segmented model;

[0096] The task-level scheduling module 702 is used to perform GPU scheduling on each pending task at the task level according to the resource data returned by the instrumented code;

[0097] The kernel-level scheduling module 703 is used to execute the computing kernels and communication kernels of different batches of tasks to be processed in parallel on the same GPU.

[0098] In one of the optional embodiments, the above device further includes:

[0099] The performance analysis module is used to obtain model parameters and generate inputs based on the model parameters; test each kernel based on the inputs to obtain performance data corresponding to the inputs, including kernel name, input scale parameters, and execution time; and store the performance data;

[0100] The kernel-level scheduling module 703 is also used to overlap the computing kernels and communication kernels of different batches of tasks to be processed according to the execution time based on the stored performance data, so that the computing kernels and communication kernels of different batches of tasks to be processed can be executed in parallel on the same GPU.

[0101] In one of the optional embodiments, the above-mentioned kernel-level scheduling module 703 is also used to decompose the kernel function whose scale is larger than the preset function scale through different decomposition strategies when there is a kernel function whose scale is larger than the preset function scale, and obtain the performance data corresponding to the kernel function under each decomposition strategy; when the scale of the computing kernel and / or the communication kernel is larger than the preset function scale, based on the performance data corresponding to different decomposition strategies, the computing kernels and communication kernels of different batches of tasks to be processed are overlapped according to the size of the execution time, so that the computing kernels and communication kernels of different batches of tasks to be processed are executed in parallel on the same GPU.

[0102] In one of the optional embodiments, the kernel-level scheduling module 703 is also used to decompose the communication kernel according to different decomposition strategies when there is a communication kernel whose scale is larger than the preset function scale and cannot match the computing kernel in the main computing flow, and obtain the performance data corresponding to each decomposition strategy; based on the performance data corresponding to each decomposition strategy and the execution time of the computing kernel in the main computing flow, the computing kernels and communication kernels of different batches of tasks to be processed are overlapped according to the size of the execution time.

[0103] In one of the optional embodiments, the kernel-level scheduling module 703 is further configured to bind the kernel operation function with the new parameters using std::bind to generate a new callback function, and construct a new parameter tuple to store the decomposed kernel function operation parameters, wherein the decomposition strategy is vertical decomposition. In one of the optional embodiments, the task-level scheduling module 702 is further configured to assign the to-be-processed tasks to a waiting queue; in the case that the execution queue is not full, the to-be-processed tasks in the waiting queue are dequeued and assigned to the execution queue, and the to-be-processed tasks in the execution queue are executed in parallel, and the execution queue assigns two cuda streams to the to-be-processed tasks at the task level according to the load request of each GPU, the priority of the to-be-processed tasks, and the memory usage, wherein the cuda stream includes a main computing stream and a communication auxiliary computing stream, the main computing stream is used to execute the computing kernel, and the communication auxiliary computing stream is used to execute the communication kernel.

[0104] In one of the optional embodiments, the task-level scheduling module 702 is further configured to simulate and infer each to-be-processed task in the execution queue to obtain a plurality of library functions, and assemble the library functions in sequence to obtain a function list. Thus, the kernel-level scheduling module 703 is further configured to scan the function list, assign the function to the main computing stream in the case that the current function is a computing kernel, scan the function list of other tasks, and assign the communication kernel function with the execution time size matching the execution time size of the current function to the communication auxiliary computing stream, and the main computing stream and the communication auxiliary computing stream are calculated in parallel.

[0105] In one of the optional embodiments, the main computing stream includes two key marker events; in the case that the first key marker event is triggered, the main computing stream and the communication auxiliary computing stream start the kernel tasks of each other; and in the case that the second key marker event is triggered, the main computing stream and the communication auxiliary computing stream execute the kernel tasks of each other.

[0106] In one of the optional embodiments, the instrumentation module is configured to instrument the code of the kernel function in the model, and obtain the resource data of the system runtime according to the instrumented code.

[0107] The modules in the GPU resource scheduling apparatus can be all or part of software, hardware, and combinations thereof. The modules can be embedded in or independent of the processor in the computer device in hardware form, or stored in the memory in the computer device in software form, so as to be called and executed by the processor to perform the operations corresponding to the modules.

[0108] In one of the exemplary embodiments, a computer device is provided, which can be a server, and the internal structure diagram thereof can be as shown in Figure 8As shown in the figure. The computer device includes a processor, a memory, an input / output interface (I / O for short) and a communication interface. Among them, the processor, the memory and the input / output interface are connected through the system bus, and the communication interface is connected to the system bus through the input / output interface. Among them, the processor of the computer device is used to provide computing and control capability. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store various types of data involved in the model reasoning process. The input / output interface of the computer device is used to exchange information between the processor and external devices. The communication interface of the computer device is used to communicate with external terminals through network connection. The computer program is executed by the processor to realize a GPU resource scheduling and optimization method for large model reasoning.

[0109] Those skilled in the art can understand that, Figure 8 The structure shown in the figure is only a block diagram of part of the structure related to the scheme of the present application, and does not constitute a limitation on the computer device to which the scheme of the present application is applied. The specific computer device can include more or fewer components than those shown in the figure, or combine certain components, or have a different component arrangement.

[0110] In one embodiment, a computer device is also provided, including a memory and a processor, the memory storing a computer program, and the processor executing the computer program to realize the steps in each of the above method embodiments.

[0111] In one embodiment, a computer readable storage medium is provided, which stores a computer program, and the computer program is executed by the processor to realize the steps in each of the above method embodiments.

[0112] In one embodiment, a computer program product is provided, including a computer program, and the computer program is executed by the processor to realize the steps in each of the above method embodiments.

[0113] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data for analysis, stored data, displayed data, etc.) involved in the present application are all information and data authorized by the user or authorized by all parties, and the collection, use and processing of related data need to comply with relevant regulations.

[0114] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a non-volatile computer readable storage medium, and when executed, can include the processes of the above-mentioned embodiment methods. Any reference to memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile memory and volatile memory. The non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical storage, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetoresistive random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. The volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration but not limitation, the RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM), etc. The database involved in the embodiments provided in the present application can include at least one of a relational database and a non-relational database. The non-relational database can include a distributed database based on a block chain, etc., without being limited thereto. The processor involved in the embodiments provided in the present application can be a general-purpose processor, a central processing unit, a graphics processing unit, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, an artificial intelligence (AI) processor, etc., without being limited thereto.

[0115] The technical features of the above embodiments can be combined in any manner. In order to make the description simple, all possible combinations of the technical features in the above embodiments are not described, however, as long as the combinations of the technical features do not exist, they should be considered as the scope of the present application.

[0116] The above-described embodiments merely represent several implementation methods of the present application. While the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the present application. It should be noted that a person of ordinary skill in the art may make various modifications and improvements without departing from the spirit of the present application, and these modifications and improvements fall within the scope of protection of the present application. Therefore, the scope of protection of the present application shall be determined by the appended claims.

Claims

1. A GPU resource scheduling and optimization method for large model reasoning, characterized in that: The method comprises: Use the model parallelization strategy to split the model and generate tasks to be processed based on the split model; Performing GPU scheduling on each of the pending tasks at the task level according to the resource data returned by the instrumented code; The computing kernels and communication kernels of the tasks to be processed in different batches are executed in parallel on the same GPU.

2. The method according to claim 1, characterized in that Before executing the computing kernels and communication kernels of the tasks to be processed in different batches in parallel on the same GPU, the method further includes: Obtaining model parameters, and generating inputs based on the model parameters; Testing each kernel based on each input to obtain performance data corresponding to the input, the performance data including kernel name, input scale parameter, and execution time; storing the performance data; The step of executing the computing kernels and communication kernels of the tasks to be processed in different batches in parallel on the same GPU includes: Based on the stored performance data, the computing kernels and communication kernels of the tasks to be processed in different batches are overlapped according to the execution time, so that the computing kernels and communication kernels of the tasks to be processed in different batches are executed in parallel on the same GPU.

3. The method according to claim 2, characterized in that The method further comprises: In the case where the scale of a kernel function is larger than the preset function scale, decomposing the kernel function larger than the preset function scale by using different decomposition strategies, and obtaining performance data corresponding to the kernel function under each decomposition strategy; The step of executing the computing kernels and communication kernels of the tasks to be processed in different batches in parallel on the same GPU further includes: When the scale of the computing kernel and / or the communication kernel is larger than the preset function scale, based on the performance data corresponding to different decomposition strategies, the computing kernels and communication kernels of the tasks to be processed in different batches are overlapped according to the size of the execution time, so that the computing kernels and communication kernels of the tasks to be processed in different batches are executed in parallel on the same GPU.

4. The method according to claim 3, characterized in that When the scale of the computing kernel and / or the communication kernel is larger than the preset function scale, overlapping the computing kernels and the communication kernels of the to-be-processed tasks in different batches according to the execution time based on the performance data corresponding to different decomposition strategies includes: If there is a communication kernel whose scale is larger than the preset function scale and cannot be matched with the computing kernel in the main computing flow, the communication kernel is decomposed according to different decomposition strategies, and performance data corresponding to each decomposition strategy is obtained; Based on the performance data corresponding to each decomposition strategy and the execution time of the computing kernel in the main computing flow, the computing kernels and communication kernels of the tasks to be processed in different batches are overlapped according to the size of the execution time.

5. The method according to claim 4, characterized in that Decomposing the communication kernel according to different decomposition strategies includes: std::bind is used to bind the kernel operation function to the new parameters, a new callback function is generated, and a new parameter tuple is constructed to store the decomposed kernel function operation parameters, wherein the decomposition strategy is vertical decomposition.

6. The method according to claim 2 or 3, characterized in that The GPU scheduling of each of the tasks to be processed at the task level according to the resource data returned by the instrumented code includes: Allocating the pending tasks to a waiting queue; When the execution queue is not full, the pending tasks in the waiting queue are dequeued and assigned to the execution queue. The pending tasks in the execution queue are executed in parallel. The execution queue allocates two CUDA streams to the pending tasks at the task level based on the load request of each GPU, the priority of the pending tasks and the memory usage. The CUDA streams include a main computing stream and a communication auxiliary computing stream. The main computing stream is used to execute the computing kernel, and the communication auxiliary computing stream is used to execute the communication kernel.

7. The method according to claim 6, characterized in that The method further comprises: Performing simulation reasoning on each of the pending tasks in the execution queue to obtain a plurality of library functions, and assembling the library functions in sequence to obtain a function list; The step of executing the computing kernels and communication kernels of the tasks to be processed in different batches in parallel on the same GPU includes: Scan the function list, and if the current function is a computing kernel, assign the function to the main computing flow; The function lists of other tasks are scanned, and the communication kernel functions whose execution time matches the execution time of the current function are allocated to the communication auxiliary computing flow, and the main computing flow and the communication auxiliary computing flow are calculated in parallel.

8. The method according to claim 6, characterized in that The main computing flow includes two key marking events; when the first key marking event is triggered, the main computing flow and the communication auxiliary computing flow start their respective kernel tasks; When the second key mark event is triggered, the main computing flow and the communication auxiliary computing flow execute their respective kernel tasks.

9. The method according to claim 6, characterized in that Before performing GPU scheduling on each of the pending tasks at the task level based on the resource data returned by the instrumented code, the method further includes: Perform code instrumentation on the kernel functions in the model and obtain resource data during system runtime based on the instrumented code.

10. A GPU resource scheduling and optimization device for large model reasoning, characterized in that: The device comprises: The segmentation module is used to segment the model using a model parallelization strategy and generate tasks to be processed based on the segmented model; A task-level scheduling module, configured to schedule the GPU for each of the tasks to be processed at the task level according to the resource data returned by the instrumented code; The kernel-level scheduling module is used to execute the computing kernels and communication kernels of the tasks to be processed in different batches in parallel on the same GPU.

Images