GPU reasoning-oriented batch processing scheduling method, device and equipment and storage medium

Abstract

The invention discloses a GPU reasoning-oriented batch processing scheduling method and device, equipment and a storage medium. The method comprises the steps that a task and batch model and a reasoning model are established, an optimization problem is put forward on the basis, and due to the fact that the number of decision variables of the problem is unknown, the decision variables are mutually coupled, direct solving faces challenges, and therefore corresponding algorithms are put forward for offline and online scenes to optimize task scheduling. For an off-line condition, a dynamic planning algorithm is utilized to effectively plan how all tasks are combined into batches to be scheduled in a time delay constraint range of the tasks, so as to maximize the completion rate of the tasks; for an online scene, i.e., the condition that task release time is unknown, task scheduling distribution is realized by utilizing an online scheduling algorithm based on a competition ratio. According to the method, whether the GPU needs to be preempted or not is judged in real time so as to adapt to random fluctuation of task loads, and therefore the stable task completion rate is ensured.

Description

Technical Field

[0001] The present invention relates to the field of parallel computing, and in particular to a batch processing scheduling method, device, equipment and storage medium for GPU reasoning. Background Art

[0002] In recent years, deep learning has achieved remarkable success in computer vision and natural language processing, leading to the increasing popularity of artificial intelligence applications such as smart monitoring, autonomous driving, voice assistants, and smart customer service. In order to meet user needs and improve their experience, efficient deep learning inference is essential for deploying AI solutions in real-time scenarios.

[0003] In artificial intelligence services, ensuring timely response is crucial, especially for real-time reasoning tasks. However, this process requires overcoming the well-known von Neumann bottleneck, which is a significant computational latency caused by frequent data switching between memory and processor during processing. In order to effectively alleviate this problem and maximize the utilization of graphics processing unit (GPU) resources, multiple reasoning tasks can be organized and executed in a batch manner. This batching strategy improves system throughput and reduces reasoning cost by aggregating multiple tasks into a single batch task within a given time window. Although increasing the batch size can improve system throughput, a larger batch size will increase the waiting latency of tasks and the reasoning latency, thereby increasing the overall latency. Therefore, the batch size used for reasoning must carefully balance the trade-off between task latency and system throughput.

[0004] Emerging deep learning service systems optimize system performance by setting the maximum batch size and maximum waiting time for inference tasks. Once the number of queued tasks reaches the maximum batch size, or the waiting delay of the earliest inference task reaches the maximum waiting delay, the system organizes the queued tasks into batches for processing. Moreover, the next batch of inferences can only be started after a batch of inferences returns. However, this mechanism is only effective under stable task loads, while the task loads commonly faced by online services are usually random, which may lead to high response delays under low loads and low throughput under high loads. Under low loads, the response delay of the first inference task in the batch is delayed by at least the maximum waiting delay, and the processing delay is also increased due to batching. Under high loads, even if there are a large number of inference tasks waiting to be processed, only the maximum batch size of tasks can be processed each time, resulting in low throughput. Summary of the invention

[0005] Purpose of the invention: Based on the above shortcomings, the present invention is aimed at the batch processing scenario of GPU inference. When the task release information is known, a dynamic programming algorithm is proposed. By effectively planning how to combine tasks into batches for scheduling, the task completion rate can be maximized. At the same time, when the task release time is unknown, an online scheduling algorithm based on competition ratio is proposed. It can adapt to the random fluctuations of task load and make real-time decisions on whether to preempt the GPU, thereby ensuring a stable task completion rate and effectively improving the efficiency of GPU resources.

[0006] The present invention also provides a batch processing scheduling device, computer equipment and storage medium for GPU reasoning.

[0007] Technical solution: A batch scheduling method for GPU inference, including the following steps:

[0008] Obtain the device and task information in the computing system, including the number of devices, task set, release time of each task, and completion time requirement. Establish batch delay functions for different batch sizes and establish an optimization problem for planning the batch allocation method of tasks under specified constraints. The goal is to maximize the completion rate of tasks.

[0009] For the optimization problem, different methods are used to determine the task batch allocation method according to different processing scenarios. In the offline scenario, a dynamic programming algorithm is used to plan the task set into each batch while satisfying the batch size constraints and task delay constraints. In the online scenario, an online scheduling algorithm based on the competition ratio that allows GPU preemption is used to adapt to the random fluctuations of the task load.

[0010] Furthermore, the batch delay function is expressed as:

[0011] p(b)=μ+σb

[0012] p(b) represents the batch processing delay, b is the batch processing size, and the parameters μ and σ are determined by the scene and GPU hardware.

[0013] Furthermore, an optimization problem for batch allocation of planning tasks is established under specified constraints, including:

[0014] Suppose there are N devices, each device has a task, and the task set is expressed as Represent the batch collection as in The optimization problem is formulated as follows:

[0015]

[0016]

[0017] st represents the constraint condition, r n is the release time of the task, D is the completion delay of the task, that is, the task must be completed within r n +D returns the result before, K is the number of batches.

[0018] Furthermore, in offline scenarios, a dynamic programming algorithm is used to plan the task set into batches while satisfying the batch size constraints and task delay constraints, including:

[0019] Define a variable B(n, t), where represents the maximum number of tasks that can be completed from the task sequence {n, ..., N} when the GPU is idle at time t. The batch start time set Φ is defined according to the batch start time being the release time of a task or the processing completion time of a batch;

[0020] Traverse the task sequence J from back to front, that is, n = N, ..., 1, for Calculate the maximum value of B(n, t) using the recursive formula under given constraints;

[0021] Calculate B(1,0) and obtain the optimal task scheduling solution.

[0022] Furthermore, the recursive formula is expressed as:

[0023]

[0024] Where b≤b max Indicates that the batch size does not exceed the upper limit, n+b≤N indicates that the task number does not exceed the upper limit; max{t, r n+b-1}+p(b)-r n ≤D means that the delay constraint of the task is met.

[0025] Furthermore, in online scenarios, an online scheduling algorithm based on competition ratio that allows GPU preemption is used to adapt to random fluctuations in task loads, including:

[0026] e) Calculate the GPU task preemption coefficient β based on the competition ratio;

[0027] f) When a new task arrives at time t, calculate the set of valid tasks ε(t) that meet the delay constraint at this time;

[0028] g) determining whether the GPU is in an idle state at time t;

[0029] h) If the GPU is idle at time t, select the largest valid task set from ε(t) and start batch processing immediately at that time; if the GPU is occupied by a batch at time t, if the largest valid task set in ε(t) meets the preemption condition, interrupt the current task set and start batch processing of the largest valid task set;

[0030] Whenever a new task arrives, repeat b)-d).

[0031] Furthermore, the maximum set of valid tasks that meets the preemption condition is expressed as: in is the largest valid batch in ε(t), is the batch occupying the GPU at time t.

[0032] A batch processing scheduling device for GPU inference, comprising:

[0033] The model building module is used to obtain the equipment and task information in the computing system, including the number of equipment, task set, release time of each task, and completion time requirement, establish a batch processing delay function model, and establish an optimization problem for planning the batch allocation method of tasks under specified constraints, with the goal of maximizing the completion rate of tasks;

[0034] The problem-solving module is used to determine the task batch allocation method according to different processing scenarios using different methods for the optimization problem. In the offline scenario, a dynamic programming algorithm is used to plan the task set into each batch while satisfying the batch size constraint and the task delay constraint. In the online scenario, an online scheduling algorithm based on the competition ratio that allows the preemption of the GPU is used to adapt to the random fluctuation of the task load.

[0035] The present invention also provides a computer device, comprising: one or more processors; a memory; and one or more programs, wherein the one or more programs are stored in the memory and are configured to be executed by the one or more processors, and when the programs are executed by the processors, the steps of the batch scheduling method for GPU inference as described above are implemented.

[0036] The present invention also provides a computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the steps of the batch scheduling method for GPU inference as described above are implemented.

[0037] Beneficial effects: In order to solve the challenges of high response delay under low load and low throughput under high load caused by random task load faced by existing batch processing methods in online services, the present invention proposes a flexible batch processing mechanism in the process of online task scheduling, and allows tasks to preempt GPU to adapt to the random fluctuations of task load. Although the exclusive GPU can ensure the stability of real-time task completion, when the task load surges, if the GPU is occupied by a few tasks, a large number of tasks that have arrived cannot be batched immediately and need to wait for the GPU to complete processing, which may cause a large number of tasks to miss the delay constraint, reducing the GPU efficiency. Previous work has proposed a reset-based preemption scheme, whose microsecond kernel preemption is negligible for millisecond task delay constraints. The elastic batch processing strategy proposed by the present invention can optimize the utilization of GPU resources while ensuring real-time performance, thereby improving the efficiency and responsiveness of the entire system. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] Figure 1 It is a task batch scheduling application scenario diagram of a computing system;

[0039] Figure 2 It is a flow chart of the batch scheduling method for GPU inference;

[0040] Figure 3 It is a graph of batch inference latency at different batch sizes;

[0041] Figure 4 It is a curve chart of the average inference latency of a single task under different batch sizes;

[0042] Figure 5 It is a flow chart of the dynamic programming algorithm;

[0043] Figure 6 It is the flow chart of online scheduling algorithm. DETAILED DESCRIPTION

[0044] The technical solution of the present invention is further described below in conjunction with the accompanying drawings.

[0045] Figure 1 The following is a diagram of a distributed computing system task batch scheduling application scenario. The design computing system has N devices, each device has a task, and unloads the task to the same server through a wired connection. Before unloading the task, each device sends the server a summary of the upcoming task, including the task release time r n , relative deadline D, the server assigns N tasks to each batch according to the information Or discard, for the kth batch, the server needs to wait for B kAll tasks are published in the server, and these tasks are collected as a batch and input into the server inference model for batch processing. n +D before returning the result, otherwise the task is considered failed. After the server completes the processing, it returns the inference result to each device.

[0046] The batch scheduling method for GPU reasoning proposed in the present invention is applied to the server to perform batch processing allocation of tasks, referring to Figure 2 , the method comprises the following steps:

[0047] S1, model the task and batch model and reasoning model in the computing system, and based on this, model the task scheduling problem;

[0048] S2, in offline scenarios, uses a dynamic programming algorithm to plan the task set into batches while satisfying the task delay constraints to maximize the task completion rate;

[0049] S3, in the online scenario, proposes an online scheduling algorithm based on competition ratio that allows GPU preemption to adapt to the random fluctuations of task load.

[0050] In step S1, a basic model of tasks and batches is constructed, including relevant attributes of tasks and basic constraints that batches need to satisfy; and for different batch sizes, a batch delay function model is derived; thereby constructing an optimization target model and defining the constraints that need to be satisfied.

[0051] As above combined Figure 1 The system application scenario described has N devices, each of which has a task, and is offloaded to the same server for reasoning through a wired connection. The task set is represented as The release time of each task is r n , and must be in r n +D before returning the result, otherwise the task is considered failed. The batch set is represented as in The start processing time of each batch is t i, the batch delay function is expressed as p(b)=μ+σb, where the batch size is b, the parameters μ and σ are related to the specific scenario and GPU hardware. The inventors of this application implemented the GoogLeNet model on the JETSON TX1 platform and the TESLAP4 platform for a data set of 224×224 pixels to fit the batch delay function and obtain the relevant parameters. . There are several constraints in the processing process. First, the batch needs to wait for all tasks in the batch to be released before processing can begin; second, each batch needs to be executed sequentially; third, the tasks must be completed before the deadline. It is required to plan the batch allocation method of tasks to maximize the completion rate of tasks while satisfying the above constraints. The optimization target model is as follows:

[0052]

[0053]

[0054]

[0055] Figure 3 and Figure 4 The curves of batch inference latency and average inference latency of a single task under different batch sizes are shown. These data are obtained by running the GoogLeNet model using the ImageNet dataset on the JETSON TX1 and TESLAP4 platforms. The observations show that as the batch size increases, the batch inference latency shows a linear growth trend, while the average inference latency of a single task continues to decrease before reaching a minimum point. This phenomenon can be attributed to the parallel processing capability of the GPU, in which the memory access time of multiple tasks is amortized. Therefore, a reasonable choice of batch size is crucial to optimizing overall performance.

[0056] Figure 5 This is the flow chart of the dynamic programming algorithm.

[0057] After the algorithm starts, the batch start time set Φ is calculated first. Since the start processing time of a batch can only be the release time of a task or the processing completion time of a batch, Φ is defined as follows:

[0058] Φ={r n +l1μ+l2σ|n=1,...,N,l1=0,1,...,N,l2=0,1,...,N}

[0059] At the same time, a variable B(n, t) is defined, where It represents the maximum number of tasks that can be completed from the task sequence {n, ..., N} when the GPU is idle at time t, (t∈Φ). Next, B(N+1, t) is initialized and its value is set to 0.

[0060] Traverse the task sequence from back to front, that is, n = N, ..., 1, and calculate B(n, t), where B(n, t) has two possible results: one is not processing task n, that is, B(n, t) = B(n+1, t); the other is to batch process b tasks starting from task n, that is, B(n, t) = B(n+b, max{t, r n+b-1}+p(b))+b, and several constraints need to be met: first, the batch size does not exceed the upper limit, that is, b≤b max ; Secondly, the task number does not exceed the upper limit, that is, n+b≤N; In addition, the task delay constraint must be met, that is, max{t,r n+b-1}+p(b)-r n ≤D, B(n, t) means comparing the two results and taking the maximum value.

[0061] Therefore, for t∈Φ, we have:

[0062]

[0063] The optimization problem of the present invention is to solve B(1,0). Since the scale of |Φ| is O(n 3 ), and the time complexity of calculating each B(n, t) is O(b max ), so the time complexity of the entire dynamic programming algorithm is O(n 4 b max ).

[0064] Therefore, based on this recursive equation, we can use the dynamic programming method to solve the problem in O(n 4 b max ) time, accurately calculate B(1, 0) and obtain a scheduling plan that maximizes the completion rate of all tasks. The algorithm ends here.

[0065] Figure 6 This is the flow chart of the online scheduling algorithm.

[0066] After the algorithm starts, the task preemption coefficient β is calculated based on the competition ratio. The calculation process is as follows:

[0067] If at time t, task i satisfies tr n +p(1)≤D, then the task is said to be valid at time t. Let ε(t) be the set of valid tasks at time t.

[0068] If for all j=1,...,k-1, quilt Preempt, and A sequence of batches is called a sequence of batches if it is preceded by either an idle time interval or a completed batch. is the preemption chain. If it is not interrupted, For a basic preemption chain.

[0069] A basic preemption chain The following conditions are met:

[0070]

[0071]

[0072] Given any task sequence The scheduling of the algorithm can be represented as consisting of several basic preemption chains, denoted as Each of these So the target value obtained by the algorithm is:

[0073]

[0074] definition That is, the set of tasks that have not been processed includes:

[0075]

[0076] Consider for An optimal schedule consisting of a series of complete batches In execution During this period, there may be multiple batches The start time is If α>2, then It can be concluded that because therefore

[0077] The above equation also holds true when α = 1, so,

[0078]

[0079]

[0080] So the algorithm achieves a competitive ratio, i.e.

[0081]

[0082] when The right side of the above inequality reaches its minimum value when This coefficient value can be used directly in practical applications.

[0083] After calculating the task preemption coefficient β based on the competition ratio, when a new task arrives at time t, the system updates the current valid task set ε(t) and determines whether the GPU is idle at this time.

[0084] If the GPU is idle, the largest valid batch is selected from ε(t) and starts processing on the GPU at time t; otherwise, a batch is running on the GPU If the largest valid batch in ε(t) satisfy Then interrupt And start If such a batch does not exist, no action is taken.

[0085] Whenever a new task arrives, the above process is repeated.

[0086] The present invention is aimed at the batch processing scenario of GPU inference. When the task release information is known, a dynamic programming algorithm is proposed. By effectively planning how to combine tasks into batches for scheduling, the task completion rate can be maximized. At the same time, an online scheduling algorithm based on competition ratio is proposed when the task release time is unknown. The algorithm can adapt to the random fluctuation of task load and make real-time decisions on whether to preempt the GPU, thereby ensuring a stable task completion rate and effectively improving the efficiency of GPU resources.

[0087] Based on the same technical concept as the method embodiment, the present invention also provides a batch processing scheduling device for GPU reasoning, which can be used as a server in a computing system, and includes:

[0088] The model building module is used to obtain the equipment and task information in the computing system, including the number of equipment, task set, release time of each task, and completion time requirement, establish a batch processing delay function model, and establish an optimization problem for planning the batch allocation method of tasks under specified constraints, with the goal of maximizing the completion rate of tasks;

[0089] The problem-solving module is used to determine the task batch allocation method according to different processing scenarios using different methods for the optimization problem. In the offline scenario, a dynamic programming algorithm is used to plan the task set into each batch while satisfying the batch size constraint and the task delay constraint. In the online scenario, an online scheduling algorithm based on the competition ratio that allows the preemption of the GPU is used to adapt to the random fluctuation of the task load.

[0090] It should be understood that the batch scheduling device for GPU inference in the embodiment of the present invention can implement all the technical solutions in the above-mentioned method embodiment, and the functions of its various functional modules can be specifically implemented according to the methods in the above-mentioned method embodiments. The specific implementation process can refer to the relevant description in the above-mentioned embodiments, which will not be repeated here.

[0091] The present invention also provides a computer device, comprising: one or more processors; a memory; and one or more programs, wherein the one or more programs are stored in the memory and are configured to be executed by the one or more processors, and when the programs are executed by the processors, the steps of the batch scheduling method for GPU inference as described above are implemented.

[0092] The present invention also provides a computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the steps of the batch scheduling method for GPU inference as described above are implemented.

[0093] It will be appreciated by those skilled in the art that embodiments of the present invention may be provided as methods, devices (systems), computer equipment or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program codes.

[0094] The present invention is described with reference to a flowchart of a method according to an embodiment of the present invention. It should be understood that each process in the flowchart and a combination of processes in the flowchart can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the process in the flowchart. Figure 1 A device that specifies functions in a process or multiple processes.

[0095] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A function specified in a process or multiple processes.

[0096] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 The steps of a specified function in a process or multiple processes.

[0097] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the above embodiments, ordinary technicians in the relevant field should understand that the specific implementation methods of the present invention can still be modified or replaced by equivalents. Any modification or equivalent replacement that does not depart from the spirit and scope of the present invention should be covered within the scope of protection of the claims of the present invention.

Claims

1. A batch scheduling method for GPU inference, characterized in that: The following steps are involved: Obtain the device and task information in the computing system, including the number of devices, task set, release time of each task, and completion time requirement. Establish batch delay functions for different batch sizes and establish an optimization problem for planning the batch allocation method of tasks under specified constraints. The goal is to maximize the completion rate of tasks. For the optimization problem, different methods are used to determine the task batch allocation method according to different processing scenarios. In the offline scenario, a dynamic programming algorithm is used to plan the task set into each batch while satisfying the batch size constraint and task delay constraint. In online scenarios, an online scheduling algorithm based on contention ratio that allows GPU preemption is used to adapt to random fluctuations in task load.

2. The method according to claim 1, characterized in that The batch delay function is expressed as: p(b)=μ+σb p(b) represents the batch processing delay, b is the batch processing size, and the parameters μ and σ are determined by the scene and GPU hardware.

3. The method according to claim 1, characterized in that Establish an optimization problem for planning the batch allocation of tasks under specified constraints, including: Suppose there are N devices, each device has a task, and the task set is expressed as Represent the batch collection as in The optimization problem is formulated as follows: st represents the constraint condition, r n is the release time of the task, D is the completion delay of the task, that is, the task must be completed within r n +D returns the result before, K is the number of batches.

4. The method according to claim 3, characterized in that In offline scenarios, a dynamic programming algorithm is used to plan the task set into batches while satisfying the batch size constraints and task latency constraints, including: Define a variable B(n,t), where It represents the maximum number of tasks that can be completed from the task sequence {n,…,N} when the GPU is idle at time t. The batch start time is defined as the release time of a task or the processing completion time of a batch. Traverse the task sequence from back to front That is, n=N,…,1, for Use the recursive formula to calculate the maximum value of B(n,t) under given constraints; Calculate B(1,0) and obtain the optimal task scheduling solution.

5. The method according to claim 4, characterized in that The recursive formula is expressed as: Where b≤b max Indicates that the batch size does not exceed the upper limit, n+b≤N indicates that the task number does not exceed the upper limit; max{t,r n+b-1 }+p(b)-r n ≤D means that the delay constraint of the task is met.

6. The method according to claim 1, characterized in that In online scenarios, an online scheduling algorithm based on competition ratio that allows GPU preemption is used to adapt to random fluctuations in task loads, including: a) Calculate the GPU task preemption coefficient β based on the competition ratio; b) When a new task arrives at time t, calculate the set of valid tasks ε(t) that meet the delay constraint at this time; c) determining whether the GPU is in an idle state at time t; d) If the GPU is idle at time t, select the largest valid task set from ε(t) and start batch processing immediately at that time; if the GPU is occupied by a batch at time t, if the largest valid task set in ε(t) meets the preemption condition, interrupt the current task set and start batch processing of the largest valid task set; Whenever a new task arrives, repeat b)-d).

7. The method according to claim 6, characterized in that The maximum set of valid tasks that meet the preemption conditions is expressed as: in is the largest valid batch in ε(t), is the batch occupying the GPU at time t.

8. A batch processing scheduling device for GPU inference, characterized in that: The following steps are involved: The model building module is used to obtain the equipment and task information in the computing system, including the number of equipment, task set, release time of each task, and completion time requirement, establish a batch processing delay function model, and establish an optimization problem for planning the batch allocation method of tasks under specified constraints, with the goal of maximizing the completion rate of tasks; A problem-solving module is used to determine the task batch allocation method according to different processing scenarios for the optimization problem using different methods. In an offline scenario, a dynamic programming algorithm is used to plan the task set into each batch while satisfying the batch size constraint and the task delay constraint. In online scenarios, an online scheduling algorithm based on contention ratio that allows GPU preemption is used to adapt to random fluctuations in task load.

9. A computer device, characterized in that: include: one or more processors; Memory; And one or more programs, wherein the one or more programs are stored in the memory and are configured to be executed by the one or more processors, and when the programs are executed by the processors, the steps of the batch scheduling method for GPU inference as described above are implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the batch scheduling method for GPU reasoning as described above are implemented.

Images