A fine-grained cooperative preemption method for multi-task GPU and medium
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- BEIJING NEW ENERGY VEHICLE TECH INNOVATION CENT CO LTD
- Filing Date
- 2026-03-12
- Publication Date
- 2026-06-26
Smart Images
Figure CN122285268A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of GPU task scheduling, and more specifically, to a fine-grained cooperative preemption method and medium for multi-tasking GPUs. Background Technology
[0002] Graphics processing units (GPUs) have become a compelling and crucial computing platform in cloud devices and data centers. GPU-based computing acceleration is a major driver of performance improvements in high-performance computing (HPC) systems, large-scale cloud infrastructures, and data analytics applications. In recent years, GPU computing and storage resources have continued to grow, leading many systems to tend to execute multiple tasks concurrently on a single GPU. For example, in autonomous driving, a GPU might simultaneously perform object recognition and UI rendering tasks. When multiple tasks are executed concurrently, GPU resources need to be scheduled and allocated.
[0003] Preemptive scheduling is a typical scheduling mechanism that dynamically allocates stream processor resources among different tasks by pausing the task being executed by the stream processor and reallocating the stream processor to other tasks. Since the execution progress of thread blocks within the same task kernel is often inconsistent, applying the same preemption technique to all thread blocks in all preempted stream processors can easily lead to significant preemption overhead, thereby degrading system performance. Developing low-performance GPU task preemption methods is a major problem that urgently needs to be solved.
[0004] Existing multi-tasking GPU stream processor preemption methods typically employ a uniform preemption technique across all stream processors to be preempted. For example, if the current kernel has not yet completed the execution of the thread block of the preempted task, exhaustion preemption is used for all stream processors; otherwise, context switching preemption is used for all stream processors. While the uniform preemption method offers simple decision-making, it makes it difficult to measure preemption overhead and fails to fully explore optimization space for reducing preemption overhead.
[0005] Therefore, it is necessary to develop a fine-grained cooperative preemption method and medium for multi-tasking GPUs.
[0006] The information disclosed in the background section of this invention is intended only to enhance the understanding of the general background of this invention, and should not be construed as an admission or in any way implying that such information constitutes prior art known to those skilled in the art. Summary of the Invention
[0007] This invention proposes a fine-grained collaborative preemption method and medium for multi-tasking GPUs, which can solve the problem of high preemption overhead in stream processor task switching and achieve efficient stream processor preemption and task resource adjustment in multi-tasking GPUs.
[0008] In a first aspect, embodiments of this disclosure provide a fine-grained cooperative preemption method for multi-tasking GPUs, including: Determine the type of preemption scheme for the thread block, including exhaustion, context switching, and flushing; Collect information on all thread blocks in the stream processor to be preempted, and construct the first cost function for that stream processor; For each thread block, determine and categorize the available preemption schemes. Calculate the throughput and latency overhead of each thread block under the preemption scheme that can be adopted, and then calculate the corresponding second cost function; The preemption scheme for each thread block is determined with the goal of minimizing the cost function of the stream processor.
[0009] Preferably, the first cost function of the stream processor is:
[0010] in, For the first cost function, For the throughput overhead of the stream processor, Let α be the latency overhead of the stream processor, and α be the weight.
[0011] Preferably, determining the preemption scheme that can be adopted for each thread block includes: When the kernel of the task to which the thread block belongs already contains a thread block that has completed execution, exhaustion can be used; When a thread block satisfies the idempotency condition, flushing can be used. The idempotency condition point is implemented by code instrumentation. If the execution process of a thread block crosses the idempotency condition point, the thread block is marked as not satisfying the idempotency condition. Context switching can be used in any thread block.
[0012] Preferably, the thread block categories include: The first category includes thread blocks that can only use context switching; The second category includes thread blocks that can employ exhaustion and context switching; The third category includes thread blocks that can employ context switching and flushing; The fourth category includes thread blocks that can employ exhaustion, context switching, and flushing.
[0013] Preferably, the preemption scheme for each thread block is determined with the goal of minimizing the first cost function of the stream processor, including: For the first category of thread blocks, the corresponding preemption scheme is context switching; For the third category of thread blocks, the corresponding preemption scheme is determined with the goal of minimizing the second cost function of each thread block; For the fourth category of thread blocks, a scheme is determined by comparing the second cost function corresponding to context switching and flushing; For the second category of thread blocks, the size of the first cost function corresponding to context switching and exhaustion is determined sequentially for each thread block, and a preemption scheme with a smaller first cost function is adopted.
[0014] Preferably, for thread blocks of the fourth category, the scheme for determining the second cost function corresponding to context switching and flushing is as follows: The size of the second cost function corresponding to context switching and flushing is determined sequentially for each thread block. If the second cost function corresponding to flushing is smaller, the preemption scheme of the thread block is used for flushing. If the second cost function corresponding to context switching is smaller, the flushing scheme is deleted and the thread block is merged into the second category.
[0015] Preferably, the throughput overhead of the stream processor is the sum of the throughput overhead of each thread block.
[0016] Preferably, determining the latency overhead of the stream processor includes: Identify the thread blocks in the stream processor that employ exhaustion and context switching respectively; The maximum value of the latency overhead of the exhausted thread block is calculated as the first data. The second data is calculated by summing the latency overhead of thread blocks that use context switching; The maximum value between the first and second data is used as the latency overhead of the stream processor.
[0017] Preferably, the second cost function is:
[0018] in, The cost function for the thread block. For the throughput overhead of thread blocks, This refers to the latency overhead of the thread block.
[0019] Secondly, embodiments of this disclosure also provide a computer-readable storage medium storing a computer program that, when executed by a processor, implements the fine-grained cooperative preemption method for multi-tasking GPUs.
[0020] Its beneficial effects are as follows: 1. Take into account both latency and throughput overhead to minimize the overall preemption cost; 2. Take into account both the execution state of the thread block and the overall preemption behavior of the stream processor; 3. Select preemption techniques at the thread block level. Thread blocks within the same stream processor can use different preemption techniques, and different stream processors can independently adopt different cooperative preemption schemes. 4. For any remaining unanswered memory access requests within the managed stream processor, release the preempted stream processor early to receive new task thread blocks.
[0021] The methods and apparatus of the present invention have other features and advantages that will be apparent from or will be set forth in detail in the accompanying drawings and following detailed description, which together serve to explain the particular principles of the invention. Attached Figure Description
[0022] The above and other objects, features and advantages of the present invention will become more apparent from the more detailed description of exemplary embodiments of the invention in conjunction with the accompanying drawings, wherein the same reference numerals generally represent the same parts.
[0023] Figure 1 A flowchart illustrating the steps of a fine-grained cooperative preemption method for multi-tasking GPUs according to an embodiment of the present invention is shown. Detailed Implementation
[0024] Preferred embodiments of the invention will now be described in more detail. While preferred embodiments of the invention are described below, it should be understood that the invention can be implemented in various forms and should not be limited to the embodiments set forth herein.
[0025] To facilitate understanding of the solutions and effects of the embodiments of the present invention, two specific application examples are given below. Those skilled in the art should understand that these examples are merely for the purpose of understanding the present invention, and any specific details therein are not intended to limit the present invention in any way.
[0026] Example 1
[0027] Figure 1 A flowchart illustrating the steps of a fine-grained cooperative preemption method for multi-tasking GPUs according to an embodiment of the present invention is shown.
[0028] like Figure 1 As shown, this fine-grained cooperative preemption method for multi-tasking GPUs includes: Step 101: Determine the type of preemption scheme for the thread block, including exhaustion, context switching, and flushing; Step 102: Collect information on all thread blocks in the stream processor to be preempted, and construct the first cost function of the stream processor; Step 103: Determine and classify the available preemption schemes for each thread block; Step 104: Calculate the throughput and latency overhead of each thread block under the preemption scheme that can be adopted, and then calculate the corresponding second cost function; Step 105: Determine the preemption scheme for each thread block with the goal of minimizing the cost function of the stream processor.
[0029] In one example, the first cost function of the stream processor is:
[0030] in, For the first cost function, For the throughput overhead of the stream processor, Let α be the latency overhead of the stream processor, and α be the weight.
[0031] In one example, determining the available preemption schemes for each thread block includes: When the kernel of the task to which the thread block belongs already contains a thread block that has completed execution, exhaustion can be used; When a thread block satisfies the idempotency condition, flushing can be used. The idempotency condition point is implemented by code instrumentation. If the execution process of the thread block crosses the idempotency condition point, the thread block is marked as not satisfying the idempotency condition. Idempotency condition: The result of re-executing the thread block is the same as before. Code instrumentation: Inserting marker code in the program, executing the marker code marks the thread block as not satisfying the idempotency condition. Context switching can be used in any thread block.
[0032] In one example, thread block categories include: The first category includes thread blocks that can only use context switching; The second category includes thread blocks that can employ exhaustion and context switching; The third category includes thread blocks that can employ context switching and flushing; The fourth category includes thread blocks that can employ exhaustion, context switching, and flushing.
[0033] In one example, the preemption scheme for each thread block is determined with the goal of minimizing the first cost function of the stream processor as the objective. For the first category of thread blocks, the corresponding preemption scheme is context switching; For the third category of thread blocks, the corresponding preemption scheme is determined with the goal of minimizing the second cost function of each thread block; For the fourth category of thread blocks, a scheme is determined by comparing the second cost function corresponding to context switching and flushing; For the second category of thread blocks, the size of the first cost function corresponding to context switching and exhaustion is determined sequentially for each thread block, and a preemption scheme with a smaller first cost function is adopted.
[0034] In one example, for thread blocks of the fourth category, the method for determining the comparison between context switching and the second cost function corresponding to flushing is as follows: The size of the second cost function corresponding to context switching and flushing is determined sequentially for each thread block. If the second cost function corresponding to flushing is smaller, the preemption scheme of the thread block is used for flushing. If the second cost function corresponding to context switching is smaller, the flushing scheme is deleted and the thread block is merged into the second category.
[0035] In one example, the throughput overhead of the stream processor is the sum of the throughput overhead of each thread block.
[0036] In one example, determining the latency overhead of a stream processor includes: Identify the thread blocks in the stream processor that employ exhaustion and context switching respectively; The maximum value of the latency overhead of the exhausted thread block is calculated as the first data. The second data is calculated by summing the latency overhead of thread blocks that use context switching; The maximum value between the first and second data is used as the latency overhead of the stream processor.
[0037] In one example, the second cost function is:
[0038] in, The cost function for the thread block. For the throughput overhead of thread blocks, This refers to the latency overhead of a thread block. The throughput overhead and latency overhead of a thread block are determined based on the preemption scheme used.
[0039] There are no conflicts in the thread block classification and preemption scheme decision-making methods. The execution boundary of "fine-grained cooperative preemption" is to complete the selection of preemption schemes for all thread blocks to be preempted.
[0040] Specifically, in a GPU, a thread block is the smallest unit of granularity that establishes a mapping relationship with a stream processor. Cooperative preemption at the thread block level can maximize the exploration of optimization space to reduce preemption costs.
[0041] The stream processor supports three thread block preemption techniques: exhaustion, context switching, and flush preemption. 1. Thread blocks marked as exhausted continue to execute normally. The exhaustion scheme waits for all currently executing thread blocks to finish executing. The process is to wait for the thread blocks to continue executing until they finish. The termination condition is that the thread blocks have finished executing.
[0042] 2. Thread blocks marked for context switching are managed by the context saving module. The module sequentially stores the context information of the thread blocks to be saved into the context stack via the context transfer channel in a polling manner. It then sends the thread block information and context storage address to the preempted thread block issuance queue in the thread block scheduler. Thread blocks not polled can continue normal execution. The context switching scheme suspends thread block execution, saves the state of all currently executing thread blocks, and allows the executable stream processor to re-import the context, resuming execution from the paused position. The process involves pausing thread block execution, saving and re-importing the context state, and resuming normal execution after the thread block context re-import is complete.
[0043] 3. Thread blocks marked for flushing terminate execution, and their information is sent to the preemptible thread block emission queue in the thread block scheduler. A thread block can only be marked for flushing preemption if it satisfies idempotency. The flushing scheme terminates the execution of a thread block without saving its state, records information on all terminated thread blocks, reloads the flushed thread blocks into the thread block emission queue, and allows the executable stream processor to re-execute the flushed thread blocks from the beginning. The process involves terminating the execution of a thread block without saving its state; the termination condition is the completion of the thread block's execution.
[0044] By evaluating the execution state of thread blocks within a stream processor and their impact on the overall preemption overhead of the stream processor, the optimal cooperative preemption strategy is selected, and independent evaluation and selection are performed for all stream processors to be preempted.
[0045] Although preemption techniques are selected at the thread block level, the preemption target remains at the stream processor level; therefore, the preemption cost also needs to be measured at the stream processor level. The throughput overhead of thread blocks using exhaustion preemption is evaluated by the difference between the number of instructions already executed and the number of instructions executed by the thread block with the fastest execution progress. The latency overhead of thread blocks using exhaustion preemption is evaluated by dividing the maximum remaining instructions by the average number of instructions executed per cycle (IPC) of the thread block. The latency overhead of thread blocks using context switching preemption is evaluated by calculating the context size and context transfer bandwidth. Since context switching involves store and load, the throughput overhead of thread blocks using context switching preemption is evaluated by multiplying the average thread block IPC by twice the latency. The average thread block IPC is obtained by dividing the total number of instructions in the thread block by the average number of execution cycles of the thread block. If no thread block has finished executing, it is obtained by dividing the sum of the number of instructions executed by all thread blocks by the sum of the number of execution cycles. The latency overhead of thread blocks using flush preemption can be considered zero, while the throughput overhead is evaluated by the number of instructions executed. The latency overhead of a thread block using exhaustion preemption can be masked by the largest of the two, while the latency overhead of a thread block using context switching preemption is cumulative. For a stream processor, its latency overhead is determined by the maximum of the latency overhead of a thread block using exhaustion preemption and the latency overhead of a thread block using context switching preemption, while the throughput overhead is evaluated by summing the throughput overhead of all thread blocks. Based on this, the cooperative preemption strategy decision includes the following steps: Collect information on all thread blocks in the stream processors awaiting preemption. Establish the first cost function for the stream processor to synthesize throughput overhead. ) and delay overhead ( ): α is a weight used to quantify the importance of preemption latency. For example, when switching from a low-efficiency task to a high-efficiency task, the preemption strategy aims to maximize the efficiency of the stream processor, and can be set as the IPC difference between the stream processor's execution of the task before and after preemption. The weight α is:
[0046] When a fast-running thread block is preempted to free up stream processor resources to run a slow-running thread block, the weight is 0; when a slow-running thread block is preempted to free up stream processor resources to run a fast-running thread block, the weight is the difference between the average number of instructions executed per cycle of the fast-running thread block and the average number of instructions executed per cycle of the slow-running thread block. The average number of instructions executed per cycle for a fast-running thread block. The average number of instructions executed per cycle for slow-running thread blocks.
[0047] For each thread block, the available preemption schemes are determined and categorized. Since assessing the overhead of exhaustion preemption requires information about the thread block's execution delay, exhaustion preemption can only be used if the kernel of the task to which the thread block belongs already contains a completed thread block. When a thread block completes execution, it sends a completion signal to the task kernel manager, which records the thread block's completion information. To ensure consistent results when re-executing thread blocks, flush preemption can only be used if the thread block meets the idempotency condition. The idempotency condition point is implemented through code instrumentation; if the thread block's execution process crosses the idempotency condition point, the thread block is marked as not meeting the idempotency condition. Context switching preemption has no usage conditions and can be used in any state.
[0048] The process for classifying thread blocks is as follows: Thread blocks are classified based on the preemption schemes they can employ. First, the state of the thread block to be preempted is checked to determine the available preemption schemes. If the kernel of the task to which the thread block belongs already has a completed thread block, exhaustion can be used; if the thread block satisfies the idempotency condition, flushing can be used; any thread block can use context switching. Therefore, thread block categories include: The first category includes thread blocks that can only use context switching; The second category includes thread blocks that can employ exhaustion and context switching; The third category includes thread blocks that can employ context switching and flushing; The fourth category includes thread blocks that can employ exhaustion, context switching, and flushing.
[0049] Evaluate the throughput and latency overhead of the preempted thread block under different preemption techniques, and then calculate the corresponding second cost function:
[0050] in, The cost function for the thread block. For the throughput overhead of thread blocks, This refers to the latency overhead of a thread block. Specifically, it involves calculating the throughput overhead and latency overhead of each thread block under the preemption scheme that can be employed: The latency overhead of exhausting the thread block is:
[0051] in, This represents the number of unexecuted instructions remaining in the thread block. This represents the average number of instructions executed per cycle for a thread block.
[0052] The overhead of exhausting the thread block throughput is:
[0053] in, This represents the number of instructions executed by the fastest-running thread block in the same program. This represents the number of instructions executed by the thread block.
[0054] The context switching thread block latency overhead is:
[0055] in, For context size, For context transmission bandwidth.
[0056] The throughput overhead of context switching thread blocks is:
[0057] in, For context switching thread block latency overhead, This represents the average number of instructions executed per cycle for a thread block.
[0058] The latency overhead of flushing thread blocks is:
[0059] The throughput overhead of flushing thread blocks is:
[0060] in, This represents the number of instructions executed by the thread block.
[0061] The cost function of a stream processor is calculated based on the overall throughput overhead and latency overhead of the stream processor.
[0062] Overall throughput overhead of stream processors:
[0063] Overall latency overhead of stream processors:
[0064] The execution logic for preemption scheme decision-making is as follows: With the goal of minimizing the first cost function of the stream processor, a preemption scheme is determined for each thread block. This involves selecting a preemption scheme from exhaustion, context switching, and flushing for each thread block. Specifically, this includes: For the first category of thread blocks, the corresponding preemption scheme is context switching; For the third category of thread blocks, the corresponding preemption scheme is determined with the goal of minimizing the second cost function of each thread block; For thread blocks in the fourth category, the size of the second cost function corresponding to context switching and flushing is determined sequentially for each thread block. If the second cost function corresponding to flushing is smaller, the preemption scheme of the thread block is used for flushing. If the second cost function corresponding to context switching is smaller, the flushing scheme is deleted and the thread block is merged into the second category. For the second category of thread blocks, the size of the first cost function corresponding to context switching and exhaustion is determined sequentially for each thread block, and a preemption scheme with a smaller first cost function is adopted.
[0065] A thread block preemption technique table is added to the stream processor to mark the preemption technique to be used for all thread blocks. When performing a preemption operation on the stream processor, the thread blocks are marked with the preemption technique according to the preemption scheme.
[0066] After all thread blocks in a stream processor are preempted and cleared, the stream processor resources are released, allowing it to receive and execute new task thread blocks. The preempted thread block emission queue in the thread block scheduler injects the thread block into the appropriate stream processor when it is available to be emitted, and imports it into the context to continue execution or re-execute it based on the marked preemption information.
[0067] Example 2
[0068] This disclosure provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the fine-grained cooperative preemption method for multi-tasking GPUs.
[0069] A computer-readable storage medium according to embodiments of the present disclosure stores non-transitory computer-readable instructions. When these non-transitory computer-readable instructions are executed by a processor, all or part of the steps of the methods described in the foregoing embodiments of the present disclosure are performed.
[0070] The aforementioned computer-readable storage media include, but are not limited to: optical storage media (e.g., CD-ROM and DVD), magneto-optical storage media (e.g., MO), magnetic storage media (e.g., magnetic tape or portable hard drive), media with built-in rewritable non-volatile memory (e.g., memory card), and media with built-in ROM (e.g., ROM cartridge).
[0071] Those skilled in the art should understand that the above description of the embodiments of the present invention is only intended to illustrate the beneficial effects of the embodiments of the present invention, and is not intended to limit the embodiments of the present invention to any of the examples given.
[0072] The various embodiments of the present invention have been described above. These descriptions are exemplary and not exhaustive, nor are they limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments.
Claims
1. A fine-grained cooperative preemption method for multi-tasking GPUs, characterized in that, include: Determine the type of preemption scheme for the thread block, including exhaustion, context switching, and flushing; Collect information on all thread blocks in the stream processor to be preempted, and construct the first cost function for that stream processor; For each thread block, determine and categorize the available preemption schemes. Calculate the throughput and latency overhead of each thread block under the preemption scheme that can be adopted, and then calculate the corresponding second cost function; The preemption scheme for each thread block is determined with the goal of minimizing the cost function of the stream processor.
2. The fine-grained cooperative preemption method for multi-tasking GPUs according to claim 1, wherein, The first cost function of the stream processor is: in, For the first cost function, For the throughput overhead of the stream processor, Let α be the latency overhead of the stream processor, and α be the weight.
3. The fine-grained cooperative preemption method for multi-tasking GPUs according to claim 2, wherein, For each thread block, the available preemption schemes include: When the kernel of the task to which the thread block belongs already contains a thread block that has completed execution, exhaustion can be used; When a thread block satisfies the idempotency condition, flushing can be used. The idempotency condition point is implemented by code instrumentation. If the execution process of a thread block crosses the idempotency condition point, the thread block is marked as not satisfying the idempotency condition. Context switching can be used in any thread block.
4. The fine-grained cooperative preemption method for multi-tasking GPUs according to claim 3, wherein, Thread block categories include: The first category includes thread blocks that can only use context switching; The second category includes thread blocks that can employ exhaustion and context switching; The third category includes thread blocks that can employ context switching and flushing; The fourth category includes thread blocks that can employ exhaustion, context switching, and flushing.
5. The fine-grained cooperative preemption method for multi-tasking GPUs according to claim 4, wherein, The preemption scheme for each thread block is determined with the goal of minimizing the first cost function of the stream processor, including: For the first category of thread blocks, the corresponding preemption scheme is context switching; For the third category of thread blocks, the corresponding preemption scheme is determined with the goal of minimizing the second cost function of each thread block; For the fourth category of thread blocks, a scheme is determined by comparing the second cost function corresponding to context switching and flushing; For the second category of thread blocks, the size of the first cost function corresponding to context switching and exhaustion is determined sequentially for each thread block, and a preemption scheme with a smaller first cost function is adopted.
6. The fine-grained cooperative preemption method for multi-tasking GPUs according to claim 5, wherein, For thread blocks of the fourth category, the scheme for determining the second cost function corresponding to context switching and flushing is as follows: The size of the second cost function corresponding to context switching and flushing is determined sequentially for each thread block. If the second cost function corresponding to flushing is smaller, the preemption scheme of the thread block is used for flushing. If the second cost function corresponding to context switching is smaller, the flushing scheme is deleted and the thread block is merged into the second category.
7. The fine-grained cooperative preemption method for multi-tasking GPUs according to claim 6, wherein, The throughput overhead of a stream processor is the sum of the throughput overhead of each thread block.
8. The fine-grained cooperative preemption method for multi-tasking GPUs according to claim 7, wherein, Determining the latency overhead of a stream processor includes: Identify the thread blocks in the stream processor that employ exhaustion and context switching respectively; The maximum value of the latency overhead of the exhausted thread block is calculated as the first data. The second data is calculated by summing the latency overhead of thread blocks that use context switching; The maximum value between the first and second data is used as the latency overhead of the stream processor.
9. The fine-grained cooperative preemption method for multi-tasking GPUs according to claim 8, wherein, The second cost function is: in, The cost function for the thread block. For the throughput overhead of thread blocks, This refers to the latency overhead of the thread block.
10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the fine-grained cooperative preemption method for multitasking GPUs as described in any one of claims 1-9.