Collaborative work-stealing scheduler

Abstract

A method for use in a computing system having a central processing unit (CPU) and a graphics processing unit (GPU), the method comprising: assigning a first memory portion and a second memory portion to: a worker thread of a work-stealing scheduler and an execution unit that is part of the GPU; retrieving a task from a queue associated with the worker thread; having the worker thread detect whether a deadline condition for the task is met; if the deadline condition is not met, dividing the task into two or more additional tasks and adding the two or more additional tasks to the queue; if the deadline condition is met, storing a first data corresponding to the task in the second memory portion and issuing a memory fence fetch instruction; and storing a first value in the first memory portion.

Description

Background Technology

[0001] GPU devices are optimized for computational throughput rather than response time (i.e., latency). Traditional design choices in GPU architecture and associated software prioritize arithmetic operations with high sustained throughput. Specifically, GPUs, with their relatively small and simple caches and software-managed scratch memory, maximize the chip space of a large number of arithmetic logic units (ALUs) for parallel processing of massive datasets. However, the current batch-synchronous programming model of GPUs limits their applicability to flexible applications that require a certain level of task scheduling. Known GPU scheduling policies are defined in hardware and do not allow for manual algorithm placement on GPU multiprocessors. For example, running applications requiring multiprocessor synchronization (e.g., producer-consumer patterns) on the same multiprocessor can lead to deadlocks. In traditional systems, it is also difficult to provide an environment where similar tasks can be co-scheduled on the same multiprocessor to benefit from locality, such as better utilizing local GPU caches.

[0002] The Compute Unified Device Architecture (CUDA) framework from NVIDIA provides the ability to launch cooperative thread meshes on GPUs, which can be synchronized via global barriers on the GPU itself (without needing to synchronize with barriers on the CPU). This feature allows users to schedule kernel pipelines (i.e., GPU programs) to the GPU. However, the off-the-shelf GPU scheduler still schedules thread blocks across the entire GPU device in an undefined manner, preventing the manual placement of thread blocks on GPU multiprocessors. This limitation may result in underutilization of GPU resources. Summary of the Invention

[0003] In one embodiment, a method for use in a computing system having a central processing unit (CPU) and a graphics processing unit (GPU) is provided, the method comprising: assigning a first memory portion and a second memory portion to: (i) a worker thread of a work-stealing scheduler executing on the CPU and (ii) an execution unit that is a portion of the GPU; retrieving a task from a queue associated with the worker thread; having the worker thread detect whether a cutoff condition of the task is met; and, if the cutoff condition is not met, dividing the task into two or more additional tasks and adding the two or more additional tasks to the queue.

[0004] In one embodiment, the first memory portion includes a doorbell register, and the second memory portion includes a task register.

[0005] In one embodiment, the execution unit includes multiple GPU cores and a synchronization mechanism shared among the multiple GPU cores.

[0006] In one embodiment, the first data includes one or more of the opcode corresponding to the task and arguments for performing the task.

[0007] In one embodiment, the method further includes: detecting that the task has been completed when the execution unit stores the second value in the first memory portion, the detection being performed by the worker thread.

[0008] In one embodiment, the method further includes: retrieving from the second memory portion second data generated by the execution unit as a result of performing the task, the second data being retrieved by the worker thread in response to detecting that a second value has been stored in the first memory portion by the execution unit.

[0009] In one embodiment, the method further includes: retrieving the first data from the second memory portion, the first data being retrieved by the execution unit in response to detecting that the first value has been stored in the first memory portion by the worker thread; and executing the task, the task being executed by the execution unit based on the first data.

[0010] In one embodiment, the method further includes: storing second data generated by the execution unit as a result of performing a task in the second memory portion; issuing a memory fence release instruction after the second data is stored in the second memory portion; and storing a second value indicating that the task is completed in the first memory portion, the second value being stored by the execution unit after the memory fence release instruction is issued.

[0011] In one embodiment, a system is provided, comprising: a graphics processing unit (GPU) having an execution unit; a random access memory having a first memory portion and a second memory portion; and a central processing unit (CPU) configured to execute a work-stealing scheduler having worker threads configured to perform the following operations: retrieving a task from a queue associated with the worker thread; detecting whether a deadline condition for the task is met; and, when the deadline condition is not met, dividing the task into two or more additional tasks and adding the two or more additional tasks to the queue.

[0012] In one embodiment, the first memory portion includes a doorbell register, and the second memory portion includes a task register.

[0013] In one embodiment, the execution unit includes multiple GPU cores and a synchronization mechanism shared among the multiple GPU cores.

[0014] In one embodiment, the first data includes one or more of an opcode corresponding to the task and actual parameters for performing the task.

[0015] In one embodiment, the worker thread is further configured to perform an operation to detect the completion of the task while the execution unit stores the second value in the first memory portion.

[0016] In one embodiment, the worker thread is further configured to perform an operation to retrieve second data generated as a result of the task execution from the second memory portion, the second data being retrieved in response to the detection that the execution unit has stored the second value in the first memory portion.

[0017] In one embodiment, the execution unit is configured to perform the following operations: retrieve the first data from the second memory portion, the first data being retrieved in response to a worker thread detecting that the first value has been stored in the first memory portion; and perform the task based on the first data.

[0018] In one embodiment, the execution unit is further configured to perform the following operations: storing second data generated by the execution unit as a result of performing a task in the second memory portion; issuing a memory fence release command after the second data is stored in the second memory portion; and storing a second value indicating that the task is completed in the first memory portion, the second value being stored after the memory fence release command is issued.

[0019] According to an aspect of this disclosure, a non-transitory computer-readable medium is provided, configured to store one or more processor-executable instructions that, when processed by at least one processor, cause the processor to execute a work-stealing scheduler with worker threads arranged to share a first memory portion and a second memory portion with an execution unit that is part of a graphics processing unit (GPU), the worker threads being arranged to perform the following operations: retrieving a task from a queue associated with the worker thread; and detecting whether a deadline condition for the task is met.

[0020] In one embodiment, the first memory portion includes a doorbell register, and the second memory portion includes a task register.

[0021] In one embodiment, the execution unit includes multiple GPU cores and a synchronization mechanism shared among the multiple GPU cores.

[0022] In one embodiment, the first data includes one or more of an opcode corresponding to the task and actual parameters for performing the task. Attached Figure Description

[0023] The above features can be more fully understood from the description of the following figures, in which:

[0024] Figure 1 This is a diagram illustrating an example of a computing system according to aspects of this disclosure;

[0025] Figure 2A This is a diagram illustrating an example of stealing a scheduler based on aspects of this disclosure;

[0026] Figure 2B It is based on the aspects of this disclosure. Figure 2A A diagram illustrating an example of a queue used by a job-stealing scheduler;

[0027] Figure 3A This is a diagram illustrating an example of a graphics processing unit (GPU) according to aspects of this disclosure;

[0028] Figure 3B Based on the aspects of this disclosure Figure 3A A diagram showing an example of an execution unit that is part of a GPU;

[0029] Figure 4A Based on the description of this disclosure Figure 2A A diagram showing the operation of stealing the scheduler's work;

[0030] Figure 4B This is a diagram illustrating an example of the shared memory portion of an aspect of this disclosure;

[0031] Figure 5A A flowchart illustrating an example of a process according to an aspect of this disclosure;

[0032] Figure 5B A flowchart illustrating an example of a process according to an aspect of this disclosure;

[0033] Figure 6A A flowchart illustrating an example of a process according to an aspect of this disclosure;

[0034] Figure 6B A flowchart illustrating an example of a process according to an aspect of this disclosure; and

[0035] Figure 6C This is a flowchart illustrating an example of a process according to aspects of this disclosure. Detailed Implementation

[0036] Work-stealing is an efficient dynamic scheduling strategy widely used on central processing units (CPUs). A work-stealing scheduler performs load balancing on parallel processors by distributing tasks across a set of deques (or "deques"), each associated with a worker thread running on a processing core. Each deque can support push and pop operations, allowing the local thread associated with that deque to schedule and unschedule work (i.e., tasks) in the same order as serial execution (i.e., stack order). Each deque can also support a stealing operation used when a non-local thread (also known as a thief thread) has exhausted its tasks on its local deque. The stealing operation may remove a task from the opposite end of the victim's deque (relative to the local thread) for the thief thread to execute. When a task is being executed, its execution may frequently generate additional tasks, which are then placed on the local deque.

[0037] Work-stealing schedulers generate and process tasks that form a directed acyclic graph (DAG) at runtime. They execute recursive algorithms represented as divide-and-conquer, where one or more finer-grained subtasks are created (i.e., forked) for each parent task. These algorithms also indicate where threads must wait (i.e., join) forked subtasks to complete execution. In the basic case of recursion (called sequential cutoff), the task to be partitioned is too small to be profitablely divided into subtasks, at which point a sequential algorithm (i.e., leaf tasks) is executed. Profitability is a measure of how much overhead is required to partition and schedule subtasks. Notably, all algorithmic work is performed within leaf tasks.

[0038] According to this disclosure, an improved work-stealing scheduler is disclosed, wherein the CPU executes a portion of the work-stealing scheduler that recursively partitions tasks until a sequential deadline is reached, wherein the scheduler overhead governs the cost of actually executing the task. Instead of executing sequential code on the CPU, the improved work-stealing scheduler offloads tasks to be executed on an available graphics processing unit (GPU). For example, in some embodiments, this disclosure can provide enhancements to existing work-stealing schedulers (which utilize only the CPU) by presenting a heterogeneous, cooperative work-stealing scheduler design that supports GPUs and maintains the execution semantics of existing general-purpose work-stealing schedulers.

[0039] Figure 1This is a diagram illustrating an example of a computing system 100 according to aspects of this disclosure. The computing system 100 may include a central processing unit (CPU) 109, a graphics processing unit (GPU) 120, memory 130, and shared memory 140. The CPU 109 may include any suitable type of processor, such as an x86 processor, a Reduced Instruction Set Computer (RISC) processor, an ARM-based processor, etc. According to this example, the CPU 109 includes multiple CPU cores 110A-110D, which are arranged to execute different worker threads 220 of a work-stealing scheduler 200 in parallel (as shown in Figure 2). Although in Figure 1 In the example, CPU 109 has four CPU cores, but alternative implementations in which the CPU has any number of cores (e.g., 1, 8, etc.) are possible.

[0040] GPU 120 may include any suitable type of graphics processing unit with the ability to directly map memory ranges from the address space of CPU 109 to the address space of GPU 120, thereby sharing a single virtual address space. In some embodiments, GPU 120 may include a GPU supporting the Computing Unified Device Architecture (CUDA) framework. Additionally or alternatively, in some embodiments, GPU 120 may include a GPU supporting the Open Computing Language (OpenCL) framework. GPU 120 may include multiple GPU cores. According to this example, GPU 120 includes 1024 GPU cores; however, it should be understood that this disclosure is not limited to providing any particular number of GPU cores 132 on GPU 120.

[0041] Memory 130 may include any suitable type of random access memory (RAM), such as Double Data Rate Synchronous Dynamic Random Access Memory (DDR SDRAM). Shared memory 140 may include a portion of memory 130 that is mapped to the memory space of GPU 120. Because shared memory 140 is mapped to the memory space of GPU 120, each of CPU 109 and GPU 120 can directly access shared memory 140. For example, in some embodiments, shared memory 140 may be mapped to the memory space of GPU 120 using the CUDA framework or the OpenCL framework. In some embodiments, GPU 120 may be arranged to accept a pointer to shared memory 140 and use that pointer to directly reference shared memory 140 from within the GPU core. Similarly, at least some CPU cores 110 may also be arranged to use pointers to directly reference shared memory 140.

[0042] Figure 2AThis is a diagram illustrating an example of a work-stealing scheduler 200 executing on CPU 109. The work-stealing scheduler 200 can be a part of any suitable type of parallel computing software and / or microcode. The work-stealing scheduler 200 can include multiple queues 210 and multiple worker threads 220. Each of the worker threads 220 can be associated with a different queue 210. According to this example, worker thread 220A is associated with queue 210A; worker thread 220B is associated with queue 210B; worker thread 220C is associated with queue 210C; and worker thread 220D is associated with queue 210D. See below for more information. Figure 2B Further discussion reveals that thread 220 uses queue 210 to store and retrieve tasks. According to... Figure 2A For example, each queue 210 is a double-ended queue. However, alternative implementations are possible, in which another type of queue and / or another type of data structure is used to store tasks. Although the work-stealing scheduler 200 is implemented in software in this example, alternative implementations in which the work-stealing scheduler 200 is implemented in hardware and / or as a combination of software (or microcode) or hardware are also possible.

[0043] Figure 2B This is a diagram of queue 210 according to an aspect of this disclosure. As indicated by the designations, queue 210 may be the same as or similar to any of queues 210A-B that are part of the work-stealing scheduler 200. As shown, queue 210 may be arranged to store one or more tasks. Each task may represent a specific operation that is expected to be performed. The tasks in queue 210 together may be part of a directed acyclic graph (DAG) for specific actions to be performed by a parallel computing algorithm implemented by the work-stealing scheduler 200. As used throughout the disclosure and depending on the context, the term “task” may refer to an operation or a descriptor corresponding to an operation. For example, in some embodiments, the term “task” may refer to any suitable type of data necessary for performing the operation. For example, a “task” may include one or more of the following: an opcode corresponding to an operation, input parameters for performing the operation, arguments of a function call associated with the operation, pointers to other data that need to be referenced in order to perform the operation, and / or any other suitable type of data required to perform the operation.

[0044] Figure 3A This is a diagram illustrating an example of the logical organization of a GPU 120 according to aspects of this disclosure. According to this example, the GPU 300 is arranged to execute execution units (or tiles) 310A, execution units (or tiles) 310B, execution units 310C (or tiles), and execution units (or tiles) 310D. Although in Figure 3AIn the example, GPU 120 is arranged to implement four execution units, but it should be understood that this disclosure is not limited to implementing any particular number of execution units on GPU 120. In some embodiments, any execution unit 310 may include something referred to as a “block” or “wrap” in the NVIDIA CUDA™ framework. In some embodiments, each execution unit 310 may include a hardware barrier mechanism for the GPU cores within the execution unit to serve as a synchronization point. Examples of execution units that may be used according to the principles and ideas of this disclosure can be found in U.S. Patent Application 16 / 372,618 entitled “ACCELERATING DATAFLOW SIGNAL PROCESSING APPLICATIONS ACROSS HETEROGENEOUS CPU / GPU SYSTEMS,” the entire contents of which are incorporated herein by reference.

[0045] Figure 3B This is a diagram illustrating an example of an execution unit 310 according to an aspect of this disclosure. As indicated by the reference numerals, the execution unit 310 can be used in conjunction with the above-mentioned... Figure 3A Any execution units 310A-D discussed are the same or similar. Figure 3B As shown, execution unit 310 may include a group of GPU cores 132 and a synchronization barrier 320 shared among all GPU cores 132 in the group. According to this example, execution unit 310 includes 32 GPU cores 132, but alternative implementations in which execution unit 310 includes any number of GPU cores 132 (e.g., 1, 4, 8, 16, 64, etc.) are possible. Although in Figure 3B In the example, execution unit 310 includes a synchronization barrier, but alternative implementations using a different type of synchronization mechanism are possible. In short, this disclosure is not limited to any particular type of synchronization mechanism in execution unit 310.

[0046] Execution unit 310 can be used to execute tasks on behalf of worker threads 220 in work-stealing scheduler 200. One of the GPU cores 132 in execution unit 310 can be assigned a primary state and is referred to herein as the "primary GPU core of execution unit 310". For example, in some implementations, the GPU core with rank "0" (in execution unit 310) can be designated as the primary GPU core of execution unit 310. All remaining GPU cores 132 in execution unit 310 can be assigned a secondary state and are referred to herein as the "secondary GPU cores of execution unit 310". The primary GPU core 132 of execution unit 310 can be arranged to: (i) obtain a task from a given worker thread in worker thread 220, (ii) divide the task into task parts, (iii) execute one of the task parts and provide the remaining task parts to the secondary GPU core, and (iv) notify the worker thread when the task is completed.

[0047] In a specific example, synchronization barrier 320 may be implemented in hardware, and it may be arranged to act as a shared synchronization point among GPU cores 132 in execution unit 310. More specifically, synchronization barrier 320 may cause each GPU core 132 (e.g., each of the primary GPU core 132 and / or secondary GPU cores 320) to enter a waiting state after it has completed executing its corresponding task portion and reached barrier 320. Entering a waiting state can effectively prevent GPU core 132 from immediately starting to execute the next part of the task, and may overwrite the task register storing the completion data of the current task before the completion data of the current task is retrieved by one of the worker threads 220 (e.g., see...). Figure 4B (Task register 430 shown). Although the synchronization barrier 320 is implemented in hardware in this example, alternative implementations of the synchronization barrier 320, such as software or a combination of hardware and software, are possible. In short, this disclosure is not limited to any specific implementation of the synchronization barrier 320.

[0048] Figure 4A This is a schematic diagram illustrating the operation of a computing system 100 according to aspects of this disclosure. According to various aspects of this disclosure, shared memory is divided into M portions 410, where M is a positive integer equal to the number of execution units 310 allocated to the work-stealing scheduler 200 (i.e., four in this example). The shared memory portions 410 can be used to exchange data between corresponding worker threads 220 and corresponding execution units 310. Figure 4BAs shown, each part 410 includes a doorbell register 420 and a task register 430. The doorbell register 420 may be one word wide and may be used to signal when a task has been stored in the task register 430. The task register 430 may be configured to store a specific task. As described above, the term "task" as used throughout this disclosure may refer to any data necessary for the execution unit to perform an operation. For example, a "task" stored in the task register 230 may include one or more of the following: an opcode associated with a specific operation, input parameters for performing the operation, actual arguments of a function call associated with the operation, and pointers to other data that needs to be referenced for performing the operation.

[0049] return Figure 4A Each execution unit 310 can be allocated a corresponding shared memory portion 410. More specifically, execution unit 310A can be allocated shared memory portion 410A, which it uses to send and receive data from CPU core 110A (and / or worker thread 220A). Execution unit 310B can be allocated shared memory portion 410B, which it uses to send and receive data from CPU core 110B (and / or worker thread 220B). Execution unit 310C can be allocated shared memory portion 410C, which it uses to send and receive data from CPU core 110C (and / or worker thread 220C). Execution unit 310D can be allocated shared memory portion 410D, which it uses to send and receive data from CPU core 110D (and / or worker thread 220D).

[0050] Furthermore, each CPU core 110 (and / or each worker thread 220) may be allocated a corresponding shared memory portion 410. More specifically, CPU core 110A (and / or worker thread 220A) may be allocated shared memory portion 410A, which is used to send or receive data from execution unit 310A. CPU core 110B (and / or worker thread 220B) may be allocated shared memory portion 410B, which is used to send or receive data from execution unit 310B. CPU core 110C (and / or worker thread 220C) may be allocated shared memory portion 410C, which is used to send or receive data from execution unit 310C. Furthermore, CPU core 110D (and / or worker thread 220D) can be allocated a shared memory portion 410D, which CPU core 110D (and / or worker thread 220D) uses to send or receive data from execution unit 310D.

[0051] Each CPU core 110 (and / or each worker thread 220) can be mapped to a corresponding execution unit 310. According to the example in Figure 410A, CPU core 110A (and / or worker thread 220A) is mapped to execution unit 310A; CPU core 110B (and / or worker thread 220B) is mapped to execution unit 310B; CPU core 110C (and / or worker thread 220C) is mapped to execution unit 310C; and CPU core 110D (and / or worker thread 220A) is mapped to execution unit 310D. The mapping between any CPU core 110 and its corresponding execution unit can be achieved by allocating the same shared memory portion 410 to both the CPU core and its corresponding execution unit 310. However, it should be understood that this disclosure is not limited to any particular method for mapping CPU cores to their corresponding execution units.

[0052] In some implementations, each CPU core 110 (and / or worker thread 220) can execute tasks on execution units 310 mapped to that CPU core 110 and / or worker thread 220. For example, CPU core 110A (and / or worker thread 220A) can execute tasks on execution unit 310A; CPU core 110B (and / or worker thread 220B) can execute tasks on execution unit 310B; CPU core 110C (and / or worker thread 220C) can execute tasks on execution unit 310C; and CPU core 110D (and / or worker thread 220D) can execute tasks on execution unit 310D. For example, when any given worker thread 220 detects that a sequence deadline for a task is met, the given worker thread 220 can identify the corresponding execution unit 310 mapped to that given worker thread 220. Next, the given worker thread 220 can provide a task to the corresponding execution unit 310 by using the shared memory portion 410 allocated to both the given worker thread 220 and the corresponding execution unit 310. Next, the corresponding execution unit 310 retrieves the task from the shared memory portion 410. Next, the corresponding execution unit 310 can execute the task and put the data generated as a result of the task execution back into the shared memory portion 410. Finally, the given worker thread can retrieve the data generated as a result of the task execution from the shared memory portion.

[0053] Although Figure 4A In the example, each CPU core 110 (and / or worker thread 220) is allocated a single shared memory portion 410, but alternative implementations in which any CPU core 110 is allocated multiple shared memory portions 410 are possible. Although in Figure 4B In the example, each CPU core 110 (and / or worker thread 220) is mapped to a single execution unit 310, but alternative implementations are possible where any CPU core 110 (and / or worker thread 220) is mapped to multiple execution units. In such an implementation, each CPU core 110 (and / or worker thread) may be mapped to the same number of execution units 310. If the number of execution units 310 is not divisible by the number of CPU cores 110 (and / or worker threads 220), then one of the CPU cores 110 (and / or worker threads 220) may be mapped to fewer execution units 310.

[0054] Figure 5AThis is a flowchart illustrating an example of a process 500A for initiating operations of the scheduler 200 according to aspects of this disclosure. According to this example, process 500A is executed by CPU core 110A. However, alternative implementations in which process 500A is executed by another CPU core 110 are possible.

[0055] In step 502, CPU core 110A instantiates queue 210 and worker thread 220 of the work-stealing scheduler 200. In step 504, CPU core 110A allocates shared memory 140 and multiple shared memory portions 410 within shared memory 140. In step 506, CPU core 110A assigns each worker thread 220A to a different CPU core 110 of CPU 109. As described above, as a result of executing step 506, worker thread 220A is assigned to CPU core 110A, worker thread 220B is assigned to CPU core 110B, worker thread 220C is assigned to CPU core 110C, and worker thread 220D is assigned to CPU core 110D. In step 508, core 110A maps each worker thread 220 to a corresponding execution unit 310. As described above, as a result of executing step 508, worker thread 220A (and / or CPU core 110A) is mapped to execution unit 310A. Worker thread 220B (and / or CPU core 110B) is mapped to execution unit 310B. Worker thread 220C (and / or CPU core 110C) is mapped to execution unit 310C. And worker thread 220D (and / or CPU core 110D) is mapped to execution unit 310D. In some embodiments, the mapping between any worker thread 220 and any execution unit 310 can be performed by assigning the same shared memory portion 410 to worker thread 220 and execution unit 310. In step 510, CPU core 110A begins executing worker thread 220. In some embodiments, any worker thread 220 can be executed according to process 500B, which will be discussed below. Figure 5B Further discussion.

[0056] Figure 5B This is a flowchart illustrating an example of a process 500B for a worker thread of a job-stealing scheduler, as specified in step 510 of process 500A, according to aspects of this disclosure. Figure 5BFor example, process 500B is executed by worker thread 220A; however, alternatively, it will be understood that process 500B can be executed by any worker thread 220. In step 522, worker thread 220A retrieves (and / or removes) a task from queue 210A. In step 524, worker thread 220A determines whether a predetermined deadline condition is met. If the condition is met, process 500B proceeds to step 526. Otherwise, if the condition is not met, process 500B proceeds to step 528. In step 526, process 500B divides the task into additional tasks and adds the task back to queue 210A. In step 528, worker thread 220A executes the task using execution unit 310A of GPU 120. The manner of task execution will be explained below regarding... Figures 6A-6C Further discussion. Specifically, Figure 6A An example of a process is provided, which is executed by worker thread 220A for the purpose of performing a task on execution unit 310A. Figure 6B An example of process 600B, executed by the main GPU core of execution unit 310A for the purpose of coordinating the execution of different parts of a task, is provided. Figure 6C An example of process 600C is provided, which is executed by either the primary GPU core or the secondary GPU core in execution unit 310A for the purpose of performing a task portion.

[0057] Figure 6A This is a flowchart of process 600A for executing the task specified in step 528 of process 500B using a GPU. In step 602, worker thread 220A stores the task (e.g., the task retrieved in step 522) in task register 430 of shared memory portion 410A. In step 604, worker thread 220A issues a memory fence acquisition instruction. In step 606, worker thread 220A stores a first value (e.g., '1') in doorbell register 420 of shared memory portion 410A. As follows regarding... Figure 6B Further discussion involves storing the first value in the doorbell register 420 to effectively notify the execution unit 310A that a task awaiting execution exists in the task register 430. In step 608, worker thread 220A checks whether the doorbell register 420 of the shared memory portion 410A is set to a second value (e.g., '0'). If the doorbell register 420 is set to the second value, process 600A proceeds to step 610. Otherwise, if the doorbell register 420 is not set to the second value, step 608 is executed again. In other words, in step 606, worker thread 220A waits until the doorbell register 420 is set to the second value (by execution unit 310A) before continuing to step 610. As discussed below... Figure 6BFurther discussion reveals that when the doorbell register 420 is set to the second value, this effectively notifies worker thread 220A that the task has been completed by execution unit 310A. In step 610, worker thread 220A retrieves the completion status word (and / or error code) that has been stored there by one or more GPU cores in execution unit 310A from the task register 430 of the shared memory portion 410A.

[0058] Figure 6BThis is a flowchart of an example of process 600B for executing the task specified in step 528 of process 500B. In step 612, the primary GPU core of execution unit 310A checks whether the doorbell register 420 of shared memory portion 410A has been set to a first value. If the doorbell register 420 is set to the first value, process 600B proceeds to step 614. Otherwise, step 612 is repeated. In step 614, the primary GPU core of execution unit 310A retrieves the task from the task register 430 of shared memory portion 410A. In step 616, the primary GPU core of execution unit 310A divides the task into multiple task parts. In step 618, the primary GPU core of execution unit 310A assigns each of the multiple task parts to a different GPU core in execution unit 310A. For example, the primary GPU core can assign a corresponding task part to itself and each secondary GPU core in execution unit 310A. Alternatively, when the number of task portions is less than the number of GPU cores in execution unit 310A, the primary GPU core 310A may assign the corresponding task portion only to some GPU cores in execution unit 310A. In step 620, the primary GPU core 132 of execution unit 310A causes each secondary GPU core 132 (which has been assigned a task portion) in execution unit 310A to begin executing its corresponding task portion. In step 622, the primary GPU core 132 executes the task portion it has been assigned to itself. As will be readily appreciated, in some embodiments, the primary GPU core 132 may not assign a task portion to itself, in which case step 622 may be omitted. In step 622, the primary GPU core of execution unit 310A detects whether each GPU core (which has been assigned a task portion) in execution unit 310A has completed executing its corresponding task portion. For example, detecting whether each GPU core has completed executing its corresponding task portion may include detecting whether each GPU core has reached the synchronization barrier 320 of execution unit 310A and is currently in a waiting state. If each GPU core has completed executing its corresponding task portion, process 600B proceeds to step 626. Otherwise, step 624 is repeated. In step 626, the primary GPU core of execution unit 310A issues a memory fence release instruction. In step 628, the primary GPU core of execution unit 310A sets the doorbell register 420 of shared memory portion 410A to a second value (e.g., '0').

[0059] Figure 6CThis is a flowchart of an example of process 600B for performing the task specified in step 506 of process 500B. In step 642, GPU core 132 of execution unit 310A receives one of the task portions (assigned in step 618). In step 644, GPU core 132 executes the task portion. In step 646, GPU core 132 stores a completion indicator in the task register of shared memory portion 410A. For example, if GPU core 132 has successfully executed the task, GPU core 132 can set one of the bits in the task register of memory portion 410A to "1". As another example, if GPU core 132 fails to execute the task successfully, GPU core 132 can set the same bit to "0". Alternatively, in some implementations, when GPU core 132 fails to execute the task successfully, GPU core 132 can store an error code in the task register that identifies the reason why the task portion could not be completed successfully. Although the completion indicator is 1 bit wide in this example, alternative implementations in which the completion status indicator has a larger size are also possible. In task 648, GPU core 132 enters a waiting state as a result of the barrier synchronization barrier 320 reaching execution unit 310A.

[0060] In some implementations, when a task is completed, a word can be stored in the task register 430 (of shared memory portion 410A) that identifies the completion status of the task. This word may include multiple bits. Each bit can be stored in the task register by a different GPU core in execution unit 310, and it can indicate whether the GPU core has successfully completed a portion of the task that it has executed. If the bit is equal to "1", it may indicate that the GPU core has successfully completed its portion of the task. Otherwise, if the bit is set to "0", it may indicate that the GPU core failed to complete its portion of the task. Therefore, when the word contains all "1", it may indicate that the task has been successfully completed by execution unit 310. In contrast, when the word contains at least one "0", it may indicate that the task could not be successfully completed by execution unit 310. Although... Figures 1-6C The shared memory portion 410A is shown as the only memory shared between the execution unit 310A and the worker thread 220A, but alternative implementations in which additional memory is shared between the execution unit 310A and the worker thread 220A for providing result data to the worker thread 220A are possible.

[0061] According to aspects of this disclosure, memory fence acquire and memory fence release instructions can be barrier instructions that cause the CPU 109 (and / or a compiler running on the CPU 109) to enforce the order in which data is stored in the doorbell register 420 and task register 430 of shared memory 410. Specifically, issuing the acquire memory fence instruction at step 604 ensures that the task will be stored in the task register 430 of shared memory portion 410A before the first value is stored in the doorbell register 420 of shared memory portion 410A (step 606) (step 602). Similarly, issuing the memory fence release instruction at step 626 ensures that all completion data associated with the task will be copied to the task register 430 of shared memory portion 410A before the doorbell register 420 of shared memory portion 410A is set to the second value (step 632) (step 624). According to this disclosure, it has been observed that using memory fence instructions (as opposed to atomic operations) to synchronously write to the doorbell and task registers (of shared memory portion 410) can significantly reduce protocol overhead (see reference). Figures 6A-6C (Description) and improve the efficiency of task execution (obtained in step 614).

[0062] In some implementations, when a memory fence instruction has been issued, seeing a first value (e.g., '1') in the doorbell register 420 of shared memory portion 410A can assure execution unit 310A that a valid task exists in the task register 430 of memory portion 410. Similarly, when a memory fence release instruction is issued, seeing a second value (e.g., '0') in the doorbell register 420 of shared memory portion 410A can assure CPU core 110A (and / or worker thread 220A) that a valid task exists in the task register 430 of memory portion 410. In some implementations, when CPU core 110A (and / or worker thread 220A) is mapped to multiple execution units 310, CPU core 110A (and / or worker thread 220A) can divide the task (obtained in step 522) into multiple tasks (e.g., task portions) and execute each task portion (with respect to a different execution unit 310). Figure 6A (Discussion methods).

[0063] according to Figure 5B and Figure 6AFor example, in step 522, a task is retrieved from queue 210A, and in step 602, the task is stored in the task register of shared memory portion 410A. However, steps 522 and 602 do not necessarily mean that the same data retrieved from queue 210A is also stored in task register 430 of shared memory portion 410A. For example, in some embodiments, one or more first data items may be retrieved from queue 210A (in step 522), and one or more second data items may be stored in task register 430 (in step 602). In such embodiments, the second data items may be the same as or different from the first data items. Additionally or alternatively, in some embodiments, at least one of the second data items may be generated at least partially based on the first data items.

[0064] In some implementations, the work-stealing scheduler 200 can disconnect one or more execution units 310 (by placing a "disconnect" opcode in the task register 430 of the execution unit 310). Upon receiving the "disconnect" opcode, the execution unit's main GPU 132 (and / or other hardware in the GPU 120) can detach the execution unit from the work-stealing scheduler 200.

[0065] This disclosure is not limited to the use of any particular parallel processing algorithm in the work-stealing scheduler 200. In this respect, it will be understood that this disclosure is not limited to evaluating any particular cutoff condition (in step 524). Although Figures 1-6C The examples are presented in the context of job-stealing schedulers, but this disclosure is not limited to any particular type of scheduler.

[0066] The process described here is not limited to... Figure 1 The hardware and software described herein are used together; they may be suitable for any computing or processing environment and any type of machine or group of machines capable of running computer programs. The processes described herein can be implemented in hardware, software, or a combination of both. The processes described herein can be implemented in a computer program that executes on a programmable computer / machine, each programmable computer / machine including a processor, a non-transitory machine-readable medium or another article of manufacture readable by the processor (including volatile and non-volatile memory and / or storage elements), at least one input device, and one or more output devices. Program code can be applied to data entered using the input devices to perform any of the processes described herein and generate output information.

[0067] The system can be implemented at least in part as a computer program product (e.g., in a non-transitory machine-readable storage medium, such as a non-transitory computer-readable medium) for execution by or for controlling the operation of data processing devices (e.g., programmable processors, computers, or multiple computers). Each such program can be implemented in a high-level procedural or object-oriented programming language to work with the rest of the computer-based system. However, the program can be implemented in assembly, machine language, or hardware description language. The language can be a compiled or interpreted language and can be deployed in any form, including as a standalone program or as a module, component, subroutine, or other unit suitable for use in a computing environment. The computer program can be deployed to execute on one or more computers at a single site, or distributed across multiple sites and interconnected via a communication network. The computer program can be stored on a non-transitory machine-readable medium readable by a general-purpose or special-purpose programmable computer for configuring and operating the computer when the computer reads the non-transitory machine-readable medium to execute the process described herein. For example, the process described herein can also be implemented as a non-transitory machine-readable storage medium configured with a computer program, wherein, upon execution, the instructions in the computer program cause the computer to operate according to the process. Non-transitory machine-readable media may include, but are not limited to, hard disk drives, optical disks, flash memory, non-volatile memory, volatile memory, and magnetic disks, but do not include transient signals themselves.

[0068] Preferred embodiments of various concepts, structures, and techniques used to illustrate the subject matter of this patent have been described, and it is now apparent that other embodiments combining these concepts, structures, and techniques may be used. Therefore, it is considered that the scope of the patent should not be limited to the described embodiments, but should be limited only by the spirit and scope of the appended claims.

Claims

1. A method for use in a computing system having a central processing unit (CPU) and a graphics processing unit (GPU), the method comprising: The first memory portion and the second memory portion are assigned to: worker threads of the work-stealing scheduler executed on the CPU, and execution units that are part of the GPU; Retrieve tasks from the queue associated with the worker thread; The worker thread checks whether the deadline conditions for the task are met. If the deadline condition is not met, the task is divided into two or more additional tasks, and the two or more additional tasks are added to the queue. When the deadline condition is met, the first data corresponding to the task is stored in the second memory portion, and the first data is stored in the second memory portion by the worker thread; After the first data is stored in the second memory portion, a memory fence acquisition command is issued; as well as The first value is stored in the first memory portion, and the first value is stored by the worker thread after the memory fence acquisition instruction is issued. Specifically, storing the first value in the first memory portion enables the execution unit to retrieve the first data from the second memory portion and perform the task based on the first data.

2. The method according to claim 1, wherein, The first memory portion includes a doorbell register, and the second memory portion includes a task register.

3. The method according to claim 1, wherein, The execution unit includes multiple GPU cores and a synchronization mechanism shared among the multiple GPU cores.

4. The method according to claim 1, wherein, The first data includes one or more of the following: the opcode corresponding to the task and the actual parameters for performing the task.

5. The method according to claim 1, further comprising: When the second value is stored in the first memory portion by the execution unit, the task is detected as completed, and the detection is performed by the worker thread.

6. The method according to claim 1, further comprising: The worker thread retrieves second data generated by the execution unit as a result of performing the task from the second memory portion. The second data is retrieved by the worker thread in response to the detection that a second value has been stored in the first memory portion by the execution unit.

7. The method according to claim 1, further comprising: The first data is retrieved from the second memory portion, the first data being retrieved by the execution unit in response to the detection that the first value has been stored in the first memory portion by the worker thread; as well as The task is performed by the execution unit based on the first data.

8. The method according to claim 7, further comprising: The second data generated by the execution unit as a result of performing the task is stored in the second memory portion; After the second data is stored in the second memory portion, a memory fence release command is issued; as well as A second value indicating the completion of the task is stored in the first memory portion, and the second value is stored by the execution unit after the memory fence release command is issued.

9. A computing system, comprising: A graphics processing unit (GPU) with execution units; A random access memory having a first memory portion and a second memory portion; A central processing unit (CPU) is configured to execute a work-stealing scheduler, the work-stealing scheduler having worker threads configured to perform the following operations: Retrieve tasks from the queue associated with the worker thread; Check whether the deadline conditions for the task are met; If the deadline condition is not met, the task is divided into two or more additional tasks, and the two or more additional tasks are added to the queue. When the cutoff condition is met, the first data corresponding to the task is stored in the second memory portion; After the first data is stored in the second memory portion, a memory fence acquisition command is issued; as well as After the memory fence acquire instruction is issued, the first value is stored in the first memory portion. Storing the first value in the first memory portion enables the execution unit to retrieve the first data from the second memory portion and perform the task based on the first data.

10. The computing system according to claim 9, wherein, The first memory portion includes a doorbell register, and the second memory portion includes a task register.

11. The computing system according to claim 9, wherein, The execution unit includes multiple GPU cores and a synchronization mechanism shared among the multiple GPU cores.

12. The computing system according to claim 9, wherein, The first data includes one or more of the following: the opcode corresponding to the task and the actual parameters for performing the task.

13. The computing system according to claim 9, wherein, The worker thread is also configured to perform an operation to detect the completion of the task when the second value is stored in the first memory portion by the execution unit.

14. The computing system according to claim 9, wherein, The worker thread is also configured to perform an operation to retrieve second data generated as a result of performing the task from the second memory portion, the second data being retrieved in response to the detection that a second value has been stored by the execution unit in the first memory portion.

15. The computing system according to claim 9, wherein, The execution unit is configured to perform the following operations: The first data is retrieved from the second memory portion, the first data being retrieved in response to the worker thread detecting that the first value has been stored in the first memory portion; as well as The task is performed based on the first data.

16. The computing system according to claim 15, wherein, The execution unit is also configured to perform the following operations: The second data generated by the execution unit as a result of performing the task is stored in the second memory portion; After the second data is stored in the second memory portion, a memory fence release command is issued; as well as A second value indicating the completion of the task is stored in the first memory portion, and the second value is stored after the memory fence release command is issued.

17. A non-transitory computer-readable medium configured to store one or more processor-executable instructions, which, when processed by at least one processor, cause the processor to execute a work-stealing scheduler with worker threads arranged to share a first memory portion and a second memory portion as a portion of execution units that are graphics processing units (GPUs), the worker threads being arranged to perform the following operations: Retrieve tasks from the queue associated with the worker thread; Check whether the deadline conditions for the task are met; If the deadline condition is not met, the task is divided into two or more additional tasks, and the two or more additional tasks are added to the queue. When the cutoff condition is met, the first data corresponding to the task is stored in the second memory portion; After the first data is stored in the second memory portion, a memory fence acquisition command is issued; After the memory fence acquire instruction is issued, the first value is stored in the first memory portion. Storing the first value in the first memory portion enables the execution unit to retrieve the first data from the second memory portion and perform the task based on the first data.

18. The non-transitory computer-readable medium according to claim 17, wherein, The first memory portion includes a doorbell register, and the second memory portion includes a task register.

19. The non-transitory computer-readable medium according to claim 17, wherein, The execution unit includes multiple GPU cores and a synchronization mechanism shared among the multiple GPU cores.

20. The non-transitory computer-readable medium according to claim 17, wherein, The first data includes one or more of the following: the opcode corresponding to the task and the actual parameters for performing the task.

Images