Data dependency aware scheduling

Abstract

To improve processing efficiency, the processing system (100) flexibly schedules workgroups among kernels based on data dependencies between the workgroups. The workgroups are divided into subsets based on data dependencies, and a workgroup of a first subset that generates data is scheduled to execute immediately before a workgroup of a second subset that consumes data generated by the first subset. Thus, rather than executing one kernel at a time, the processing system schedules workgroups among kernels based on data dependencies between the kernels. By limiting the size of the subset to the amount of data that can be stored in the local cache (204, 208), the processing system increases the probability that data consumed by the workgroups of the subset will reside in the local cache and will not require memory access.

Description

[Background technology]

[0001] Typically, a processing unit, such as a graphics processing unit (GPU) or other parallel processor or central processing unit (CPU), implements multiple processing elements (called compute units in the case of a GPU or processor cores in the case of a CPU) that execute instructions simultaneously or in parallel. For example, a compute unit in a GPU executes a kernel that contains multiple workgroups, and each workgroup contains several threads, such as multiple threads that execute the same instruction on various data sets. Instructions within a kernel correspond to shaders that perform graphics processing, neural networks that perform machine learning tasks, etc. The processing unit also includes a command processor that fetches commands from a command buffer, allocates resources, and schedules the commands for execution on one or more processing elements within the processing unit. Some applications require the resources of multiple processing units. For example, a machine learning application that runs using a neural network can be executed using several GPUs operating in parallel. GPUs communicate with each other by sending data to other GPUs' buffers and notifying other GPUs that data is available in their buffers. However, traditional approaches to kernel scheduling can require a large number of memory fetch cycles relative to the number of compute cycles for some sets of operations, which can create bottlenecks as data size and the number of devices increase, leading to uneven resource utilization and potentially negatively impacting processor performance. Summary of the Invention [Means for solving the problem]

[0002] In a first embodiment, a method includes scheduling, with a command processor, a first subset of one or more workgroups of a first kernel to execute immediately before a second subset of one or more workgroups of a second kernel based on a data dependency between the first subset and the second subset, wherein the second subset of one or more workgroups executes before the first kernel completes execution and consumes data produced by the first subset of one or more workgroups.

[0003] The method may further include, at runtime, partitioning the plurality of workgroups into a first subset of one or more workgroups and a second subset of one or more workgroups based on data dependencies. In some embodiments, the partitioning is based on a size and locality of a cache to a set of one or more computing units executing one or more workgroups in the second subset, the cache storing data generated by the one or more workgroups in the second subset.

[0004] In some embodiments, the method further includes specifying, by the runtime, a mapping between a first subset of the one or more workgroups of the first kernel and a second subset of the one or more workgroups of the second kernel. The mapping between the first subset of the one or more workgroups of the first kernel and the second subset of the one or more workgroups of the second kernel may be declared in a companion function of each of the first kernel and the second kernel. The number of workgroups in the first subset may be different from the number of workgroups in the second subset.

[0005] The method may further include inserting a barrier into the kernel packet to delay execution of the first subset of the one or more workgroups until the second subset of the one or more workgroups completes execution. In some embodiments, the method includes initializing a counter with a number of workgroups of the second subset of the one or more workgroups that must complete execution before the first subset of the one or more workgroups executes, and decrementing the counter in response to the workgroup reaching the barrier.

[0006] The method may include generating commands describing pairing of two or more kernels based on data dependencies, and scheduling one or more workgroups of the first subset and one or more workgroups of the second subset for execution on a processor based on the commands. In some embodiments, the method further includes placing commands and identifiers of kernel launches for one or more workgroups of the first subset and one or more workgroups of the second subset in a ring buffer, and scheduling the workgroups with a command processor based on the commands and identifiers.

[0007] In another embodiment, the command processor is configured to schedule a first subset of one or more workgroups of a first kernel for execution on the parallel processors, and immediately after execution of the first subset but before the first kernel completes execution, schedule a second subset of one or more workgroups of a second kernel for execution on the parallel processors based on data dependencies between the first and second subsets, wherein the second subset of one or more workgroups consumes data produced by the first subset of one or more workgroups.

[0008] The number of workgroups in the first subset may differ from the number of workgroups in the second subset. The command processor may be further configured to schedule the second subset based on a barrier inserted in the kernel packet to delay execution of the second subset of one or more workgroups until the first subset of one or more workgroups complete execution. In some embodiments, the command processor is further configured to initialize a counter with the number of workgroups in the second subset of one or more workgroups that must complete execution before the first subset of one or more workgroups executes, and to decrement the counter in response to the workgroups reaching the barrier.

[0009] The command processor may be further configured to generate commands describing pairings of two or more kernels based on data dependencies, and schedule one or more workgroups of the first subset and one or more workgroups of the second subset for execution on the processor based on the commands. In some embodiments, the command processor is further configured to place commands and identifiers of kernel launches for one or more workgroups of the first subset and one or more workgroups of the second subset in a ring buffer, and schedule the workgroups on the command processor based on the commands and identifiers.

[0010] In another embodiment, a processing system includes a parallel processor, the parallel processor including a command processor configured to schedule a first subset of one or more workgroups of a first kernel to execute immediately before a second subset of one or more workgroups of a second kernel, the second subset of one or more workgroups consuming data generated by the first subset of one or more workgroups and executing before the first kernel completes execution. The plurality of workgroups of the first subset of one or more workgroups may be different from the plurality of workgroups in the second subset of one or more workgroups.

[0011] The processing system may further include a runtime configured to insert a barrier into the kernel packet to delay execution of a first subset of the one or more workgroups until a second subset of the one or more workgroups completes execution, and the runtime may be further configured to initialize a counter with a number of workgroups of the second subset of the one or more workgroups that must complete execution before the first subset of the one or more workgroups executes, and to decrement the counter in response to the workgroups reaching the barrier.

[0012] The runtime may be configured to generate commands describing pairings of two or more kernels based on data dependencies between the work-groups of the two or more kernels, and the command processor is further configured to schedule a first subset of the one or more work-groups and a second subset of the one or more work-groups for execution on the processor based on the commands. The processing system further includes a ring buffer, and the runtime is configured to place commands and identifiers of kernel launches for the first subset of the one or more work-groups and the second subset of the one or more work-groups in the ring buffer, and the command processor is further configured to schedule the work-groups based on the commands and identifiers.

[0013] The present disclosure may be better understood, and its numerous features and advantages made apparent to those skilled in the art by referencing the accompanying drawings, in which: The use of the same reference numbers in different drawings indicates similar or identical items. [Brief explanation of the drawings]

[0014] [Figure 1] FIG. 1 is a block diagram of a processing system that performs data dependency-aware scheduling according to some embodiments. [Figure 2] FIG. 1 is a diagram of a convolution with data dependencies between workgroups of multiple layers, according to some embodiments. [Figure 3] FIG. 10 illustrates the addition of two convolutions with data dependency between layers, according to some embodiments. [Figure 4] 2 is a block diagram of a portion of the processing system of FIG. 1 that performs data dependency-aware scheduling according to some embodiments. [Figure 5] FIG. 1 is a block diagram of a data packet including a command indicating a data dependency within or between kernels, according to some embodiments. [Figure 6]FIG. 1 is a flow diagram illustrating a method for scheduling workgroups based on data dependencies in one or more parallel processors, according to some embodiments. [Figure 7] FIG. 1 is a block diagram of an all-reduce operation performed across multiple parallel processors. [Figure 8] FIG. 1 is a block diagram of a fine-grained producer and all-reduce operation running across multiple parallel processors in a ring-based configuration, according to some embodiments. [Figure 9] FIG. 1 is a block diagram of an all-reduce operation in which blocks of producers are scheduled in a staggered manner across multiple parallel processors, according to some embodiments. [Figure 10] FIG. 1 is a block diagram of an all-reduce operation performed among a set of fine-grained producers and parallel processors, according to some embodiments. [Figure 11] FIG. 1 is a block diagram of an all-reduce operation with blocks of producers scheduled in a staggered fashion across a set of parallel processors, according to some embodiments. [Figure 12] FIG. 1 is a block diagram of a command processor including a block schedule checker according to some embodiments. [Figure 13] FIG. 5 is a block diagram of a portion of the processing system of FIGS. 1 and 4 that performs data dependency-aware scheduling according to some embodiments. DETAILED DESCRIPTION OF THE INVENTION

[0015] Traditional hardware scheduling for parallel processors such as GPUs takes a relatively inflexible approach to kernel execution. For example, when a programmer launches a kernel by specifying the number of workgroups for the kernel and the number of threads in each workgroup, the workgroups are typically scheduled to execute one entire layer sequentially on the parallel processor's computational units (referred to herein as "breadth-first" scheduling). Typically, each layer generates more data than can fit in its local cache (i.e., a cache proximal to and dedicated to the computational unit executing the workgroup) and often consumes data produced by previously executed layers in the same order. Data that does not fit in the local cache is stored in one or more shared levels of cache or memory. Thus, subsequent layers or kernels that depend on data produced by previous layers or kernels are unlikely to find the data in the local cache and must instead retrieve the data from another level of the cache hierarchy or memory, resulting in increased latency and reduced processing performance.

[0016] For example, dispatches of a grid of work items (also referred to herein as threads or streams), e.g., corresponding to pixels of a display, access data sets stored in the cache of a parallel processing unit. Some data sets, such as surfaces or textures, are modified and read by successive dispatches. For example, a first dispatch, called a producer dispatch, writes data to the cache, and subsequent dispatches, called consumer / producer dispatches, read and modify the data written by the producer dispatch. Consumer dispatches read the data written by the previous dispatch. However, the amount of data that can be stored in the cache is constrained by the size of the cache, and the amount of data generated by producer dispatches often exceeds the storage capacity of the cache, resulting in cache "thrashing." Such data is often evicted from the cache before it can be read by subsequent dispatches.

[0017] An inflexible approach to scheduling kernels can also adversely affect resource utilization. For example, multiple workgroups within a kernel or across two or more kernels may utilize different resources, such as different memory channels, of a parallel processor. Sequential scheduling of such workgroups may leave other resources of the parallel processor idle. Furthermore, workgroups scheduled to execute across multiple parallel processors, such as communication / reduction operations, may create bottlenecks as data size and the number of devices increase, potentially resulting in inefficient resource utilization due to, for example, devices remaining idle while communicating with each other or the network remaining idle while devices are performing operations.

[0018] 1-13 illustrate a technique for flexibly scheduling workgroups within a kernel, between kernels, within a parallel processor, and across multiple parallel processors based on data dependencies between the workgroups to enhance processing efficiency. The processing system's configurable scheduling mechanism divides workgroups into subsets, referred to herein as "generalized regions of interest" (GROIs), based on data dependencies and schedules workgroups in a first GROI that generate data to execute immediately before workgroups in a second GROI that consume data produced by the first GROI. For example, in the context of image processing, a GROI could be a tile of an image. In the context of convolutional neural networks, a GROI could be a subset of filters.

[0019] In some embodiments, a first GROI includes a workgroup of a first kernel, and a second GROI includes a workgroup of a second kernel. For example, in some embodiments, a processing system schedules a first subset of a first kernel's workgroups for execution, followed by a second subset of a second kernel's workgroups that consume data generated by the first subset of workgroups. The processing system may then schedule a third subset of a third kernel's workgroups that consume data generated by the second subset of workgroups, or in some embodiments, another subset of the first kernel's workgroups, followed by another subset of the second kernel's workgroups, executing one subset of workgroups at a time through the graphics pipeline. Thus, rather than executing one kernel at a time, the processing system instead schedules workgroups across multiple kernels based on data dependencies between the kernels, and schedules a subset of workgroups from one kernel without waiting for all workgroups of another kernel to complete execution (referred to herein as "depth-first" scheduling). By limiting the size of the GROI to the amount of data that can be stored in a local cache, the processing system increases the probability that data consumed by a workgroup in the GROI will reside in the local cache and not require a memory access. Furthermore, by scheduling subsets of workgroups active within a kernel to use different resources, such as different memory channels, the processing system allows the workgroups to more effectively utilize the resources of a parallel processor.

[0020] In some embodiments, a programmer indicates data dependencies between workgroups in an application programming interface (API). The processing system then generates commands describing pairings of two or more kernels based on the indicated data dependencies and further schedules one or more workgroups of the first GROI and one or more workgroups of the second GROI for execution on a processor based on the commands. In some embodiments, the commands and identifiers of kernel launches for the workgroups of the first GROI and the workgroups of the second GROI are placed in a ring buffer, and a command processor schedules the workgroups based on the commands and identifiers.

[0021] In some embodiments, a programmer indicates data generation ordering requirements for a producer kernel, including multiple workgroups that generate blocks of data. In some embodiments, an indication of the data generation ordering requirements is included in the metadata of the producer kernel. In some embodiments, a scheduler specifies, during launch of a producer kernel, that data generated by the producer kernel be generated in a particular order based on other kernels that launch before, at the same time as, or after the producer kernel. For example, in some embodiments, subsequent kernels are all-reduce. The scheduler receives the indication and, based on the indication, schedules multiple workgroups of the producer kernel to run in order to generate blocks of data starting with an initial block of data.

[0022] In some embodiments, the neural network is implemented on multiple parallel processors, such as GPUs of a processing system, whereby each processor performs the same producer (e.g., general matrix multiplication (GEMM)) operation, followed by a reduction operation, such as an all-reduce operation, using the computational units of the parallel processors. As used herein, an "all-reduce operation" is defined as a reduction operation that combines multiple data inputs from each processor's producer operations into a single data output using arithmetic and / or logical operators, possibly followed by the broadcast of a single data set. In a ring-based all-reduce operation, a parallel processor receives data from a previous parallel processor, reduces the received data with its own data, and sends the reduced data to the next parallel processor. In such cases, scheduling all work-groups of the producer operations first, and then all work-groups of the consumer all-reduce operations, may result in underutilization / idling of computational and network resources.

[0023] To increase the processing efficiency of the allreduce operation across multiple parallel processors, in some embodiments, the parallel processors of the processing system schedule a first set of workgroups (GROIs) of producer operations to immediately follow a second GROI of the allreduce operation that consumes data produced by a first GROI. Thus, the steps of the allreduce operation are coordinated among the parallel processors to overlap computation and communication between the parallel processors, resulting in faster workload execution and more efficient resource utilization.

[0024] In some embodiments, the processing system includes a scheduling mechanism for generating data for fine-grained producers, such as for device-to-device communications over a network to enable overlap of AllReduce communications with producer computations. The scheduling mechanism enables a first parallel processor to schedule and execute a first set of workgroups (blocks) of producer operations (GROI) to generate data for transmission to a second parallel processor in a desired traffic pattern. Concurrently, the second parallel processor schedules and executes a different GROI of producer operations to generate data for transmission to a third parallel processor or back to the first parallel processor in the desired traffic pattern. In some embodiments, the scheduling mechanism schedules workgroups of producer kernels to execute in a first order on the first parallel processor and in a second order on the second parallel processor. In some embodiments, the order of the different workgroups is scheduled by having multiple versions or kernel implementations of the same operation of commands / packets queued in a buffer, one of which is dynamically selected to generate the desired traffic.

[0025] In some embodiments, multiple kernels are scheduled to compute across multiple chiplets of a chiplet-based device in a manner that efficiently utilizes inter-chiplet bandwidth. For example, in some embodiments, a first kernel is scheduled on a first chiplet that primarily accesses local resources, such as memory, during a time when a second kernel is scheduled on a second chiplet that is expected to generate significant inter-chiplet traffic. Similarly, in embodiments with multiple concurrently executing kernels, the scheduling mechanism selects kernels for scheduling based on whether the kernels are latency-aware or resource-aware and whether the operations are running singly or concurrently.

[0026] 1 is a block diagram of a processing system 100 configured to perform data dependency-aware scheduling according to some embodiments. Processing system 100 is generally configured to execute a set of instructions (e.g., a program) or commands (e.g., draw commands) to perform tasks on behalf of an electronic device. Accordingly, in different embodiments, processing system 100 may be incorporated into any of a variety of electronic devices, such as a desktop computer, a laptop computer, a server, a smartphone, a tablet, a gaming console, etc.

[0027] Processing system 100 includes or has access to memory 105 or other storage components implemented using a non-transitory computer-readable storage medium, such as dynamic random-access memory (DRAM). However, memory 105 may also be implemented using other types of memory, including static random access memory (SRAM), non-volatile RAM, etc. Processing system 100 also includes a bus 110 that facilitates communication between entities executing in processing system 100, such as memory 105. Some embodiments of processing system 100 include other buses, bridges, switches, routers, etc., which are not shown in FIG. 1 for clarity.

[0028] The processing system 100 includes one or more parallel processors 115 configured to render images for presentation on the display 120. A parallel processor is a processor capable of executing a single instruction on multiple data, or threads, in parallel. Examples of parallel processors include processors such as graphics processing units (GPUs), massively parallel processors, single instruction, multiple data (SIMD) architecture processors, and single instruction, multiple thread (SIMT) architecture processors for performing graphics, machine intelligence, or computational operations. In some embodiments, the parallel processors are separate devices included as part of a computer. In other embodiments, such as an advanced processor unit, the parallel processors are included in a single device along with a host processor, such as a central processor unit (CPU). The following description uses a graphics processing unit (GPU) for illustrative purposes, but the embodiments and implementations described below are applicable to other types of parallel processors.

[0029] The parallel processors 115 can render objects to generate pixel values ​​that are provided to the display 120, which uses the pixel values ​​to display an image representing the rendered objects. Some embodiments of the parallel processor 115 can also be used for general-purpose computing. For example, the parallel processor 115 can be used to execute machine learning algorithms, such as neural networks. In some cases, the operations of multiple parallel processors 115 are coordinated to execute machine learning algorithms, for example, when a single parallel processor 115 does not possess sufficient processing power to execute the machine learning algorithm on its own. The multiple processors 115 communicate via one or more interfaces (not shown in FIG. 1 for ease of illustration).

[0030] The parallel processor 115 implements multiple processing elements (also called compute units) 125 configured to execute instructions simultaneously or in parallel. The parallel processor 115 also includes internal (or on-chip) memory 130, including a local data store (LDS), and caches, registers, or buffers utilized by the compute units 125. The internal memory 130 stores data structures describing tasks to be performed by one or more of the compute units 125. In the illustrated embodiment, the parallel processor 115 communicates with the memory 105 via the bus 110. However, some embodiments of the parallel processor 115 communicate with the memory 105 via a direct connection or via other buses, bridges, switches, routers, etc. The parallel processor 115 can execute instructions stored in the memory 105, and the parallel processor 115 can store information, such as results of executed instructions, in the memory 105. For example, memory 105 may store copies 135 of program code instructions to be executed by parallel processor 115, such as program code corresponding to a machine learning algorithm or a neural network. Parallel processor 115 also includes a command processor 140 that receives task requests and dispatches tasks to one or more of the compute units 125. Command processor 140 is a set of hardware configured to receive commands from CPU 145 and prepare the received commands for processing. For example, in some embodiments, command processor 140 buffers received commands, organizes received commands into one or more queues for processing, performs operations such as decoding or otherwise interpreting received commands, and so forth.

[0031] In some embodiments, parallel processor 115 implements a graphics pipeline (not shown in FIG. 1 for simplicity) that includes multiple stages configured to simultaneously process various primitives in response to draw calls. The stages of the graphics pipeline in parallel processor 115 can simultaneously process various primitives generated by an application, such as a video game. When geometry is presented to the graphics pipeline, hardware state settings are selected to define the state of the graphics pipeline. Examples of state include rasterizer state, blend state, depth stencil state, primitive topology type of the presented geometry, and shaders (e.g., vertex shader, domain shader, geometry shader, hull shader, pixel shader, etc.) used to render the scene. Shaders embodied in the graphics pipeline state are represented by corresponding bytecode. In some cases, information representing the graphics pipeline state is hashed or compressed to provide a more efficient representation of the graphics pipeline state.

[0032] Processing system 100 also includes a central processing unit (CPU) 145 connected to bus 110 and in communication with parallel processors 115 and memory 105 via bus 110. In the illustrated embodiment, CPU 145 implements multiple processing elements (also referred to as processor cores) 150 configured to execute instructions simultaneously or in parallel. CPU 145 may execute instructions, such as program code 155, stored in memory 105, and may also store information, such as results of executed instructions, in memory 105. CPU 145 may also initiate graphics operations by issuing commands or instructions (sometimes referred to herein as "draw calls") to parallel processors 115.

[0033] Input / output (I / O) engine 160 processes input or output operations associated with display 120 and other elements of processing system 100, such as a keyboard, mouse, printer, external disk, etc. I / O engine 160 is coupled to bus 110 such that I / O engine 160 communicates with memory 105, parallel processors 115, or CPU 145.

[0034] In operation, CPU 145 issues draw calls to parallel processors 115 to initiate processing of kernels corresponding to program instructions executed by parallel processors 115. Multiple instances of a kernel, referred to herein as threads or work items, execute simultaneously or in parallel using subsets of compute units 125. In some embodiments, threads execute according to a single-instruction-multiple-data (SIMD) protocol, such that each thread executes the same instruction on different data. Threads are collected into workgroups that run on various compute units 125. For example, command processor 140 can receive draw calls and schedule tasks to be executed on compute units 125.

[0035] To increase processing efficiency, command processor 140 is configured to receive commands inserted into kernel packets that indicate dependencies between workgroups of one or more kernels. For example, the command may indicate that one workgroup of kernel 2 depends on two workgroups of kernel 1. Based on this command, command processor 140 enqueues the workgroups of one or more kernels for a hardware scheduler (not shown), which dispatches the workgroups to the compute units 125 of parallel processor 115, as described in more detail with respect to FIG. 3 .

[0036] FIG. 2 illustrates an execution path 200 of convolutional layers 210, 212, and 214 with data dependencies between workgroups of the layers, according to some embodiments. In some embodiments, a programmer or API declares dependencies between workgroups of convolutional layers 210, 212, and 214. In other embodiments, an inference engine (not shown) automatically infers dependencies between workgroups of convolutional layers 210, 212, and 214. The inference engine is an automated software framework that orchestrates the overall computation and, in some embodiments, automatically infers layers that require cross-device reduction. In some embodiments, the inference engine invokes kernels in an appropriate dynamic scheduling order. The inference engine may execute on parallel processors 115, dedicated logic (e.g., an application specific integrated circuit (ASIC)), other types of components, other types of logic, and / or any combination of multiple different types of components or processing units. In some embodiments, the “inference engine” is hardware and / or software that receives image data and generates one or more label probabilities for the image data.

[0037] In applications such as computer vision and machine learning, the generalized region of interest (GROI) of a layer usually depends on the GROI of the previous layer. A GROI is a multidimensional region of interest that stores rectangular constraints for height and width, constraints for channel (C) and batch (N) dimensions, and filters (K) divided for group-wise computation. Machine learning convolution kernels usually require multiple workgroups to compute filters, and in some cases, if the GROI is too large, any one workgroup cannot compute all the filters in the GROI.

[0038] In the illustrated example, either a programmer, an API, or an inference engine declares data dependencies indicating producer / consumer relationships between one or more workgroup subsets 222 of convolutional layer 210, one or more workgroup subsets 224 of convolutional layer 212, and one or more workgroup subsets 226 of convolutional layer 214, and between one or more workgroup subsets 232 of convolutional layer 210, one or more workgroup subsets 234 of convolutional layer 212, and one or more workgroup subsets 236 of convolutional layer 214. One or more workgroup subsets 232 overlap with one or more workgroup subsets 222, one or more workgroup subsets 234 overlap to a lesser extent with one or more workgroup subsets 224, and one or more workgroup subsets 236 are adjacent to but do not overlap with one or more workgroup subsets 226.

[0039] Additionally, the programmer, API, or inference engine declares dependencies indicating producer / consumer relationships between one or more subsets 216 of workgroups in convolutional layer 210, one or more subsets 218 of workgroups in convolutional layer 212, and one or more subsets 220 of workgroups in convolutional layer 214. In some embodiments, the programmer, API, or inference engine declares a 1-to-1, N-to-1, 1-to-N, or N-to-M data dependency pattern between workgroups, such that a first number of workgroups in a first layer depend on a second number of workgroups in a second layer. In some embodiments, the programmer, API, or inference engine explicitly declares data dependencies by workgroup ID. Based on the declared data dependencies, the processing system 100 schedules workgroups so that the consumer workgroup executes immediately after the producer workgroup, thereby increasing the likelihood that the generated data will be found in the L1 or L2 (local) cache.

[0040] In addition to performing a "depth-first" rather than "breadth-first" ordering of the entire kernel, the processing system 100 specifies how many workgroups of a data producer kernel should be scheduled at one time to ensure that enough workgroups are in flight to fully occupy the assigned compute units 402. Thus, the size of the GROI is determined based on the criteria that it is large enough to be divided into a sufficient number of workgroups to occupy most or all of the compute units CU-1 202 and CU-2 206 (each processing a micro-tile / micro-GROI), yet small enough not to slash (i.e., generate data that exceeds the size of) their associated local caches, cache-1 204 and cache-2 208.

[0041] Specifically, as an example, a machine learning pipeline includes two kernels that run back-to-back: kernel_1, which has input size in_1 and generates output data of size out_1; and kernel_2, which has excess input data in_2 that exceeds size out_1 and generates data of size out_2. Kernel_1 includes wg_1 workgroups, and kernel_2 includes wg_2 workgroups, where M_1 workgroups of kernel_1 generate data consumed by M_2 workgroups of kernel_2. The processing system 100 calculates the memory footprint of launching m_1 workgroups of kernel_1 and further ensures that all dependent workgroups of kernel_2 are processed depth-first (m_2) before continuing to the next GROI of kernel_1. Here, m_2 / m_1=M_2 / M_1, and therefore, m_2=(m_1 / M_1)*M_2. The processing system 100 determines that the m_1 workgroups of kernel_1 require (m_1 / M_1)*in_1 data and produce (m_1 / M_1)*out_1 data, which are immediately consumed by the m_2 workgroups of kernel_2. Furthermore, each workgroup of kernel_2 requires additional data (m_2 / M_2)*in_2 and produces output (m_2 / M_2)*out_2. By summing the memory contributions of each GROI, the processing system 100 determines the total amount of concurrent memory required as a function of m_1. In some embodiments, the processing system 100 determines the global optimum of the function via stochastic gradient descent (SGD) based on theoretical analysis or benchmarking in a preprocessing stage of the optimization.

[0042] In the illustrated example, based on the data dependencies described above, CU-1 202 is scheduled to execute subset 232 of one or more workgroups of convolutional layer 210 and store data generated by subset 232 in cache-1 204, while CU-2 206 is scheduled to execute subset 222 of one or more workgroups of convolutional layer 210 and store data generated by subset 222 in cache-2 208. In some embodiments, subsets 232 and 222 overlap such that some workgroups are included in both subsets 232 and 222. Before the remainder of the workgroups of convolutional layer 210 (such as those of subset 216) are executed, CU-1 202 is scheduled to execute subset 234 of one or more workgroups of convolutional layer 212 based on the data generated by subset 232 and stored in cache-1 204. Data generated by subset 234 is stored in cache-1 204 (e.g., by overwriting data generated by subset 232). Similarly, CU-2 206 is scheduled to execute subset 224 of one or more workgroups of convolutional layer 212 based on data generated by subset 222 and stored in cache-2 208. Data generated by subset 224 is stored in cache-2 208 (e.g., by overwriting data generated by subset 222). In some embodiments, subsets 234 and 224 overlap such that some workgroups are included in both subsets 234 and 224 to a lesser extent than the overlap between subsets 232 and 222.

[0043] Before the remainder of the workgroups in the convolutional layer 212 (such as those in subset 218) are executed, CU-1 202 is scheduled to execute subset 236 of one or more workgroups in the convolutional layer 214 based on data generated by subset 234 and stored in cache-1 204. The data generated by subset 236 is stored in cache-1 204 (e.g., by overwriting the data generated by subset 234). Similarly, CU-2 206 is scheduled to execute subset 226 of one or more workgroups in the convolutional layer 212 based on data generated by subset 224 and stored in cache-2 208. The data generated by subset 226 is stored in cache-2 208 (e.g., by overwriting the data generated by subset 224). In some embodiments, subsets 236 and 226 are adjacent but non-overlapping. For example, if subset 236 of convolutional layer 214 includes one workgroup that computes 16x16 pixel microtiles, data is computed from 18x18 pixel microtiles of subset 234 of convolutional layer 212, then from 20x20 pixel microtiles of subset 232 of convolutional layer 210. Enabling multi-level depth-first traversal of multiple convolutional layers is implicit (e.g., a runtime feature) in some embodiments and explicit (e.g., by a programmer calling a runtime function) in other embodiments. In this way, processing system 100 efficiently uses cache-1 204 and cache-2 208.

[0044] FIG. 3 is a diagram of a merger 300 for two execution paths 302, 304. Each of the execution paths 302, 304 executes similarly to the execution path 200 of FIG. 2. Thus, for example, in execution path 302, subset 222 of convolutional layer 210 executes and generates data for consumption from subset 224 of convolutional layer 212. Subset 224 is blocked from beginning execution, e.g., by a barrier, until subset 222 completes execution on the compute unit and writes the data to the compute unit's local cache. Once subset 222 completes execution, the barrier is removed and subset 224 executes on the compute unit, consuming the data written to the local cache by subset 222 and producing data written to the local cache. Subset 226 is blocked from beginning execution until subset 224 completes execution. Once subset 224 completes execution, subset 226 executes on the computational unit, consuming data written to the local cache by subset 224 and producing data stored in the local cache.

[0045] In simultaneous or overlapping time frames, subset 232 executes on a compute unit and generates data that is consumed in the compute unit's local cache by subset 234. Subset 234 executes on a compute unit and consumes the data produced by subset 232 and generates data that is consumed in the local cache by subset 236. Subset 236 executes on a compute unit and consumes the data produced by subset 234.

[0046] A similar process occurs in execution path 304, in which subset 322 of convolutional layer 310 executes and generates data for consumption from subset 324 of convolutional layer 312. Subset 324 is blocked from starting execution, for example, by a barrier, until subset 322 completes execution on the compute unit and writes the data to the compute unit's local cache. Once subset 322 completes execution, the barrier is removed and subset 324 executes on the compute unit, consuming the data written to the local cache by subset 322 and producing the data written to the local cache. Subset 326 is blocked from starting execution until subset 324 completes execution. Once subset 324 completes execution, subset 326 executes on the compute unit, consuming the data written to the local cache by subset 324 and producing data stored in the local cache.

[0047] In simultaneous or overlapping time frames, subset 332 executes on a compute unit and generates data that is consumed in the compute unit's local cache by subset 334. Subset 334 executes on the compute unit, consumes the data produced by subset 332, and generates data that is consumed in the compute unit's local cache by subset 336. Subset 336 executes on the compute unit and consumes the data produced by subset 334. Execution paths 302 and 304 execute concurrently in some embodiments and sequentially in other embodiments.

[0048] Execution paths 302, 304 are merged by adding subset 236 of convolutional layers 214 with subset 336 of convolutional layers 314 to produce result 338 of convolutional layer 316. Similarly, subset 226 of convolutional layers 214 is added to subset 326 of convolutional layers 314 to produce result 328 of convolutional layer 316. Depth-first scheduling of subsets of work-groups across convolutional layers 210, 212, 214 and 310, 312, 314 improves data locality in local caches, reducing memory access and latency and improving processing performance.

[0049] FIG. 4 is a block diagram of a portion 400 of the processing system 100 of FIG. 1 that performs data dependency-aware scheduling, according to some embodiments. The processing system 100 maintains, in memory 105, one or more control logic modules for execution by the processing system 100. The control logic modules include an operating system 422, a kernel mode driver 414, a user mode driver 416, and an inference engine 424. These control logic modules control various aspects of the operation of the CPU 145 and the parallel processor 115. For example, the operating system 422 communicates directly with the hardware and provides an interface to the hardware for other software executing on the CPU 145. The kernel mode driver 414 controls the operation of the parallel processor 115, for example, by providing an API for software (e.g., applications) executing on the CPU 145 to access various features of the parallel processor 115. The kernel mode driver 414 also includes a just-in-time compiler that compiles programs for execution by the processing components of the parallel processor 115 (such as the compute units 402, discussed in more detail below).

[0050] Parallel processors 115 execute commands and programs for selected functions, such as graphics and non-graphics operations, that are suitable for parallel and / or out-of-order processing. Parallel processors 115 are used to perform graphics pipeline operations, such as pixel operations, geometry calculations, and rendering of images to display 120, based on commands received from CPU 145. Parallel processors 115 also perform computational operations not directly related to graphics operations, such as operations related to video, physics simulations, computational fluid dynamics, etc., based on commands received from CPU 145.

[0051] The parallel processor 115 includes a compute unit 402 that includes one or more SIMD units (not shown) that perform operations at the request of the CPU 145 in parallel according to the SIMD paradigm. The SIMD paradigm allows multiple processing elements to share a single program control flow unit and program counter, thus executing the same program but with different data. In one example, each SIMD unit includes 16 lanes, each lane executing the same instruction simultaneously with other lanes in the SIMD unit, but capable of executing the instruction on different data. In one embodiment, each compute unit 402 may have a local L1 cache 404. In one embodiment, multiple compute units 402 share an L2 cache 406.

[0052] The basic unit of execution in compute unit 402 is the work item. Each work item represents a single instantiation of a program executing in parallel on a particular lane. Work items can execute simultaneously on a single SIMD processing unit as a "wavefront." One or more wavefronts are included in a "workgroup," which includes a collection of work items designated to execute the same program. A workgroup is executed by executing each of the wavefronts that make up the workgroup. Alternatively, a wavefront may execute sequentially on a single SIMD unit, or may execute partially or fully in parallel on different SIMD units. A wavefront can be thought of as the largest collection of work items that can execute simultaneously on a single SIMD unit. Thus, if a command received from CPU 145 dictates that a particular program be parallelized to the extent that it cannot be executed simultaneously on a single SIMD unit, the program is split into two or more wavefronts that are either parallelized on two or more SIMD units or serialized on the same SIMD unit (or both parallelization and serialization are implemented, as appropriate). The hardware scheduler 412 performs operations related to scheduling the various wavefronts onto the various compute units 402 and SIMD units.

[0053] The parallelism provided by the compute units 402 is suitable for graphics-related operations such as pixel value calculations, vertex transformations, and other graphics operations. Thus, in some cases, the graphics processing pipeline 410 receives graphics processing commands from the CPU 145 and provides computational tasks to the compute units 402 for parallel execution.

[0054] Computation units 402 are also used to perform computational tasks that are not related to graphics or that are not performed as part of the "normal" operation of graphics processing pipeline 410 (e.g., custom operations performed to supplement the processing performed for the operation of graphics processing pipeline 410). An application (not shown) or other software running on CPU 145 sends programs defining such computational tasks to parallel processors 115 for execution.

[0055] The parallel processor 115 further includes a runtime 418, a command buffer such as a ring buffer 420, and a command processor 140. The runtime 418 is a software execution environment that interfaces between the host program and the parallel processor 115. The runtime 418 sends work to a hardware queue in the ring buffer 420 and specifies the order in which workloads are executed on the compute units 402.

[0056] To support scheduling that is aware of data dependencies between one or more kernel workgroups, user mode driver 416 receives instructions (not shown) from user processes and sends work requests to kernel mode driver 414. In some embodiments, the instructions include hints from an API or inference engine 424 regarding data dependencies between one or more kernel workgroups. Kernel mode driver 414 aggregates requests from user mode driver 416 and sends the requests to a ring buffer 420 accessible to both CPU 145 and parallel processor 115. Furthermore, kernel mode driver 414 provides runtime 418 with indications 430 of data dependencies between one or more kernel workgroups based on information received from the API or inference engine 424. In some embodiments, block synchronizer 426 serves as a synchronization point for a specified subset of workgroups. The subset of workgroups is either explicitly specified or determined via a descriptor, such as the active workgroup executing the current GROI.

[0057] The runtime 418 checks the user-specified data dependency patterns and / or explicit dependencies indicated by the indicators 430 against the specified subset of workgroups in the block synchronizer 426. Based on the indicators 430 and / or the specified subset of workgroups, the runtime 418 generates and inserts commands 432 into kernel packets that the runtime 418 enqueues in the ring buffer 420. The command processor 140 is configured to read the kernel packets containing the commands 432 from the ring buffer 420 and consume work based on the kernel group's dependent workgroups, rather than simply consuming the kernel's workgroups in linear order. The hardware scheduler 412 is configured to schedule the workgroups of the kernel group "in flight" on one or more of the compute units 402 according to the declared data dependencies.

[0058] In some embodiments, runtime 418 supports fine-grained control of workgroup synchronization by inserting barriers or memory fences into kernel packets that store information about which workgroups of a first kernel must wait due to the production of data consumed by a workgroup of a second kernel. Thus, the barrier delays the execution of consumer workgroups of a second kernel until the execution of producer workgroups of the first kernel has completed. In contrast, traditional scheduling techniques specify that all workgroups of a kernel complete execution before any workgroups of a subsequent kernel. In some embodiments, the barrier is implemented using a global atomic as the synchronization medium, such as initializing a counter with the number of workgroups that need to complete execution and decrementing the counter when a workgroup reaches the barrier. In other embodiments, the barrier is implemented based on signaling between workgroups. For example, in some embodiments, a condition variable or mwait / monitor state is programmed by runtime 418. In other embodiments, the condition variables or mwait / monitor states are explicitly programmed in the kernel's logic, and each workgroup also asserts completion on an interrupt line to trigger the scheduler interrupt routine. The scheduler firmware maintains a dependency tree, and once dependencies between "producer" workgroups are satisfied, it schedules the execution of "consumer" workgroups on compute units that have the same access rights to the producer workgroup's caches (typically L2, L3, etc. caches are shared between multiple compute units).

[0059] In some embodiments, to express dependencies between workgroups of various kernels at the level of runtime calls rather than at the level of kernel code, when declaring a consumer kernel, the runtime 418 explicitly specifies the mapping between the workgroups of the producer kernel and the workgroups of the consumer kernel. For example, a traditional API for launching a kernel might be: LaunchKernel(function_address,numBlocks,dimBlocks,args,shared MemBytes,stream) is. If the producer kernel (kernel_1) has no dependencies and further the consumer kernel (kernel_2) depends on kernel_1, then to express the dependencies between the workgroups of the various kernels at the level of runtime calls, an exemplary embodiment would be as follows:

number

number

[0060] In the above example embodiment, when declaring kernel_2, runtime 418 explicitly specifies a mapping between the workgroups of kernel_1 and kernel_2, such that a block in the "X" dimension of kernel_1 generates one block of data in the "X" dimension of kernel_2.

[0061] In another embodiment, the programmer implementing the kernel declares the GROI that each workgroup of the kernel computes, for example, in the following companion function for each kernel implementation:

number

number

number

[0062] In traditional scheduling, workgroups of a single kernel are assigned to processing elements, such as shader engines, in round-robin order, which are then launched onto compute units, also in round-robin order or a similar order. By declaring the GROI that each workgroup of a kernel computes in a companion function of each kernel implementation, runtime 418 and hardware scheduler 412 understand the dependencies between workgroups of various kernels and orchestrate the scheduling of workgroups with appropriate shader engines and compute units to maximize the probability that data for workgroup consumption will be found in the local cache of the compute unit executing the workgroup. In some embodiments, hardware scheduler 412 ensures that a set of consumer kernel workgroups and any producer kernel workgroups on which the set of workgroups depend execute either (1) on a set of processing elements that share an L1 cache (e.g., on the same compute unit) or (2) on a set of processing elements that share an L2 cache (e.g., on the same shader engine). The command processor 140 logic uses workgroup dependencies declared by the user or inferred by GROI declarations, and in some embodiments, causes the target shader engine to follow the workgroup dependencies. In other embodiments, the command processor 140 logic inserts signaling mechanisms, such as barriers, into the kernel code that specify that execution is delayed depending on a predetermined condition variable, and after execution, a signal is sent indicating that execution is complete.

[0063] 5 is a block diagram of a kernel packet 500 including commands 432 indicating data dependencies within or between kernels, according to some embodiments. In some embodiments, kernel packet 500 is a PM4 command packet generated by runtime 418 and including a unique ID 502 for the kernel launch. Command 432 describes the pairing of dependent kernels, for example, by referencing the dependent kernel's unique ID in the dataflow graph. For example, in some embodiments, KerRefID1_2 through KerRefID2_3 mean that two workgroups of kernels with unique ID KerRefID1 produce data that is consumed by three workgroups of kernels with unique ID KerRefID2.

[0064] 6 is a flow diagram illustrating a method 600 for scheduling workgroups based on data dependencies in one or more parallel processors 115, according to some embodiments. In some embodiments, method 600 is performed by a processing system, such as processing system 100 shown in FIG. 1. At block 602, a programmer, API, or inference engine 424 identifies data dependency patterns or explicit data dependencies between workgroups of one or more kernels. In some embodiments, kernel-mode driver 414 provides indications 430 of the identified data dependencies to runtime 418 based on information received from API or inference engine 424.

[0065] At block 604, runtime 418 receives indications 430 of the identified data dependencies and generates commands 432 that describe the pairing of the workgroups based on the data dependencies. In some embodiments, runtime 418 inserts commands 432 into a kernel packet and enqueues commands 432 in ring buffer 420.

[0066] In block 606, the command processor 140 reads the kernel packet containing the command 432 from the ring buffer 420 and schedules the workgroups based on the identified data dependencies. The hardware scheduler 412 schedules the workgroups of one or more "in-flight" kernel groups of the compute units 402 according to the declared data dependencies.

[0067] Some machine learning applications use input partitioning or data parallelism to distribute work across multiple parallel processor devices, where devices operate on independent data and parameter gradient reduction is required on all devices before the model is updated. Alternatively, tensor slicing or model parallelism requires partitioning each layer's parameters across devices and reducing output activations. Figure 7 is a block diagram of an all-reduce operation 700 conventionally performed across multiple parallel processors, where the communication / reduction operation runs as a separate kernel on the GPU after all data (e.g., all blocks on devices GPU-0 702, GPU-1 704, GPU-2 706, and GPU-3 708) is ready and relies on bulk all-reduce. However, such reduction operations can become a bottleneck as the data size and number of devices increase, because the devices are mostly idle during communication between them, and the communication network is idle while the devices are performing calculations. This can be mitigated by a fine-grained mechanism that communicates and reduces data as it is generated (all-reduce) among processors, but this is inefficient because the execution order of workgroups on each parallel processing device is uniform.

[0068] To illustrate a traditional fine-grained mechanism, in the example of FIG. 7, GPU-0 702, GPU-1 704, GPU-2 706, and GPU-3 708 each simultaneously produce their own version of block 1, then block 2, then block 3, and then block 4. All blocks of data are produced by the producer on each device and then processed by AllReduce. The number of each block (1, 2, 3, 4) within a device represents the order in which the blocks are produced by the producer. Therefore, all devices produce the same block of data at the same time. To reduce the same block of data (e.g., block 1) across multiple devices (e.g., when all of these devices next produce block 2), all processors must send their respective copies of block 1 to a single device. In such a scenario, all traffic is directed to a single device at once, creating inefficiencies in the network link and preventing the processing system from leveraging the reduction capabilities of multiple parallel devices. Another approach would be to further divide the generated block 1 among each device, but this can be inefficient and time-consuming due to small message sizes and high synchronization costs. Therefore, it is not possible to overlap computation and communication at a fine-grained level, as in traditional computation scheduling, where all participating devices generate the same block of data at once (e.g., all devices first generate their version of block 1, then block 2, etc.).

[0069] To more efficiently schedule work between devices and enable overlap of AllReduce communications and producer computations over the network, processing system 100 includes a scheduling mechanism for generating data for fine-grained communications between devices. This scheduling mechanism enables a first parallel processor to schedule and execute a first set of workgroups (blocks) of producer operations (GROIs) to generate data for transmission to a second parallel processor in a desired traffic pattern. In some embodiments, the second parallel processor simultaneously schedules and executes a different GROI of producer operations to generate data for transmission to a third parallel processor or back to the first parallel processor in a desired traffic pattern. As used herein, the term "traffic pattern" refers to the timing of data transmission from one parallel processor to another parallel processor or between computational units of a single parallel processor. It should further be understood that in some embodiments, a block includes multiple data elements, and in some embodiments, the multiple data elements are generated by multiple WGs packed into a GROI.

[0070] To enable this, in some embodiments, the API specifies during kernel launch that data produced by the kernel should be reduced across a specified set of parallel processors 115. Similar to the example described above with respect to FIG. 4 , the runtime 418 embeds information about the required reduction as a command 432 in a kernel packet 500 that is enqueued in the ring buffer 420 or other queue of each parallel processor 115 (GPU-0 702, GPU-1 704, GPU-2 706, GPU-3 708, etc.). The command processor 140 or other scheduler for each parallel processor 115 reads the command 432 from the kernel packet 500 and calculates the workgroup or sequence of workgroups to launch. The command processor 140 passes the information about the workgroup or sequence of workgroups to launch to the hardware scheduler 412, which begins execution of the workgroups from the given workgroup ID or sequence of workgroups.

[0071] 8 is a block diagram of an example 800 of fine-grained computation and all-reduce operations performed across multiple parallel processors in a ring-based configuration, according to some embodiments. In step 802, each GPU generates and prepares one block of data. GPU-0 702 has block 1, GPU-1 704 has block 4, GPU-2 706 has block 3, and GPU-3 708 has block 2 ready. GPU-0 702 communicates block 1 to GPU-1 704 while generating block 2 and reducing it with the version received from GPU-3 708. Simultaneously, GPU-1 704 communicates block 4 to GPU-2 706 while generating block 1 while reducing it with the version received from GPU-0 702. At the same time, GPU-2 706 communicates block 3 to GPU-3 708, while generating block 4 and reducing it with the version received from GPU-1 704. GPU-3 708 communicates block 2 to GPU-0 702, while simultaneously generating block 3 and reducing it with the version received from GPU-2 706. In this way, blocks are communicated between GPUs at predetermined times such that the network link between the GPUs is active while the GPUs are computing the block, allowing communication to be hidden by the computation.

[0072] In step 804, GPU-0 702 communicates reduced block 2 to GPU-1 704, while generating block 3 and reducing it with the version received from GPU-3 708. Concurrently, GPU-1 704 communicates reduced block 1 to GPU-2 706, while generating block 2 and reducing it with the version received from GPU-0 702. Concurrently, GPU-2 706 communicates block 4 to GPU-3 708, while generating block 1 and reducing it with the version received from GPU-1 704. Concurrently, GPU-3 708 communicates block 3 to GPU-0 702, while generating block 4 and reducing it with the version received from GPU-2 706.

[0073] In step 806, GPU-0 702 communicates reduced block 3 to GPU-1 704, while generating block 4 and reducing it with the version received from GPU-3 708. Simultaneously, GPU-1 704 communicates reduced block 2 to GPU-2 706, while generating block 3 and reducing it with the version received from GPU-0 702. Simultaneously, GPU-2 706 communicates reduced block 1 to GPU-3 708, while generating block 2 and reducing it with the version received from GPU-1 704. Simultaneously, GPU-3 708 communicates reduced block 4 to GPU-0 702, while generating block 1 and reducing it with the version received from GPU-2 706.

[0074] At step 808, each GPU has a fully reduced sub-array of blocks 1, 2, 3, and 4, ready for the all gather step. Staggering block production at each device allows each device to generate and send different blocks of data at each step, thereby maximizing the use of the computational and network resources of the processing system 100.

[0075] 9 is a block diagram 900 of an all-reduce operation with staggered block scheduling across multiple parallel processors, according to some embodiments. In some embodiments, the numbers in each matrix indicate the order of block execution for each of GPU-0 702, GPU-1 704, GPU-2 706, and GPU-3 708. It should be understood that a distributed computing operation utilizing staggered block scheduling may involve more or less than four GPUs, and further, the operation may include more than four blocks.

[0076] In some embodiments, the processing system 100 schedules a set of parallel processors 115 to simultaneously generate versions of the same block. The reduction can also be performed automatically once the output is generated and stored in a dedicated buffer / memory. This can be done in hardware using an in-network or in-memory based approach. FIG. 10 is a block diagram 1000 of fine-grained computation and all-reduce operations performed among a set of parallel processors, according to some embodiments. In step 1002, GPU-0 702 and GPU-3 708 generate block 1 while GPU-0 702 communicates block 2 to GPU-3 and simultaneously reduces this block with its local version, and GPU-1 704 and GPU-2 706 generate block 2 while GPU-2 706 communicates block 1 to GPU-1 and simultaneously reduces this block with its local version. In step 1004, GPU-0 702 and GPU-3 708 produce block 4 while communicating block 1 to GPU-1 704 and simultaneously reducing this block with a version of block 1, and GPU-1 704 and GPU-2 706 produce block 3 while communicating block 2 to GPU-3 708 and simultaneously reducing this block with a version of block 2. In step 1006, GPU-0 702 and GPU-3 708 generate block 3, while GPU-3 708 communicates block 4 to GPU-0 702 and simultaneously reduces this block with a version of block 4, while GPU-1 704 and GPU-2 706 generate block 4, while GPU-1 704 communicates block 3 to GPU-2 706 and simultaneously reduces this block with a version of block 3. In step 1008, GPU-0 702 and GPU-3 708 communicate block 3 to GPU-2 706 and simultaneously reduces this block with a version of block 3, and GPU-1 704 and GPU-2 706 communicate block 4 to GPU-0 702 and simultaneously reduces this block with a version of block 4.At the end of step 1008, each GPU has a reduced sub-array of blocks 1, 2, 3, and 4, ready for the all-gather step. As shown in the example of Figure 10, each device is solely responsible for reducing a particular block. By dividing the devices into two sets, with each set producing blocks in a particular order, processing system 100 distributes traffic between the devices and parallelizes the reduction.

[0077] 11 is a block diagram 1100 of an all-reduce operation with staggered block scheduling across a set of parallel processors, according to some embodiments. In some embodiments, the numbers in each matrix indicate the order of block execution for each of GPU-0 702, GPU-1 704, GPU-2 706, and GPU-3 708. It should be understood that a distributed computing operation utilizing staggered block scheduling may involve more or less than four GPUs, and further, the operation may include more than four blocks.

[0078] In some embodiments, the kernel invocation API call is modified for the programmer to indicate that the producer kernel's output should be reduced among devices within the node, followed by a vector that stores the participating device IDs. In embodiments where the output is not reduced or all devices in the system participate in the reduction, the vector is omitted. For example, in some embodiments, the API is modified as follows:

number

[0079] FIG. 12 is a block diagram of the command processor 140, which includes a block schedule checker 1202. The block schedule checker 1202 determines the scheduling order of blocks among the parallel processors 115 based on the size of the data to be reduced, the number of participating devices, and the device ID of the parallel processor 115 that includes the command processor 140. In some embodiments, the block schedule checker 1202 first determines chunk_size=#blocks / #GPUs. The block schedule checker 1202 determines the number of blocks from the number of workgroups in the kernel. Based on the number of blocks, the block schedule checker 1202 shifts block execution in each participating device (i.e., each participating parallel processor 115) by a multiple of the chunk size (i.e., device_id*chunk_size). For example, device 0 is staggered by 0*chunk_size (i.e., no staggering), device 1 is staggered by 1*chunk_size, ..., device n is staggered by n*chunk_size. In other embodiments, other scheduling policies are implemented by block schedule checker 1202. In some embodiments, block schedule checker 1202 is implemented as a hardware extension of command processor 140 or by modifying the hardware of command processor 140.

[0080] In embodiments requiring a fixed workgroup activation order, kernel packet 500 links multiple different kernel implementations by staggering the mapping of output blocks to workgroups, as shown in FIG. 13. FIG. 13 illustrates an implementation of a portion 1300 of the processing system of FIGS. 1 and 4 that provides data dependency-aware scheduling, according to some embodiments. A ring buffer 420 or other queue stores commands 1310, 1312, 1314 placed by runtime 418. Block schedule checker 1202 calculates the staggering, for example, based on the information stored in command 1310. Based on the staggering calculated by block schedule checker 1202, either implementation can be selected for execution during runtime by each parallel processor 115 using a baseline sequential workgroup scheduling strategy, such as sending workgroup WG-0 1320, then workgroup WG-1 1322, then workgroup WG-2 1324, etc., to hardware scheduler 412. In such a case, the library 1330 stores and returns a number of different embodiments equal to the number of participating parallel processors 115 of the same producer kernel with various block to workgroup mappings, where the mappings are staggered by a multiple of size / number of devices.

[0081] In some embodiments, the command processor 140 delays communication between participating parallel processors 115 or modifies the sequence of workgroup execution to ensure that participating parallel processors 115 are synchronized during reduction operations. The hardware scheduler 412 of each parallel processor 115 keeps a count of the total blocks per workgroup dispatched from the kernel. When the hardware scheduler 412 dispatches n blocks (e.g., chunk_size), it notifies the command processor 140. The command processor 140 then sets a bit to broadcast to the other parallel processors 115. The next set of communications begins when all of the parallel processors 115 have received a total of p bits (p is the number of devices). Thus, the frequency of synchronization can be modified by configuring the value of “n.” If one or more parallel processors 115 are determined to be lagging, the block schedule checker 1202 modifies the workgroup schedule of each parallel processor 115 to maintain synchronization between the devices.

[0082] In some embodiments, the above-described apparatus and techniques are implemented in a system including one or more integrated circuit (IC) devices (also called integrated circuit packages or microchips), such as the processing systems described above with reference to FIGS. 1-13. Electronic design automation (EDA) and computer-aided design (CAD) software tools can be used in the design and manufacture of these IC devices. These design tools are typically represented as one or more software programs. The one or more software programs include code executable by a computer system for operating the computer system to operate on code representing the circuits of one or more IC devices to perform at least a portion of a process for designing or adapting a manufacturing system for producing the circuits. This code may include instructions, data, or a combination of instructions and data. The software instructions representing the design or manufacturing tools are typically stored in a computer-readable storage medium accessible to the computing system. Similarly, code representing one or more stages of the design or manufacture of the IC devices is stored in and accessed from the same or a different computer-readable storage medium.

[0083] A computer-readable storage medium includes any non-transitory storage medium or combination of non-transitory storage media that can be accessed by a computer system during use to provide instructions and / or data to the computer system. Such storage media may include, but are not limited to, optical media (e.g., compact discs (CDs), digital versatile discs (DVDs), Blu-ray discs), magnetic media (e.g., floppy disks, magnetic tape, magnetic hard drives), volatile memory (e.g., random access memory (RAM) or cache), non-volatile memory (e.g., read-only memory (ROM) or flash memory), or micro-electromechanical systems (MEMS)-based storage media. The computer-readable storage medium (e.g., system RAM or ROM) may be internal to the computing system, the computer-readable storage medium (e.g., a magnetic hard drive) may be permanently attached to the computing system, the computer-readable storage medium (e.g., an optical disk or Universal Serial Bus (USB)-based flash memory) may be removably attached to the computing system, or the computer-readable storage medium (e.g., network-accessible storage (NAS)) may be coupled to the computer system via a wired or wireless network.

[0084] In some embodiments, certain aspects of the techniques described above are implemented by one or more processors of a processing system executing software. The software includes one or more sets of executable instructions stored or otherwise tangibly embodied in a non-transitory computer-readable storage medium. The software may include instructions and specific data that, when executed by one or more processors, operate the one or more processors to perform one or more aspects of the techniques described above. The non-transitory computer-readable storage medium may include, for example, a magnetic or optical disk storage device, a solid-state storage device such as flash memory, a cache, a random access memory (RAM), or other non-volatile memory device(s). The executable instructions stored on the non-transitory computer-readable storage medium may be implemented as source code, assembly language code, object code, or other form of instructions that can be interpreted or otherwise executed by one or more processors.

[0085] In addition to the above, it should be noted that not all activities or elements described in the summary description are required, that some of the particular activities or devices may not be required, that one or more additional activities may be performed, and that one or more additional elements may be included. Furthermore, the order in which the activities are listed is not necessarily the order in which they are performed. Also, the concepts have been described with reference to specific embodiments. However, those skilled in the art will recognize that various modifications and variations can be made without departing from the scope of the invention as set forth in the claims. Accordingly, the specification and drawings should be regarded in an illustrative rather than a restrictive sense, and all such modifications are intended to be included within the scope of the present invention.

[0086] Benefits, other advantages, and solutions to problems have been described above with regard to specific embodiments. However, the benefits, advantages, solutions to problems, and features from which any benefit, advantage, or solution may arise or be manifested are not construed as critical, essential, or essential features of any or all claims. Moreover, the specific embodiments described above are illustrative only, since the disclosed invention may be modified and practiced in different, but similar manners apparent to those skilled in the art having the benefit of the teachings herein. No limitations are intended to the details of construction or design herein shown, other than as described in the appended claims. It is therefore apparent that the specific embodiments described above may be altered or modified, and that all such variations are considered within the scope of the disclosed invention. Accordingly, the protection sought herein is set forth in the appended claims.

Claims

1. 1. A method comprising: scheduling, in a command processor, a first subset of one or more workgroups of a first kernel to execute immediately before a second subset of one or more workgroups of a second kernel based on a data dependency between the first subset and the second subset, wherein the second subset of one or more workgroups executes before the first kernel completes execution and consumes data produced by the first subset of the one or more workgroups; method.

2. at runtime, dividing a plurality of workgroups into the first subset of the one or more workgroups and the second subset of the one or more workgroups based on the data dependencies.

10. The method of claim 1.

3. the dividing is based on cache size and locality for a set of one or more computing units executing the one or more workgroups of the second subset; the cache stores data generated by the one or more workgroups of the second subset. The method of claim 2.

4. specifying, by the runtime, a mapping between the first subset of the one or more workgroups of the first kernel and the second subset of the one or more workgroups of the second kernel. The method according to claim 2 or 3.

5. a mapping between the first subset of the one or more workgroups of the first kernel and the second subset of the one or more workgroups of the second kernel is declared in a companion function of each of the first kernel and the second kernel; 10. The method of claim 1.

6. the first subset of workgroups is different from the second subset of workgroups; 10. The method of claim 1.

7. inserting a barrier into a kernel packet to delay execution of the first subset of the one or more workgroups until the second subset of the one or more workgroups has completed execution.

10. The method of claim 1.

8. initializing a counter with the number of workgroups of the second subset of the one or more workgroups that must complete execution before the first subset of the one or more workgroups executes; decrementing the counter in response to a workgroup reaching the barrier. The method of claim 7.

9. generating commands that describe pairing of two or more kernels based on the data dependencies; scheduling the one or more workgroups of the first subset and the one or more workgroups of the second subset for execution on a processor based on the command; 10. The method of claim 1.

10. placing the command and an identifier of a kernel invocation for the one or more workgroups of the first subset and the one or more workgroups of the second subset in a ring buffer; scheduling a workgroup in a command processor based on the command and the identifier; 10. The method of claim 9.

11. A command processor, The command processor scheduling a first subset of one or more workgroups of a first kernel for execution on the parallel processors; scheduling a second subset of one or more workgroups of a second kernel to execute on the parallel processors immediately after execution of the first subset and before the first kernel completes execution based on data dependencies between the first subset and the second subset, wherein the second subset of the one or more workgroups consumes data produced by the first subset of the one or more workgroups; configured to: Command processor.

12. the first subset of workgroups is different from the second subset of workgroups; The command processor of claim 11.

13. configured to schedule the second subset of the one or more workgroups based on a barrier inserted in a kernel packet to delay execution of the second subset of the one or more workgroups until the first subset of the one or more workgroups has completed execution.

13. A command processor according to claim 11 or 12.

14. initializing a counter with the number of workgroups of the second subset of the one or more workgroups that must complete execution before the first subset of the one or more workgroups executes; decrementing the counter in response to a workgroup reaching the barrier; configured to: The command processor of claim 13.

15. generating commands that describe pairing of two or more kernels based on the data dependencies; scheduling the one or more workgroups of the first subset and the one or more workgroups of the second subset for execution on a processor based on the command; configured to: The command processor of claim 11.

16. placing the command and an identifier of a kernel invocation for the one or more workgroups of the first subset and the one or more workgroups of the second subset in a ring buffer; scheduling a workgroup in a command processor based on the command and the identifier; configured to:

16. The command processor of claim 15.

17. 1. A processing system comprising: having parallel processors, The parallel processors include: a command processor configured to schedule a first subset of the one or more workgroups of a first kernel to execute immediately before a second subset of the one or more workgroups of a second kernel; the second subset of the one or more workgroups consumes data produced by the first subset of the one or more workgroups and executes before the first kernel completes execution. Processing system.

18. a plurality of workgroups of the first subset of the one or more workgroups are different from a plurality of workgroups of the second subset of the one or more workgroups; 20. The processing system of claim 17.

19. a runtime configured to insert a barrier into a kernel packet to delay execution of the first subset of the one or more workgroups until the second subset of the one or more workgroups has completed execution.

19. The processing system of claim 17 or 18.

20. The runtime is initializing a counter with the number of workgroups of the second subset of one or more workgroups that must complete execution before the first subset of one or more workgroups executes; decrementing the counter in response to a workgroup reaching the barrier; configured to:

20. The processing system of claim 19.

21. a runtime configured to generate commands describing a pairing of two or more kernels based on data dependencies between work-groups of the two or more kernels; the command processor is configured to schedule the first subset of the one or more workgroups and the second subset of the one or more workgroups for execution on a processor based on the command.

20. The processing system of claim 17.

22. It has a ring buffer, the runtime is configured to place the command and an identifier of a kernel invocation for the first subset of the one or more workgroups and the second subset of the one or more workgroups in the ring buffer; the command processor is configured to schedule a workgroup based on the command and the identifier; 22. The processing system of claim 21.

Images