Cache Blocking for Dispatch

Abstract

The processing system [100] divides successive dispatches [135] of work items into portions [145]. The successive dispatches are separated from one another by barriers [202], [204], each barrier indicating that work items of a previous dispatch must complete execution before work items of a subsequent dispatch can begin execution. In some embodiments, the processing system interleaves execution of a portion of a first dispatch with a portion of a subsequent dispatch that consumes data produced by the first dispatch. The processing system thereby reduces the amount of data written to a local cache [120] by a producer dispatch while preserving data locality for subsequent consumer (or consumer / producer) dispatches, facilitating processing efficiency.

Description

[Background technology]

[0001] Parallel processing units, such as Graphics Processing Units (GPUs) and other multi-threaded processing units, typically implement multiple processing elements (also referred to as processor cores or computational units) that simultaneously execute multiple instances of a single program on multiple data sets. For example, a processing element may implement the Single-Instruction-Multiple-Data (SIMD) protocol to execute the same instruction on multiple data sets simultaneously using multiple processor cores.

[0002] A hierarchical execution model defines a kernel of instructions that is executed using different data sets. For example, dispatches of a grid of work items (also referred to herein as threads or streams), e.g., corresponding to pixels of a display, access data sets stored in the caches of parallel processing units. Some data sets, such as surfaces or textures, are modified and read by successive dispatches. For example, a first dispatch, called a producer dispatch, writes data to the cache, and a subsequent dispatch, called a consumer / producer dispatch, reads and modifies the data written by the producer dispatch. The consumer dispatch reads the data written by the previous dispatch. However, the amount of data that can be stored in a cache is constrained by the size of the cache, and the amount of data generated by a producer dispatch often exceeds the storage capacity of the cache, resulting in "thrashing" of the cache. Such data is often evicted from the cache before it can be read by a subsequent dispatch. Summary of the Invention [Means for solving the problem]

[0003] In one example, the method includes executing a first portion of a first dispatch of the work items that writes data to a cache, the first dispatch including a first plurality of portions. An amount of data written to the cache by the first portion of the first dispatch does not exceed a storage capacity of the cache. In response to the first portion of the first dispatch of the work items completing execution, the method further includes executing a first portion of a second dispatch of the work items that reads the data written to the cache by the first portion of the first dispatch, the second dispatch including a second plurality of portions.

[0004] In some examples, the method further includes executing, in response to the first portion of the second dispatch of the work items completing execution, a first portion of a third dispatch of the work items that reads data written to the cache by at least one of the first portion of the first dispatch and the first portion of the second dispatch. The third dispatch includes a third plurality of portions. In some examples, the method further includes executing, in response to the first portion of the second dispatch of the work items completing execution, a second portion of the first dispatch of the work items that writes the data to the cache.

[0005] In some examples, the method further includes, in response to the second portion of the first dispatch of the work items completing execution, executing a second portion of a second dispatch of the work items that reads data written to the cache by at least one of the first portion of the first dispatch and the second portion of the first dispatch. In some examples, the first portion of the first dispatch of the work items is larger than the first portion of the second dispatch of the work items.

[0006] In some examples, the method further includes dividing the first dispatch of the work items into a first plurality of portions and dividing the second dispatch of the work items into a second plurality of portions, In some examples, the first plurality of portions and the second plurality of portions are of unequal size.

[0007] In another example, the method includes dividing a first dispatch of a work item that writes data to a cache into a first multiple portions, dividing a second dispatch of a work item that reads data written by the first dispatch from the cache into a second multiple portions, and interleaving execution of the first multiple portions with execution of the second multiple portions. In some examples, the method further includes dividing a third dispatch of a work item that reads data written by at least one of the first and second dispatches from the cache into a third multiple portions, and interleaving execution of the third multiple portions with execution of the first multiple portions and the second multiple portions.

[0008] In some examples, the first portion of the first plurality of portions is larger than the first portion of the second plurality of portions. In some examples, an amount of data written to the cache by the first portion of the first plurality of portions does not exceed a storage capacity of the cache. In some examples, the first and second plurality of portions are unequal sizes. In some examples, the interleaving includes executing the first portion of the first plurality of portions and, in response to the first portion of the first plurality of portions completing execution, executing the first portion of the second plurality of portions.

[0009] In another example, a device includes a cache and a processor configured to execute a first portion of a first dispatch of work items that writes data to the cache, the first dispatch including a first plurality of portions, and an amount of data written to the cache by the first portion of the first dispatch does not exceed a storage capacity of the cache. The processor is configured to execute a first portion of a second dispatch of work items that reads the data written to the cache by the first portion of the first dispatch, the second dispatch including a second plurality of portions, in response to the first portion of the first dispatch of work items completing execution.

[0010] In some examples, the processor is further configured to, in response to the first portion of the second dispatch of the work items completing execution, execute a first portion of a third dispatch of the work items that reads data written to the cache by at least one of the first portion of the first dispatch and the first portion of the second dispatch, the third dispatch including a third plurality of portions. The processor is further configured in some examples to, in response to the first portion of the second dispatch of the work items completing execution, execute a second portion of the first dispatch of the work items that writes the data to the cache.

[0011] In some examples, the processor is further configured to execute, in response to the second portion of the first dispatch of the work items completing execution, a second portion of a second dispatch of the work items that reads data written to the cache by at least one of the first portion of the first dispatch and the second portion of the first dispatch. The first portion of the first dispatch of the work items in some examples is larger than the first portion of the second dispatch of the work items.

[0012] In some examples, the processor is further configured to divide the first dispatch of the work items into a first plurality of portions and to divide the second dispatch of the work items into a second plurality of portions, hi some examples, the first plurality of portions and the second plurality of portions are of unequal size.

[0013] The present disclosure may be better understood, and its numerous features and advantages made apparent to those skilled in the art by referencing the accompanying drawings, in which: The use of the same reference numbers in different drawings indicates similar or identical items. [Brief description of the drawings]

[0014] [Figure 1] FIG. 2 is a block diagram of a processing system configured to divide a continuous dispatch into portions and interleave the execution of the portions, according to some embodiments. [Diagram 2] FIG. 2 is a block diagram of a dispatch partitioning and interleaving module that partitions a continuous dispatch into portions and interleaves the execution of the portions by inserting barriers between the partitions, according to some embodiments. [Diagram 3] FIG. 1 illustrates repeated execution of interleaved portions of a continuous dispatch in accordance with some embodiments. [Figure 4] FIG. 13 illustrates the starting locations of interleaved portions of a continuous dispatch according to some embodiments. [Diagram 5] FIG. 13 illustrates the starting locations of interleaved portions of a continuous dispatch according to some embodiments. [Figure 6] FIG. 2 is a flow diagram illustrating a method for dividing a continuous dispatch into portions and interleaving the execution of the portions according to some embodiments. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0015] A processor core of a parallel processing unit often performs dispatches of work items that access the same data set consecutively, such that spatial and / or temporal data locality exists within and between dispatches. To illustrate, a surface, such as a texture, that is attached to a succession of dispatches of work items is read in successive rendering or denoising passes. For example, a first dispatch of a work item corresponding to a pixel that renders an object in a scene generates global illumination, and a second dispatch of the work item generates shadowing or performs filtering. Both the first and second dispatches require the same texture data to map to the pixel. In some cases, the work item of the first dispatch generates data, such as an unordered access view (UAV) buffer or a render target, that is consumed by the work item of the subsequent dispatch as a texture in what is called read-after-write locality.

[0016] In the course of executing instructions, the processor cores of the parallel processing units access data stored in local caches, where accesses initiated by the processor cores are performed relatively quickly and with relatively low energy consumption, or in more remote levels of the memory hierarchy, where accesses initiated by the processor cores are performed relatively slowly and with relatively high energy consumption. Typically, the closer the data is stored in the memory hierarchy to the processor cores, the more quickly and energy-efficiently it is accessed by the processor cores. To improve processing efficiency, some processing systems implement memory management protocols that move data that has been recently requested for access to levels of the memory hierarchy closer to the processor cores and move data that has not been recently accessed to more remote levels of the memory hierarchy, with the expectation that the data will be accessed again by the processor cores in the near future.

[0017] For successive dispatches of work items in which subsequent consumer (or consumer / producer) dispatches consume data generated by previous producer dispatches, the data generated by the producer dispatches can be stored in a local cache accessible by the consumer dispatches to facilitate relatively quick access by the processor cores. However, in many cases, the amount of data generated by the producer dispatches exceeds the storage capacity of the local cache, resulting in eviction of the data to more remote levels of the memory hierarchy, from which the data must be fetched for processing by subsequent consumer dispatches.

[0018] To reduce the amount of data written by a producer dispatch to a local cache while preserving data locality for subsequent consumer (or consumer / producer) dispatches, the processing system divides successive dispatches of a work item into portions. The successive dispatches are separated from each other by barriers, each barrier indicating that a work item of a previous dispatch must complete execution before a work item of a subsequent dispatch can begin execution. In some embodiments, the processing system divides a first producer dispatch (referred to as the first dispatch) into a finite number of portions, with the first portion (having a starting location of (0,0) for a two-dimensional (2-D) dispatch or (0,0,0) for a three-dimensional (3-D) dispatch) being sized such that the data written by the first portion does not exceed the storage capacity of the local (lowest level) cache. In some embodiments, the portions of the first producer dispatch are of equal size. In other embodiments, the portions of the first producer dispatch are of unequal size. For example, a first portion of the first dispatch is a larger size than other portions of the first dispatch.

[0019] The processing system divides the consumer / producer or consumer dispatches (called second dispatches) that immediately follow the first producer dispatch and consume the data produced by the first producer dispatch into a finite number of parts. To ensure coherency so that the consumer dispatches consume the correct (intended) data produced by the previously executed part of the producer dispatch, the processing system inserts barriers between the parts to ensure that the consumer dispatch parts do not execute until the producer dispatch parts have completed execution.

[0020] In some embodiments, the portions of the second dispatch are equal in size. In other embodiments, the portions of the second dispatch are non-uniformly sized. For example, in some embodiments, the processing system sizes a first portion of the second dispatch (having a starting location of (0,0) for a two-dimensional (2-D) dispatch or (0,0,0) for a three-dimensional (3-D) dispatch) to be smaller than the first portion of the first dispatch such that work items in the first portion of the second dispatch can access data generated by adjacent work items in the first portion of the first dispatch to perform, for example, filtering or averaging.

[0021] The processing system divides a set of dependency dispatches into the same number of parts (i.e., blocks or tiles) regardless of the nature of the dependencies (i.e., whether the dependencies are simple or complex). The part size for producer dispatches can be different from the part size for consumer dispatches. However, the number of parts between producer and consumer dispatches remains the same. This allows the processing system to place barriers between the parts in an efficient manner.

[0022] To promote processing efficiency, in some embodiments, the processing system interleaves execution of portions of a first dispatch with portions of subsequent dispatches that consume data generated by the first dispatch. For example, in some embodiments, the processing system executes a first portion of the first dispatch followed by a first portion of a second dispatch followed by a second portion of the first dispatch followed by a second portion of the second dispatch. By interleaving the execution of portions of the dispatches, the processing system balances the workload among the computational units.

[0023] FIG. 1 illustrates a processing system 100 configured to divide a continuous dispatch into portions and interleave the execution of the portions, according to some embodiments. The processing system 100 includes a parallel processing unit 110, such as a graphics processing unit (GPU) for creating visual images intended to be output to a display 175, according to some embodiments. A parallel processor is a processor that can execute a single instruction on multiple data or threads in parallel. Examples of parallel processors include processors such as graphics processing units (GPUs), massively parallel processors, single instruction multiple data (SIMD) architecture processors, and single instruction multiple thread (SIMT) architecture processors for performing graphics, machine intelligence, or computational operations. In some embodiments, the parallel processor is a separate device included as part of a computer. In other embodiments, such as an advanced processor unit, the parallel processor is included in a single device together with a host processor, such as a central processor unit (CPU). The following description uses a graphics processing unit (GPU) for illustrative purposes, but the embodiments described below are applicable to other types of parallel processors.

[0024] The processing system 100 includes a memory 150. Some embodiments of the memory 150 are implemented as dynamic random access memory (DRAM). However, the memory 150 may be implemented using other types of memory, including static random access memory (SRAM), non-volatile RAM, etc. In the illustrated embodiment, the parallel processing unit 110 communicates with the memory 150 over a bus 160. However, some embodiments of the parallel processing unit 110 communicate with the memory 150 via a direct connection or through other buses, bridges, switches, routers, etc. The parallel processing unit 110 executes instructions stored in the memory 150, and the parallel processing unit 110 stores information in the memory 150, such as results of the executed instructions. For example, the memory 150 may store copies of instructions from an application 155 executed by the parallel processing unit 110. Some embodiments of the parallel processing unit 110 include multiple processor cores (called computational units) 115 that independently execute instructions simultaneously or in parallel.

[0025] The processing system 100 is generally configured to execute a set of instructions (e.g., a computer program), such as an application 155, to perform designated tasks for the electronic device. Examples of such tasks include controlling aspects of the operation of the electronic device, displaying information to a user to provide a particular user experience, communicating with other electronic devices, etc. Thus, in different embodiments, the processing system 100 is employed in any of a number of types of electronic devices, such as a desktop computer, a laptop computer, a server, a game console, a tablet, a smartphone, etc. It should be understood that the processing system 100 may include more or less components than those shown in FIG. 1. For example, the processing system 100 may further include one or more input interfaces, non-volatile storage, one or more output interfaces, a network interface, and one or more displays or display interfaces.

[0026] The processing system 100 includes a central processing unit (CPU) 105 for executing instructions. Some embodiments of the CPU 105 include multiple processor cores (not shown for clarity) that independently execute instructions simultaneously or in parallel. The CPU 105 is also connected to a bus 160 and thus communicates with the parallel processing units 110 and memory 150 via the bus 160. The CPU 105 executes instructions, such as program code for an application 155 stored in the memory 150, and the CPU 105 stores information, such as results of executed instructions, in the memory 150. The CPU 105 can also initiate graphics processing by issuing draw calls to the parallel processing units 110. A draw call is a command generated by the CPU 105 and sent to the parallel processing units 110 to instruct the parallel processing units 110 to render an object (or a portion of an object) in a frame. Some embodiments of a draw call include information defining textures, states, shaders, rendering objects, buffers, etc., used by the parallel processing units 110 to render the object or portions thereof. Parallel processing units 110 render objects to generate pixel values ​​that are provided to display 175, which uses the pixel values ​​to display an image representing the rendered objects.

[0027] In some embodiments, each frame to be rendered is processed by the graphics pipeline of the parallel processing unit 110 in multiple passes. For example, during a first pass over the scene's geometry, only the attributes necessary to compute per-pixel lighting are written to the G-buffer. During a second pass, the graphics pipeline outputs only diffuse and specular lighting data. In a frame's third pass through the graphics pipeline, the graphics pipeline reads backlight data and outputs the final per-pixel shading. Thus, in multi-pass rendering, the scene and associated objects of a frame are rendered multiple times. Each time an object is drawn, the graphics pipeline computes additional aspects of the object's appearance and combines the additional aspects with previous results. Each time a frame or an object of a frame is rendered by the graphics pipeline is referred to as a rendering pass.

[0028] An Input / Output (I / O) engine 170 handles input or output operations associated with a display 175 and other elements of the processing system 100, such as a keyboard, a mouse, a printer, an external disk, etc. The I / O engine 170 is coupled to a bus 160 such that the I / O engine 170 communicates with the parallel processing units 110, the memory 150, or the CPU 105. In the illustrated embodiment, the I / O engine 170 is configured to read information stored on an external storage medium 180, such as a compact disc (CD), a digital video disc (DVD), or the like. The external storage medium 180 stores information representing program code used to execute an application, such as a video game. The program code on the external storage medium 180 can be written to the memory 150 to form a copy of instructions to be executed by the parallel processing units 110 or the CPU 105.

[0029] In some embodiments, parallel processing unit 110 implements a graphics pipeline (not shown in FIG. 1 for clarity) that includes multiple stages configured to process different primitives simultaneously in response to draw calls. The stages of the graphics pipeline in parallel processing unit 110 can simultaneously process different primitives generated by an application, such as a video game. When geometry is presented to the graphics pipeline, hardware state settings are selected to define the state of the graphics pipeline. Examples of state include rasterizer state, blend state, depth stencil state, primitive topology type of the presented geometry, and shaders (e.g., vertex shader, domain shader, geometry shader, hull shader, pixel shader, etc.) used to render the scene. Shaders implemented in the graphics pipeline state are represented by corresponding bytecodes. In some cases, information representing the graphics pipeline state is hashed or compressed to provide a more efficient representation of the graphics pipeline state.

[0030] Driver 165 is a computer program that allows higher level graphic computing programs, such as from application 155, to interact with parallel processing unit 110. For example, driver 165 converts standard code received from application 155 into a native format command stream understood by parallel processing unit 110. Driver 165 allows input from application 155 to dictate settings of parallel processing unit 110. Such settings include selection of rendering mode, anti-aliasing control, texture filter control, batch binning control, and deferred pixel shading control.

[0031] To execute a set of commands received from the CPU, parallel processing unit 110 includes a command processor 125, a dispatch unit 130, multiple computation units 115, and a cache 120. Command processor 125 is a set of hardware configured to receive commands from CPU 105 and prepare the received commands for processing. For example, in some embodiments, command processor 125 performs operations such as buffering received commands, organizing received commands into one or more queues for processing, decoding or otherwise interpreting received commands, etc.

[0032] Dispatch unit 130 is a set of hardware configured to perform dispatch operations in response to dispatch commands (not shown) received from command processor 125. In response to the dispatch commands, dispatch unit 130 accesses a grid of work items and dispatches a subset of the work items (e.g., dispatch 135) to compute units 115 for processing as a wavefront. Unlike draw calls, dispatch 135 can be processed by parallel processing units 110 without going through the graphics pipeline.

[0033] The multiple compute units 115 together perform shading operations for dispatches (e.g., dispatch 135) dispatched by dispatch unit 130. In different embodiments, the compute units 115 perform geometry operations, texture operations, tessellation operations, vertex operations, mesh operations, primitive operations, ray tracing operations, compute operations, and the like, or any combination thereof, based on commands received from command processor 125. In some embodiments, to perform these operations, each of the compute units 115 includes one or more SIMD elements configured to perform the specified operations using work items of the received dispatches.

[0034] Cache 120 stores data for multiple compute units 115. Thus, in the course of executing shader operations, multiple compute units 115 store and retrieve data from cache 120. The data stored and retrieved is based on the particular work item being processed. For example, in some embodiments, each work item in dispatch 135 corresponds to an individual pixel of an image, and cache 120 stores data (e.g., texture values) for each individual pixel or for a subset of the individual pixels included in dispatch 135. In some embodiments, parallel processing unit 110 is associated with a memory hierarchy having multiple cache levels and system memory 150, and cache 120 represents any of the multiple cache levels.

[0035] To facilitate access to cache 120 that exploits data locality between dispatches, parallel processing unit 110 includes dispatch partitioning and interleaving module 140. Dispatch partitioning and interleaving module 140 may be implemented in hardware, firmware, software, or a combination thereof. Dispatch partitioning and interleaving module 140 analyzes the locality of data stored in cache 120 for successive dispatches, such as dispatch 135, and partitions the dispatch into a finite number of portions 145 based on the data locality of the data stored in cache 120 and the storage capacity of cache 120 for storing data generated by work items of application 155. For example, if dispatch partitioning and interleaving module 140 determines that data generated by dispatch 135 will be consumed by a subsequent dispatch (not shown) and will exceed the storage capacity of cache 120, in some embodiments, dispatch partitioning and interleaving module 140 divides dispatch 135 and the subsequent dispatch into a finite number of portions 145. By partitioning dispatch 135 into portions 145 that generate data that fits within cache 120, dispatch partitioning and interleaving module 140 increases the probability that data required for a subsequent access to cache 120 will still be present in cache 120 at the time of the subsequent access, thus increasing the hit rate in cache 120 and reducing the latency between dispatches.

[0036] In some embodiments, the dispatch partition and interleaving module 140 inserts barriers between portions 145 from different dispatches to ensure that all work items of a portion of a producer dispatch complete execution before work items of the portion of a dispatch that consumes data from that portion of the producer dispatch begin execution. The dispatch partition and interleaving module 140 interleaves the execution of portions 145 of a dispatch 135 to balance the workload of the compute units 115 that execute work items of the producer and consumer dispatches 135.

[0037] 2 is a block diagram 200 of a dispatch partition and interleaving module 140 that partitions consecutive dispatches into portions and interleaves execution of the portions by inserting barriers between the partitions, according to some embodiments. In the illustrated example, the dispatch partition and interleaving module 140 receives three dispatches, dispatch-1 210, dispatch-2 220, and dispatch-3 230, that the dispatch partition and interleaving module 140 identifies as having data locality and as generating data that exceeds the capacity of the cache 120. Dispatch-1 210 is separated from dispatch-2 220 by a barrier 202 inserted by the application 155. Similarly, dispatch-2 220 is separated from dispatch-3 230 by a barrier 204 inserted by the application 155. The barriers 202, 204 indicate that the first dispatch must complete execution before execution of the second dispatch can begin.

[0038] The dispatch partition and interleaving module 140 divides each of dispatch-1 210, dispatch-2 220, and dispatch-3 230 into two portions. Thus, the dispatch partition and interleaving module 140 divides dispatch-1 210 into portion 1-0 211 and portion 1-1 212. The dispatch partition and interleaving module 140 divides dispatch-2 220 into portion 2-0 221 and portion 2-1 222, and divides dispatch-3 230 into portion 3-0 231 and portion 3-1 232. In some embodiments, the portions of each dispatch are equal in size, while in other embodiments, the portions of each dispatch are unequal in size.

[0039] The dispatch partition and interleaving module 140 interleaves the portions and inserts barriers between the portions to ensure that a first portion separated from a second portion by a barrier completes execution before the second portion begins execution. In the illustrated example, the dispatch partition and interleaving module 140 interleaves the portions as follows: the first portion to execute is portion 1-0 211, followed by barrier 250. After portion 1-0 211 completes execution, portion 2-0 221 executes. Barrier 252 separates portion 2-0 221 from the next portion, portion 3-0 231. Thus, portion 2-0 221 completes execution before portion 3-0 231 executes. Portion 3-0 231 is separated from the next portion, portion 1-1 212, by barrier 254 such that portion 3-0 231 must complete execution before portion 1-1 212 begins execution. A barrier 256 separates portion 1-1 212 from the next portion, portion 2-1 222, such that portion 1-1 212 must complete execution before portion 2-1 222 executes. The last portion, portion 3-1 232, is separated from portion 2-1 222 by a barrier 258. Thus, portion 2-1 222 must complete execution before portion 3-1 232 executes. By dividing the dispatches 210, 220, 230 into portions of a size that generates data that does not exceed the storage capacity of the cache 120, the dispatch partitioning and interleaving module 140 prevents the dispatches 210, 220, 230 from thrashing the cache 120. Furthermore, by interleaving the portions for execution, the dispatch partition and interleaving module 140 balances the workload across the compute units 115 such that fewer compute units 115 are idle while other compute units 115 are executing work items of their portions of the dispatch 210, 220, 230.

[0040] Figure 3 is a diagram 300 illustrating repeated execution of interleaved portions of consecutive dispatches, according to some embodiments. In the illustrated example, each of the three dispatches 210, 220, 230 of Figure 2 is divided into four portions. Thus, dispatch-1 210 is divided into portion 1-0 211, portion 1-1 212, portion 1-2 213, and portion 1-3 214. Dispatch-2 220 is divided into portion 2-0 221, portion 2-1 222, portion 2-2 223, and portion 2-3 224. Dispatch-3 230 is divided into portion 3-0 231, portion 3-1 232, portion 3-2 233, and portion 3-3 234. Dispatch-1 210 is a producer dispatch that produces data for consumption by subsequent dispatches. Dispatch-2 220 is a consumer / producer dispatch that consumes data produced by dispatch-1 210 and produces data for consumption by subsequent dispatches. Dispatch-3 230 is a consumer dispatch that consumes data produced by previous dispatches 210, 220.

[0041] The portions of dispatches 210, 220, 230 are executed in six iterations starting with iteration 0 302. In iteration 0 302, portion 1-0 211 of producer dispatch-1 210 is executed. In some embodiments, portion 1-0 is sized to be larger than the other portions of producer dispatch-1 210 and larger than portions of consumer / producer dispatch-2 220 and consumer dispatch-3 230. In iteration 1 304, portion 1-1 212 of producer dispatch-1 210 and portion 2-0 221 of consumer / producer dispatch-2 220 are executed. Since portion 2-0 221 consumes data produced by portion 1-0 211, portion 2-0 221 is executed in iteration 1 304 after execution of portion 1-0 211 is completed in iteration 0 302.

[0042] In iteration 2 306, part 1-2 213 of producer dispatch-1 210, part 2-1 222 of consumer / producer dispatch-2 220, and part 3-0 231 of consumer dispatch-3 230 are executed. Since part 2-1 222 consumes data produced by part 1-1 212, part 2-1 222 is executed in iteration 2 306 after part 1-1 212 completes execution in iteration 1 304. Similarly, since part 3-0 231 consumes data produced by part 2-0 221, part 3-0 231 is executed in iteration 2 306 after part 2-0 221 executes in iteration 1 304.

[0043] In iteration 3 308, part 1-3 214 of producer dispatch-1 210, part 2-2 223 of consumer / producer dispatch-2 220, and part 3-1 232 of consumer dispatch-3 230 are executed. Since part 2-2 223 consumes data produced by part 1-2 213, part 2-2 223 is executed in iteration 3 308 after part 1-2 213 has completed execution in iteration 2 306. Similarly, part 3-2 232 consumes data produced by part 2-1 222, so part 3-1 232 is executed in iteration 3 308 after part 2-1 222 has executed in iteration 2 306.

[0044] In iteration 4 310, part 2-3 224 of consumer / producer dispatch-2 220 and part 3-2 233 of consumer dispatch-3 230 are executed. Since part 2-3 224 consumes data produced by part 1-3 214, part 2-3 224 is executed in iteration 4 310 after part 1-3 214 completes execution in iteration 3 308. Similarly, since part 3-2 233 consumes data produced by part 2-2 223, part 3-2 233 is executed in iteration 4 310 after part 2-2 223 executes in iteration 3 308.

[0045] In iteration 5 312, portion 3-3 234 of consumer dispatch-3 230 is executed. Since portion 3-3 234 consumes data produced by portion 2-3 234, portion 3-3 234 is executed in iteration 5 312 after execution of portion 2-3 224 in iteration 4 310. By interleaving and executing multiple portions in parallel, dispatch partition and interleaving module 140 mitigates any performance penalty incurred by inserting barriers (e.g., barriers 250, 252, 254, 256, 258) between portions.

[0046] 4 is a diagram 400 illustrating the relative starting positions of interleaved portions of consecutive dispatches, according to some embodiments. In the illustrated embodiment, referred to as a "simple mapping," the portions of dispatches 210, 220, 230 are of equal size and share the same starting positions. For example, in the illustrated embodiment, portions 1-0 211, 2-0 221, and 3-0 231 use the same block dimensions and starting positions.

[0047] At time T1, portions 1-0 211 execute with a starting location in the upper left corner of the work-item grid. At time T2, portions 1-0 211 and 1-1 212 execute. Part 1-1 212 has a starting location in the top center of the work-item grid, starting where portion 1-0 211 ends on the horizontal axis. Because portions 1-0 211 and 1-1 212 are both portions of dispatch-1 210, there is no barrier between them and they can have overlapping execution. At time T3, portions 2-0 221 and 1-1 212 execute, with portion 2-0 221's starting location in the upper left corner of the work-item grid. Part 2-0 221 has the same size and starting location as portion 1-0 211, which produces the data that portion 2-0 221 consumes. Because part 2-0 221 does not consume data generated by part 1-1 212, part 2-0 221 and part 1-1 212 can have overlapping executions.

[0048] At time T4, portions 2-0 221, 1-1 212, and 1-2 213 are executed. Part 1-2 213 has a start location left of center of the grid of work items, starting where part 1-0 211 ends on the vertical axis, and ending at the bottom of the grid of work items. At time T5, portions 2-0 221, 2-1 222, and 1-2 213 are executed. Part 2-1 222 has the same size and start location as part 1-1 212, which produces the data that part 2-1 222 consumes. At time T6, portions 3-0 231, 2-1 222, and 1-2 213 are executed. Part 3-0 231 has the same size and start location as part 2-0 221, which produces the data that part 3-0 231 consumes.

[0049] At time T7, portions 3-0 231, 2-1 222, 1-2 213, and 1-3 214 are executed. Portion 1-3 214 has a start location in the center of the grid of work items, starting where portion 1-2 213 ends on the horizontal axis and portion 2-1 222 ends on the vertical axis, and ending at the bottom right of the grid of work items. At time T8, portions 3-0 231, 2-1 222, 2-2 223, and 1-3 214 are executed. Portion 2-2 223 has the same size and start location as portion 1-2 213, which generates the data that portion 2-2 223 consumes. At time T9, portions 3-0 231, 3-1 232, 2-2 223, and 1-3 214 are executed. Part 3-1 232 has the same size and starting location as part 2-1 222, which produces the data that part 3-1 232 consumes.

[0050] At time T10, portions 3-1 231, 3-1 232, 2-2 223, 2-3 224 are executed. Portion 2-3 224 has the same size and starting position as portion 1-3 214 which generates the data consumed by portion 2-3 224. At time T11, portions 3-0 231, 3-1 232, 3-2 233, 2-3 224 are executed. Portion 3-2 233 has the same size and starting position as portion 2-2 223 which generates the data consumed by portion 3-2 233. At time T12, portions 3-0 231, 3-1 232, 3-2 233, 3-3 234 are executed. Portion 3-3 234 has the same size and starting position as portion 2-3 224 which generates the data consumed by portion 3-3 234.

[0051] 5 is a diagram 400 illustrating the relative starting positions of interleaved portions of consecutive dispatches, according to some embodiments. In the illustrated embodiment, the corresponding portions of dispatches 210, 220, 230 are not all evenly sized and do not always share the same starting position. In the illustrated example, dispatch-3 230 is a consumer dispatch, so dispatch partition and interleaving module 140 divides dispatch-3 230 into four equally sized portions, portions 3-0 231, 3-1 232, 3-2 233, 3-3 234, with the same starting positions as the simple mapping scenario shown in FIG. 4. The portion of dispatch-2 220 that starts at the first row and column of the grid of work items is wider and taller than the corresponding portion of dispatch-3 230, and the portion of dispatch-1 210 that starts at the first row and column of the grid of work items is wider and taller than the corresponding portion of dispatch-2 220.

[0052] In some embodiments, for the remaining rows and columns, the dimensions of the portions of dispatch-1 210 and dispatch-2 220 are identical to the portions of dispatch-3 230, except for the last row and column, where the dimensions of the portions of dispatch-1 210 and dispatch-2 220 are clamped to the dimensions of dispatch-1 210 and dispatch-2 220, respectively. The dispatch partition and interleaving module 140 calculates the starting positions of the remaining portions of dispatch-1 210 and dispatch-2 220 (i.e., portions 1-1 212, 1-2 213, 1-3 214, 2-1 222, 2-2 223, 2-3 234) by adding to the starting positions the amount of work items (thread groups) that were added to the dimensions of the first row and first column of the portions of dispatch-1 210 and dispatch-2 220. By sizing portions of a producer dispatch to be larger than corresponding portions of a consumer dispatch and offsetting the starting positions of the portions, the dispatch partition and interleaving module 140 allows work items of a consumer dispatch to access data generated by adjacent work items of a producer dispatch and can therefore perform operations such as filtering or averaging.

[0053] At time T1, portions 1-0 211 execute with a starting location in the upper left corner of the work-item grid. At time T2, portions 1-0 211 and 1-1 212 execute. Part 1-1 212 has a starting location to the right of the top center of the work-item grid, starting where portion 1-0 211 ends on the horizontal axis. Because portions 1-0 211 and 1-1 212 are both portions of dispatch-1 210, there is no barrier between them and they can have overlapping execution. At time T3, portions 2-0 221 and 1-1 212 execute, with portion 2-0 221's starting location in the upper left corner of the work-item grid. Part 2-0 221 is smaller than part 1-0 211, which generates the data that part 2-0 221 consumes, allowing work items in part 2-0 to perform operations on adjacent data of corresponding work items in part 1-0 211. Because part 2-0 221 does not consume data generated by part 1-1 212, part 2-0 221 and part 1-1 212 can have overlapping execution.

[0054] At time T4, portions 2-0 221, 1-1 212, and 1-2 213 are executed. Part 1-2 213 has a starting location at the bottom left of the work-item grid, starting where part 1-0 211 ends on the vertical axis, and ending at the bottom of the work-item grid. At time T5, portions 2-0 221, 2-1 222, and 1-2 213 are executed. Part 2-1 222 overlaps partially with parts 1-0 211 and 1-1 212. Part 2-1 222 is shorter and wider than part 1-1 212, which generates data that part 2-1 222 consumes. At time T6, portions 3-0 231, 2-1 222, and 1-2 213 are executed. Portion 3-0 231 has the same starting position as portion 2-0 221 which generates the data that portion 3-0 231 consumes, and is smaller than portion 2-0 221.

[0055] At time T7, portions 3-0 231, 2-1 222, 1-2 213, and 1-3 214 are executed. Portion 1-3 214 has a starting location in the lower right corner of the work-item grid, starting where portion 1-2 213 ends on the horizontal axis and portion 2-1 222 ends on the vertical axis, and ending at the bottom right of the work-item grid. At time T8, portions 3-0 231, 2-1 222, 2-2 223, and 1-3 214 are executed. Portion 2-2 223 is taller and narrower than portion 1-2 213, which generates data that portion 2-2 223 consumes. At time T9, portions 3-0 231, 3-1 232, 2-2 223, and 1-3 214 are executed. Portion 3-1 232 is wider and shorter than portion 2-1 222 which produces the data that portion 3-1 232 consumes.

[0056] At time T10, portions 3-1 231, 3-1 232, 2-2 223, and 2-3 224 are executed. Portion 2-3 224 is higher and wider than portion 1-3 214, which generates data consumed by portion 2-3 224. At time T11, portions 3-0 231, 3-1 232, 3-2 233, and 2-3 224 are executed. Portion 3-2 233 is higher and narrower than portion 2-2 223, which generates data consumed by portion 3-2 233. At time T12, portions 3-0 231, 3-1 232, 3-2 233, and 3-3 234 are executed. Portion 3-3 234 is higher and wider than portion 2-3 224, which generates data consumed by portion 3-3 234. As mentioned earlier, portions 3-0 231, 3-1 232, 3-2 233, 3-3 234 are of equal size as are their source dispatches, and dispatch-3 230 is a consumer dispatch that does not produce data for subsequent dispatches.

[0057] 6 is a flow diagram illustrating a method 600 for partitioning a continuous dispatch into portions and interleaving execution of the portions, according to some embodiments. In some embodiments, the method 600 is performed by a processing system, such as the processing system 100 shown in FIG. 1. At block 602, the dispatch partition and interleaving module 140 determines that a producer dispatch of a continuous dispatch having consumer / producer dependencies generates data that exceeds the storage capacity of the cache 120. The dispatch partition and interleaving module 140 partitions a producer dispatch (e.g., dispatch-1 210) into portions (e.g., portions 1-0 211, 1-1 212, 1-2 213, 1-3 214). In some embodiments, the portions have equal sizes, while in other embodiments, the portions are unequal sizes, with a first portion (e.g., portion 1-0 211) being larger than the remaining portions. At block 604, the dispatch partition and interleaving module 140 divides a consumer / producer dispatch (e.g., dispatch-2 220) into portions (e.g., portions 2-0 221, 2-1 222, 2-2 223, 2-3 224). In some embodiments, the portions have equal sizes, while in other embodiments, the portions are unequal sizes, with a first portion (e.g., portion 2-0 221) being smaller than the corresponding portion 1-0 211 that generates the data consumed by portion 2-0 221.

[0058] At block 606, the dispatch partition and interleaving module 140 divides a consumer dispatch (e.g., dispatch-3 230) into portions (e.g., portions 3-0 231, 3-1 232, 3-2 233, 3-3 234). In some embodiments, the portions have equal sizes, while in other embodiments, the portions are unequal sizes, with a second portion (e.g., portion 3-0 231) being smaller than the corresponding portion 1-0 221 that generates the data consumed by portion 3-0 231. At block 608, the dispatch partition and interleaving module 140 determines the starting location of each portion of each dispatch. In embodiments where the portions are of equal size, each corresponding portion of each dispatch has the same starting location (i.e., portions 1-0 211, 2-0 221, and 3-0 231 have the same starting location, portions 1-1 212, 2-1 222, and 3-1 232 have the same starting location, portions 1-2 213, 2-2 223, and 3-2 233 have the same starting location, and portions 1-3 214, 2-3 224, and 3-3 234 have the same starting location). In embodiments where the portions are of unequal size, each corresponding portion of each consumer / producer or consumer dispatch other than the first portion of each dispatch has a starting location that is offset from the starting location of the producer dispatch that consumes the data.

[0059] At block 610, the dispatch partition and interleaving module 140 interleaves the execution of the producer, producer / consumer, and consumer dispatch portions. For example, the dispatch partition and interleaving module 140 schedules a first portion (portion 1-0 211) of dispatch-1 210 for execution on one or more of the compute units 115. In response to portion 1-0 211 completing execution, the dispatch partition and interleaving module 140 schedules a first portion (portion 2-0 221) of dispatch-2 220 for execution on one or more of the compute units 115. In response to portion 2-0 221 completing execution, dispatch partition and interleaving module 140 schedules a first portion (portion 3-0 231) of dispatch-3 230 for execution on one or more of the compute units 115. In response to portion 3-0 231 completing execution, dispatch partition and interleaving module 140 schedules a second portion (portion 1-1 212) of dispatch-1 210 for execution on one or more of the compute units 115. Dispatch partition and interleaving module 140 continues scheduling portions of the dispatch, alternating between dispatching each portion, until the last consumer dispatch has completed execution.

[0060] In some embodiments, the above apparatus and techniques are implemented in a system that includes one or more integrated circuit (IC) devices (also referred to as integrated circuit packages or microchips), such as the processing systems described above with reference to FIGS. 1-5. Electronic design automation (EDA) and computer aided design (CAD) software tools can be used in the design and manufacture of these IC devices. These design tools are typically represented as one or more software programs. The one or more software programs include code executable by a computer system for operating the computer system to operate on code representing the circuits of one or more IC devices to perform at least a portion of a process for designing or adapting a manufacturing system for manufacturing the circuits. This code may include instructions, data, or a combination of instructions and data. The software instructions representing the design tool or manufacturing tool are typically stored in a computer readable storage medium accessible to the computing system. Similarly, code representing one or more stages of the design or manufacture of the IC device is stored in and accessed from the same computer readable storage medium or a different computer readable storage medium.

[0061] A computer-readable storage medium includes any non-transitory storage medium or combination of non-transitory storage media that can be accessed by a computer system during use to provide instructions and / or data to the computer system. Such storage media may include, but are not limited to, optical media (e.g., compact discs (CDs), digital versatile discs (DVDs), Blu-ray discs), magnetic media (e.g., floppy disks, magnetic tape, magnetic hard drives), volatile memory (e.g., random access memory (RAM) or cache), non-volatile memory (e.g., read-only memory (ROM) or flash memory), or micro-electromechanical systems (MEMS) based storage media. The computer-readable storage medium (e.g., system RAM or ROM) may be internal to the computing system, the computer-readable storage medium (e.g., a magnetic hard drive) may be permanently attached to the computing system, the computer-readable storage medium (e.g., an optical disk or Universal Serial Bus (USB)-based flash memory) may be removably attached to the computing system, or the computer-readable storage medium (e.g., network-accessible storage (NAS)) may be coupled to the computer system via a wired or wireless network.

[0062] In some embodiments, certain aspects of the techniques described above are implemented by one or more processors of a processing system executing software. The software includes one or more sets of executable instructions stored or otherwise tangibly embodied in a non-transitory computer-readable storage medium. The software may include instructions and specific data that, when executed by the one or more processors, operate the one or more processors to perform one or more aspects of the techniques described above. The non-transitory computer-readable storage medium may include, for example, a magnetic or optical disk storage device, a solid-state storage device such as a flash memory, a cache, a random access memory (RAM), or other non-volatile memory device(s), etc. The executable instructions stored in the non-transitory computer-readable storage medium may be implemented as source code, assembly language code, object code, or other form of instructions that can be interpreted or otherwise executed by one or more processors.

[0063] In addition to the above, it should be noted that not all activities or elements described in the summary description are required, some of the specific activities or devices may not be required, one or more additional activities may be performed, and one or more additional elements may be included. Furthermore, the order in which the activities are listed is not necessarily the order in which they are performed. Also, the concepts have been described with reference to specific embodiments. However, those skilled in the art will appreciate that various changes and modifications can be made without departing from the scope of the invention as set forth in the claims. Thus, the specification and drawings should be regarded in an illustrative rather than a restrictive sense, and all such modifications are intended to be included within the scope of the invention.

[0064] Benefits, other advantages, and solutions to problems have been described above with respect to specific embodiments. However, the benefits, advantages, solutions to problems, and features by which any benefit, advantage, or solution may occur or be manifested are not to be construed as critical, essential, or essential features of any or all claims. Moreover, the specific embodiments described above are illustrative only, as the disclosed invention may be modified and practiced in different but similar manners apparent to those skilled in the art having the benefit of the teachings herein. No limitations are intended to the details of construction or design herein shown, other than as set forth in the appended claims. It is therefore apparent that the specific embodiments described above may be altered or modified, and all such variations are considered to be within the scope of the disclosed invention. Accordingly, the protection sought herein is set forth in the appended claims.

Claims

1. 1. A method comprising: executing a first portion of a first dispatch of work items that writes data to a cache, the first dispatch including a first plurality of portions, an amount of data written to the cache by the first portion of the first dispatch does not exceed a storage capacity of the cache; responsive to the first portion of the first dispatch of the work items completing execution, executing a first portion of a second dispatch of work items that reads data written to the cache by the first portion of the first dispatch, the second dispatch including a second plurality of portions. method.

2. responsive to the first portion of the second dispatch of the work items completing execution, executing a first portion of a third dispatch of work items that reads data written to the cache by at least one of the first portion of the first dispatch and the first portion of the second dispatch, the third dispatch including a third plurality of portions.

2. The method of claim 1.

3. executing a second portion of the first dispatch of the work items that writes data to the cache in response to the first portion of the second dispatch of the work items completing execution.

2. The method of claim 1.

4. responsive to the second portion of the first dispatch of the work items completing execution, executing a second portion of the second dispatch of the work items that reads data written to the cache by at least one of the first portion of the first dispatch and the second portion of the first dispatch. The method of claim 3.

5. the first portion of the first dispatch of the work items is greater than the first portion of the second dispatch of the work items; 2. The method of claim 1.

6. Dividing the first dispatch of the work items into the first plurality of portions; dividing the second dispatch of the work items into the second plurality of portions.

2. The method of claim 1.

7. the first plurality of portions and the second plurality of portions are unequal in size.

2. The method of claim 1.

8. 1. A method comprising: Dividing a first dispatch of work items that write data to a cache into a first plurality of portions; splitting a second dispatch of work items that reads data written by the first dispatch from the cache into a second plurality of portions; and interleaving execution of the first plurality of portions with execution of the second plurality of portions. method.

9. dividing a third dispatch of work items into a third plurality of portions that reads data from the cache written by at least one of the first dispatch and the second dispatch; and interleaving execution of the third plurality of portions with execution of the first plurality of portions and execution of the second plurality of portions.

9. The method of claim 8.

10. a first portion of the first plurality of portions is larger than a first portion of the second plurality of portions; 10. The method of claim 8 or 9.

11. an amount of data written to the cache by the first portion of the first plurality of portions does not exceed a storage capacity of the cache; The method of claim 10.

12. the first plurality of portions and the second plurality of portions are unequal in size.

9. The method of claim 8.

13. Interleaving is executing a first portion of the first plurality of portions; executing a first portion of the second plurality of portions in response to the first portion of the first plurality of portions completing execution.

9. The method of claim 8.

14. A device, comprising: Cache and A processor, The processor, executing a first portion of a first dispatch of work items that writes data to the cache, the first dispatch including a first plurality of portions, an amount of data written to the cache by the first portion of the first dispatch does not exceed a storage capacity of the cache; responsive to the first portion of the first dispatch of the work items completing execution, executing a first portion of a second dispatch of work items that reads data written to the cache by the first portion of the first dispatch, the second dispatch including a second plurality of portions; [0023] device.

15. The processor, responsive to the first portion of the second dispatch of the work items completing execution, executing a first portion of a third dispatch of work items that reads data written to the cache by at least one of the first portion of the first dispatch and the first portion of the second dispatch, the third dispatch including a third plurality of portions. The device of claim 14.

16. The processor, and configured to execute a second portion of the first dispatch of the work items that writes data to the cache in response to the first portion of the second dispatch of the work items completing execution. The device of claim 14.

17. The processor, in response to the second portion of the first dispatch of the work items completing execution, executing a second portion of the second dispatch of the work items, the second portion reading data written to the cache by at least one of the first portion of the first dispatch and the second portion of the first dispatch.

17. The device of claim 16.

18. the first portion of the first dispatch of the work items is greater than the first portion of the second dispatch of the work items; The device of claim 14.

19. The processor, Dividing the first dispatch of the work items into the first plurality of portions; Dividing the second dispatch of the work items into the second plurality of portions; [0023] The device of claim 14.

20. the first plurality of portions and the second plurality of portions are unequal in size. The device of claim 14.

Images