Multi-phase thread scheduling

Abstract

Techniques related to multi-stage thread scheduling are disclosed. In some embodiments, a processor circuit includes a plurality of lane pipelines for a plurality of lanes and a plurality of execution pipelines shared by the lane pipelines and configured to perform different types of operations provided by the lane pipelines. A first scheduler circuit can arbitrate among threads to assign threads to lanes. A second scheduler circuit can arbitrate among lanes to assign operations from a given lane to a given execution pipeline. The execution pipeline can provide backpressure information to the first scheduler circuit based on an execution state, and the first scheduler circuit can adjust a priority of threads for assignment to a lane based on the backpressure information. The disclosed techniques can reduce lane contention and starvation for execution resources.

Description

BACKGROUND TECHNICAL FIELD

[0002] The present disclosure relates generally to computer processors, and more specifically to thread scheduling.

[0003] Related Art

[0004] Computer processors often execute a large number of threads, sometimes in parallel. For example, graphics processors often operate on thread groups having hundreds or thousands of threads, which can be grouped into single instruction multiple data (SIMD) groups for parallel processing. A graphics processor can implement a large number of shader pipelines, with various execution circuitry to perform operations (e.g., integer operations, floating point operations, load / store operations, sample operations, etc.). Mapping threads / SIMD groups to execution resources in an efficient manner can improve performance, reduce power consumption, reduce chip area for a given level of performance, etc. BRIEF DESCRIPTION OF DRAWINGS

[0005] FIG. 1A is a diagram illustrating an overview of an example graphics processing operation in accordance with some embodiments.

[0006] FIG. 1B is a block diagram illustrating an example graphics unit in accordance with some embodiments.

[0007] FIG. 2 is a block diagram illustrating an example pipeline circuit having two-stage scheduling in accordance with some embodiments.

[0008] FIG. 3 is a block diagram illustrating an example shader that can include a pipeline in accordance with some embodiments. FIG. 2

[0009] FIG. 4 is a block diagram illustrating an example pipeline having hint-based scheduling in accordance with some embodiments.

[0010] FIG. 5 is a block diagram illustrating an example pipeline configured to handle convertible barriers in accordance with some embodiments.

[0011] FIG. 6 is a block diagram illustrating an example barrier manager in accordance with some embodiments.

[0012] FIG. 7A and FIG. 7B is a diagram illustrating an example instruction sequence having lane stall and lane stall disable in accordance with some embodiments.

[0013] FIG. 8 is a flow diagram illustrating an example method for handling preemption notifications in accordance with some embodiments. ​

[0014] FIG. 9A and FIG. 9B is a block diagram illustrating example circuitry associated with cache flush / invalidation for context switching, in accordance with some embodiments.

[0015] FIG. 10 is a block diagram illustrating example control circuitry configured to process page information for context switching, in accordance with some embodiments.

[0016] FIG. 11 is a block diagram illustrating example pipeline configured to stall a lane based on an instruction cache miss, in accordance with some embodiments.

[0017] FIG. 12 is a block diagram illustrating an example two-level instruction cache implementation, in accordance with some embodiments.

[0018] FIG. 13 is an example state diagram for instruction cache control, in accordance with some embodiments.

[0019] FIG. 14 to FIG. 17 is a flow diagram illustrating an example method, in accordance with some embodiments.

[0020] FIG. 18 is a block diagram illustrating an example computing device, in accordance with some embodiments.

[0021] FIG. 19 is a diagram illustrating an example application of the disclosed systems and devices, in accordance with some embodiments.

[0022] FIG. 20 is a block diagram illustrating an example computer-readable medium storing circuit design information, in accordance with some embodiments. DETAILED DESCRIPTION

[0023] As discussed above, mapping thread / SIMD groups to execution resources in an efficient manner can improve performance, reduce power consumption, reduce chip area for a given performance level, or some combination thereof. In various disclosed embodiments, data path units are shared by multiple SIMD groups. For example, scheduling circuitry can arbitrate among multiple SIMD groups to utilize a particular floating point unit. Sharing data path blocks can provide efficiency in terms of circuit area and power consumption, but scheduling and arbitration can be particularly important. For example, saturating a data path unit can cause work to stall, while idling a data path unit can also degrade performance. Additionally, as discussed in detail below, certain operations can have different latencies depending on execution conditions. For example, an instruction that misses in an instruction cache or a data cache can have substantially longer latency than an instruction that hits in both.

[0024] In the disclosed implementations, a two-stage scheduling technique first arbitrates among threads to assign threads to lanes, and then arbitrates among lanes to assign instructions from the lanes to execution pipelines (e.g., ALUs). Thus, the overall pipeline can include multiple lane pipelines (which can include stages such as decode, hazard control, barrier check, instruction cache check, etc.) from which the control circuitry maps instructions to multiple execution pipelines (e.g., floating point pipelines, integer pipelines, load pipelines, store pipelines, etc.). This sharing of execution resources can meet performance goals while reducing chip area (relative to using dedicated execution circuitry for each thread).

[0025] In some implementations, the first-stage scheduling circuitry can utilize compiler hints and backpressure information from the execution pipelines to adjust its priority values. The compiler hints can indicate which execution pipelines are targets for a given thread. The disclosed techniques can advantageously suppress threads that target busy execution pipelines or prioritize threads that target unused execution pipelines.

[0026] In some implementations, the control circuitry can determine whether to stall a thread in a lane or to deactivate the thread from the lane (e.g., to send it back to the first-stage scheduler) based on instruction status information from the execution pipelines. For example, the control circuitry can determine to stall a thread with a cache hit instruction or to deactivate a thread with a cache miss instruction. This can advantageously reduce blocking of threads with long-running instructions.

[0027] In some implementations, architectural registers such as general purpose registers (GPRs), special registers, stack registers, etc. are memory backed. In these implementations, the control circuitry can flush and invalidate a cache line for an architectural register when a context switch occurs, and save a pointer to the memory page targeted by the flush. This can allow for correct retrieval of register data at context restoration. More generally, the control circuitry can move data from a memory structure marked by a private memory identifier to a level of a memory hierarchy marked by a system memory address.

[0028] In some implementations, the control circuitry can determine whether to deactivate a thread from a lane based on an instruction cache miss. The control circuitry can implement multiple types of instruction cache requests, including a request type that disallows deactivation of the corresponding thread in order to provide forward progress. This can advantageously reduce blocking of threads that miss one or more instruction caches while still providing forward progress.

[0029] Graphics Processing Overview

[0030] Reference FIG. 1A, a flowchart illustrating an example processing flow 100 for processing graphics data is shown. In some embodiments, a transform and lighting procedure 110 can involve processing lighting information for vertices received from an application based on defined light source positions, reflectivity, etc., assembling vertices into polygons (e.g., triangles), and transforming the polygons to the correct size and orientation based on positioning in three-dimensional space. A clipping procedure 115 can involve discarding polygons or vertices that are outside of a viewable area. A rasterization procedure 120 can involve defining fragments within each polygon and assigning an initial color value for each fragment based on, for example, texture coordinates for polygon vertices. Fragments can specify properties for pixels they overlap, but actual pixel properties can be determined based on combining multiple fragments (e.g., in a frame buffer), ignoring one or more fragments (e.g., if they are covered by other objects), or both. A shading procedure 130 can involve altering pixel components based on lighting, shading, bump mapping, translucency, etc. Shaded pixels can be assembled in a frame buffer 135. Modern GPUs often include programmable shaders that allow application developers to customize the shading and other processing procedures. Thus, in various embodiments, FIG. 1A Example elements of FIG. 1 can be performed in various orders, in parallel, or omitted. Additional processing procedures can also be implemented.

[0031] Referring now to FIG. 1B , a simplified block diagram illustrating a graphics unit 150 is shown, in accordance with some embodiments. In the illustrated embodiment, the graphics unit 150 includes a programmable shader 160, a vertex pipe 185, a fragment pipe 175, a texture processing unit (TPU) 165, an image write buffer 170, and a memory interface 180. In some embodiments, the graphics unit 150 is configured to process both vertex data and fragment data using the programmable shader 160, which can be configured to process graphics data in parallel using multiple execution pipelines or instances.

[0032] In the illustrated embodiment, the vertex pipe 185 can include various fixed function hardware configured to process vertex data. The vertex pipe 185 can be configured to communicate with the programmable shader 160 in order to coordinate vertex processing. In the illustrated embodiment, the vertex pipe 185 is configured to send processed data to the fragment pipe 175 or the programmable shader 160 for further processing.

[0033] In the illustrated embodiment, the fragment pipe 175 can include various fixed function hardware configured to process pixel data. The fragment pipe 175 can be configured to communicate with the programmable shader 160 in order to coordinate fragment processing. The fragment pipe 175 can be configured to perform rasterization on polygons from the vertex pipe 185 or the programmable shader 160 to generate fragment data. The vertex pipe 185 and the fragment pipe 175 can be coupled to the memory interface 180 (coupling not shown) in order to access graphics data.

[0034] In the illustrated embodiment, programmable shader 160 is configured to receive vertex data from vertex pipe 185 and fragment data from fragment pipe 175 and TPU 165. Programmable shader 160 can be configured to perform vertex processing tasks on the vertex data, which can include various transformations and adjustments of the vertex data. For example, in the illustrated embodiment, programmable shader 160 is also configured to perform fragment processing tasks on the pixel data, such as texturing and shading. Programmable shader 160 can include multiple sets of multiple execution pipelines for processing data in parallel.

[0035] In some embodiments, a programmable shader includes pipelines configured to execute one or more different SIMD groups in parallel. Each pipeline can include various stages configured to operate (such as fetch, decode, issue, execute, etc.) in a given clock cycle. The concept of a processor “pipeline” is well understood and refers to the concept of breaking the “work” of a processor’s execution of an instruction into multiple stages. In some embodiments, the decoding, dispatching, execution (i.e., doing), and retirement of an instruction can be examples of different pipeline stages. Many different pipeline architectures can have different ordering of elements / sections. The various pipeline stages perform such steps on an instruction during one or more processor clock cycles, then pass the instruction or operations associated with the instruction to other stages for further processing.

[0036] The term “SIMD group” is intended to be interpreted according to its well-known meaning, which includes a group of threads for which processing hardware processes the same instruction in parallel using different input data for different threads. A SIMD group can also be referred to as a SIMT (single instruction multiple thread group), single instruction parallel thread (SIPT), or lane stack thread. Various types of computer processors can include multiple sets of pipelines configured to execute SIMD instructions. For example, graphics processors often include programmable shader cores configured to execute instructions for a group of related threads in SIMD fashion. Other examples of names that can be used for a SIMD group include: wavefront, clique, or warp. A SIMD group can be part of a larger group of threads, which can be split into multiple SIMD groups based on the parallel processing capabilities of the computer. In some embodiments, each thread is assigned to a hardware pipeline (which can be referred to as a “lane”) that fetches operands for that thread and performs the specified operation in parallel with other pipelines for the group of threads. Note that a processor can have a large number of pipelines such that multiple separate SIMD groups can also execute in parallel. In some embodiments, each thread has private operand storage, e.g., in a register file. Thus, reading a particular register from the register file can provide a version of the register for each thread in the SIMD group.

[0037] As used herein, the term "thread" includes its meaning well known in the art and refers to a sequence of program instructions that can be scheduled for execution independently of other threads. Multiple threads can be included in a SIMD group to be executed in lockstep. Multiple threads can be included in a task or process, which can correspond to a computer program. Threads of a given task can or can not share resources such as registers and memory. Thus, a context switch can or can not be performed when switching between threads of the same task.

[0038] In some embodiments, multiple programmable shader units 160 are included in the GPU. In these embodiments, global control circuitry can assign work to different sub- portions of the GPU, which in turn can assign work to shader cores for processing by the shader pipelines.

[0039] In the illustrated embodiment, TPU 165 is configured to schedule fragment processing tasks from programmable shaders 160. In some embodiments, TPU 165 is configured to pre-fetch texture data and assign initial colors to fragments for further processing by programmable shaders 160 (e.g., via memory interface 180). TPU 165 can be configured to provide fragment components, for example, in normalized integer format or floating point format. In some embodiments, TPU 165 is configured to provide fragments in groups of four ("fragment quads") in 2x2 format, which are processed by a group of four execution pipelines in programmable shaders 160.

[0040] In some embodiments, image write buffer 170 is configured to store processed tiles of an image and can perform operations on the rendered image before it is transferred for display or to memory for storage. In some embodiments, graphics unit 150 is configured to perform tile-based deferred rendering (TBDR). In tile-based rendering, different portions of screen space (e.g., squares or rectangles of pixels) can be processed separately. In various embodiments, memory interface 180 can facilitate communication with one or more of various memory hierarchies.

[0041] As discussed above, graphics processors typically include specialized circuitry configured to perform certain graphics processing operations requested by a computing system. For example, this can include fixed function vertex processing circuitry, pixel processing circuitry, or texture sampling circuitry. Graphics processors can also perform non-graphics computing tasks that can use GPU shader cores but can not use fixed function graphics hardware. As one example, machine learning workloads (which can include inference, training, or both) are often assigned to GPUs due to the parallel processing capabilities of GPUs. Thus, a compute kernel executed by a GPU can include program instructions that specify a machine learning task such as implementing a neural network layer or other aspect of a machine learning model to be performed by a GPU shader. In some cases, non-graphics workloads can also use specialized graphics circuitry, e.g., for purposes other than originally intended.

[0042] Additionally, in other embodiments, various circuitry and techniques discussed herein with reference to graphics processors can be implemented in other types of processors. Other types of processors can include general purpose processors such as CPUs or machine learning or artificial intelligence accelerators with specialized parallel processing capabilities. These other types of processors can not be configured to execute graphics instructions or perform graphics operations. For example, other types of processors can not include fixed function hardware included in typical GPUs. Machine learning accelerators can include specialized hardware for certain operations such as implementing a neural network layer or other aspect of a machine learning model. Generally, there can be design tradeoffs between memory requirements, computing capabilities, power consumption, and programmability of machine learning accelerators. Thus, different implementations can focus on different performance goals. Developers can select from multiple potential hardware targets for a given machine learning application, e.g., from general purpose processors, GPUs, and different specialized machine learning accelerators.

[0043] Pipelining Overview with Two-Stage Scheduling

[0044] It is generally desirable to avoid over-saturation or starvation of data path units. In embodiments discussed in detail below, a two-stage scheduling approach first assigns threads to lanes and then assigns lanes to issue instructions to given execution units in a given cycle. Scheduling can take into account backpressure information to incorporate the status of various execution units into the first stage scheduling.

[0045] FIG. 2 is a block diagram illustrating an example pipeline circuit with two-stage scheduling according to some embodiments. In the illustrated example, the pipeline includes stage 1 scheduling circuit 210, lane pipelines 220A-220M, stage 2 scheduling circuit 250, and execution pipelines 260A-260N.

[0046] Stage 1 scheduling circuit 210 can assign incoming SIMD groups to lanes, as shown. Circuit 210 can also be referred to as a first stage scheduler or a SIMD group scheduler. Note that other implementations can not implement SIMD technology, and can assign individual threads to a given lane. In other implementations, any suitable set of one or more threads can be assigned to a given lane.

[0047] Stage 1 scheduling circuit 210 can implement an age-based scheduler, in which older threads are generally prioritized for assignment to lanes. However, as discussed in detail below, this age-based scheme (or any other suitable default scheme) can be modified in certain scenarios.

[0048] In the illustrated implementation, each lane pipeline 220 includes a decode stage 230 and a hazard stall stage 240. Decode stage 230 can identify the nature of an instruction, which can specify which execution pipeline 260 (or set of potential execution pipelines 260) should execute the instruction. Although partial decoding can occur before stage 230, the target pipeline for an instruction can not be known until after stage 230. Thus, when a thread is assigned to a lane, the particular instructions included in the thread can not be considered (except based on compiler hints, as discussed below with reference to FIG. 3

[0049] In the illustrated implementation, hazard stall stage 240 can stall an instruction until any hazards are resolved. Thus, stalled instructions can not qualify for arbitration at stage 2 scheduling circuit 250.

[0050] In the illustrated implementation, stage 2 scheduling circuit 250 is configured to arbitrate between ready instructions from lane pipelines 220 for assignment to execution pipelines 260. Stage 2 scheduling circuit 250 can select multiple instructions for assignment to different execution pipelines in a given cycle.

[0051] Execution pipelines 260 are configured to perform the operations specified by a given instruction and can include various different units, e.g., integer units, sample units, floating point units, load / store units, etc. Some execution pipelines 260 can be different instances of the same type of unit, e.g., multiple integer units. In other implementations, an execution pipeline includes at most one of a given type of unit. Generally, at certain times, there can be more instructions in lane pipelines 220 that target a particular type of execution pipeline 260 than there are available execution pipelines 260 of that particular type. In this scenario, stage 2 scheduling circuit 250 can arbitrate among lane pipelines 220.

[0052] FIG. 3 is an example of a processor that can include​FIG. 2 FIG. 3 is a block diagram of an example shader of the pipeline of FIG. 2. In the illustrated embodiment, shader 160 includes director 305, private memory page allocator 310, token parser 315, tile and thread group manager 320, special register memory 325, SIMD group scheduler 330, lane manager 335, data path block 312, data level 0 (DL0) cache 370, instruction level 0 (IL0) cache 375, instruction level 1 (IL1) cache 376, fabric 380, and unified level 1 (UL1) cache 385.

[0053] Director circuit 305 can provide work to token parser 315 from multiple data masters (e.g., compute data master, vertex data master, and pixel data master). Private memory page allocator 310 can allocate pages for private memory space as requested by token parser 315. Note that elements 305 and 310 can be external to shader 160 and can communicate with multiple shaders 160.

[0054] In some embodiments, token parser 315 is configured to receive work tokens from multiple data masters, form SIMD groups, and interact with allocator 310 to allocate pages for private memory.

[0055] In some embodiments, tile and thread group manager 320 is configured to coordinate execution of SIMD groups within a tile (e.g., for pixel work) or thread group (e.g., for compute work). For example, this can include implementing various types of synchronization.

[0056] In some embodiments, SIMD group scheduler 330 is configured to manage SIMD group-wide state information and identify the highest priority gang ready for execution according to an arbitration scheme. The arbitration scheme can be primarily age-based, but other factors can also be considered. The SIMD group scheduler can correspond to stage 1 scheduling circuit 210.

[0057] In some embodiments, lane manager circuit 335 is configured to fetch and dispatch instructions to instruction scheduler 340. Lane activation and deactivation can be managed, program counters for a given SIMD group can be managed, architectural state can be managed (e.g., access special register memory 325, which can implement SIMD group-wide architectural special registers such as program counters), instructions can be fetched and dispatched. Lane manager 335 can read special register memory 325 when activating a SIMD group into a lane and write to special register memory 325 when deactivating a SIMD group from a lane.

[0058] In some embodiments, data path block 312 is configured to execute dispatched instructions and can include a lane pipeline and a shared execution pipeline. Data path block 312 can be instantiated multiple times in a given GPU. In the illustrated embodiment, data path block 2312 includes an instruction scheduler 340, a pipeline circuit 345 (which can correspond to lane pipeline 220), an operand cache 348, execution units 360 (which can correspond to execution pipeline 260), a writeback circuit 365, a control flow circuit 350, and a barrier manager 355.

[0059] In some embodiments, instruction scheduler 340 is configured to manage execution resources within data path block 312 and schedule individual instruction execution. This can include fine decoding of incoming instructions, sequencing micro-operations, data dependency and hazard detection, managing read operand cache and write buffer circuits, priority-based instruction scheduling, generating read and write requests to DL0 375, generating pipeline control signals, and implementing SIMD group stalls.

[0060] Pipeline 345 can include one or more math pipelines (which can execute, for example, floating point, integer, and iteration instructions), one or more address generator pipelines (for example, for load, store, atomic, sample, and image write instructions), and one or more control flow units (shown separately as control flow circuit 350) configured to execute conditional and branch instructions. Execution units 360 can perform various types of operations for pipeline 345. As shown, operand cache 348 can be the lowest level of operand storage. Writeback stage 365 can write results to DL0. Note that write operations can be post-processed.

[0061] In some embodiments, barrier manager circuit 355 is configured to ensure that data dependencies outside of data path block 312 are maintained. As discussed in detail below, barrier manager 355 can implement a barrier counter for each barrier per SIMD group (e.g., where a non-zero barrier count indicates a significant dependency). Barrier manager 355 can also implement an in-order instruction queue for each lane (referred to as a lane queue) to track the pipelined barriers of committed instructions. In some embodiments, barrier manager 355 can trigger a stall of a lane in certain situations.

[0062] In some embodiments, DL0 cache 370 is configured to cache all or a portion of registers including in thread private memory. In some embodiments, a given DL0 cache 370 is associated with one data path block 312. Instruction scheduler 340 can initiate a tag lookup in DL0 cache 370.

[0063] In some embodiments, IL0 cache 375 is a lowest level instruction cache and is configured to provide instructions to one or more stages of channel manager 335. In some embodiments, IL1 cache 376 is configured to respond to fill requests from IL0 cache 375 and can retrieve instruction data from UL1 cache 385 via fabric 380 for misses. See FIG. 12 Example embodiments of IL0 cache 375 and IL1 cache 376 are discussed in detail below.

[0064] In some embodiments, fabric circuitry 380 is a packet switched network that provides communication between a plurality of shader modules. As some examples of communication via fabric circuitry 380, a cache can access thread private memory, a token resolver 315 can initialize a SIMD group and thread group state stored in UL1 prior to launching the SIMD group, a sample and image write pipeline can read interface private memory, texture processing results can be forwarded to stack registers, vertex circuitry can transmit fetch requests for vertex data, IL1 cache 376 can request IL1 miss data from global memory, global memory can receive eviction and line fill requests, etc.

[0065] In some embodiments, UL1 cache 385 is a unified instruction and data cache configured to store data evicted from IL1 cache 376 and DL0 cache 370. In other embodiments, IL1 cache 376 is a read-only cache that can retrieve data from UL1 cache 385 but does not evict data to IL1.

[0066] Example Hint-Based Scheduling

[0067] It can be challenging to determine what datapath units a given thread is targeted to in the first dispatch stage, e.g., because decoding occurs later. Thus, in some embodiments, the compiler provides hints about a set of execution resources targeted by a given thread. These hints can be encoded in various ways. As one example, a bit for each execution unit can be set or cleared based on whether the execution unit is targeted. The bit can be set based on a threshold number of instructions that target the unit. In other embodiments, multiple bits per target can be used to provide more information about how many times a given execution unit is targeted.

[0068] FIG. 4 is a block diagram illustrating an example pipeline with hint-based dispatch according to some embodiments. The illustrated example is similar to FIG. 2 but with compiler hints and backpressure information provided to stage 1 dispatch circuitry 210.

[0069] In some embodiments, the stage 1 scheduling circuitry can use compiler hints and backpressure information to adjust arbitration. Compiler hints can indicate, for a given thread or SIMD group, the execution pipeline targeted by that SIMD group (the compiler can be able to determine this information at the time of compiling the program, even though runtime instruction decoding can not occur until the lane decode stage 230).

[0070] As shown, the execution pipeline 260 can provide backpressure information to the stage 1 scheduling circuitry. In other embodiments, the stage 2 scheduling circuitry can provide the backpressure information. For example, this information can indicate the status of the buffers for instructions assigned to a given execution pipeline. The backpressure information can be encoded in various suitable formats, such as the number of stall cycles, the number of buffered instructions, information averaged over a window, etc.

[0071] In some embodiments, the stage 1 scheduling circuitry 210 is configured to adjust the priority of incoming SIMD groups for lane assignment. For example, the stage 1 scheduling circuitry 210 can prioritize the SIMD groups that target execution pipelines that are currently starved (e.g., by artificially adjusting the age values of those SIMD groups, using separate flags for those SIMD groups, adjusting the weight of those SIMD groups in the arbitration calculation, etc.). Similarly, the stage 1 scheduling circuitry 210 can deprioritize the SIMD groups that target execution pipelines that are currently busy.

[0072] Example Convertible Fence Technique

[0073] Fences are commonly used to enforce ordering constraints, for example, in a weakly-ordered memory model. As used herein, a "fence" refers to an instruction or operation that causes a processor to enforce an ordering constraint on operations before and after the fence. A fence can also be referred to as a "barrier." In general, a processor can allow out-of-order or weakly-ordered execution with the use of fences to implement additional ordering constraints. A given processor can provide multiple fences for different ordering constraints.

[0074] The time that an instruction waits on a fence can vary substantially, for example, based on the state of the older instruction. For example, if the older instruction is waiting on a cache fill, that wait can be substantially longer than if the older instruction had hit in the data cache. Thus, in some embodiments, the scheduling circuitry determines whether to stall a thread from a lane based on the execution state of one or more older instructions that have been assigned to a data path unit.

[0075] FIG. 5 is a block diagram illustrating an example pipeline configured to handle a convertible fence, according to some embodiments. Control circuitry can convert this type of fence from an initial stop instruction to enforce an ordering to stall a thread that includes instructions from its lane.

[0076] In the illustrated embodiment, the pipeline includes the above-referenced FIG. 2 described elements and the following additional elements: a submission check stage 510 and a barrier manager 355 for each lane. A given operation (e.g., a load operation) can set a barrier, which can be cleared when the load data is returned.

[0077] In the illustrated embodiment, the barrier manager 355 includes a lane queue 550, which is configured to store state information for certain submitted instructions in the execution pipeline 260. The barrier manager can report this information to the barrier / submission check stage 510. In some embodiments, the lane manager 335 is configured to determine whether to continue to stall or to deactivate the SIMD group from the lane based on the barrier at stage 510. Once the instruction progresses through the submission check stage 510, in some embodiments, it cannot be deactivated, but will proceed to stage 2 dispatch circuit 250 for assignment to the execution pipeline.

[0078] For example, if the newer operation that the barrier is waiting on misses in the data cache (e.g., misses in the UL1 cache 385), the lane manager 335 can deactivate the SIMD group and pass it back to stage 1 dispatch. The SIMD group can then eventually be activated back into the lane, and the newer instruction can have completed, such that execution can continue past the barrier. However, if the newer instruction hits in the data cache, the lane manager 335 can continue to stall the SIMD group even though it can not have completed, as it knows that the newer instruction will complete soon. Specifically, the dependency circuit can stall a given operation in response to all of the input operands of the newer operation on which the given operation depends being available in registers or in one or more lower level data caches, but can deactivate the thread in the event that one or more of the inputs in the thread miss in the one or more lower level data caches.

[0079] The lane queue 550 can store information for a subset of instruction types (e.g., those that potentially run long). The lane manager 335 can consider the status of multiple newer instructions when determining whether to deactivate or stall the SIMD group. The status information can indicate the current pipeline stage of a given operation, an indication of the operation type, whether the operation needs a response from a cache (e.g., the UL1 cache 385) before it can continue, whether the operation accesses certain data structures (e.g., a color buffer), whether the operation is setting a barrier, an identifier of the barrier that is set, and whether there are dependencies on older lane queue entries.

[0080] When an instruction is pushed to the tail of a channel queue (which can be a FIFO), the control circuitry can compare the wait barrier associated with the instruction to all valid set barriers set by older instructions in the channel queue. A match can be recorded in the dependency field indexed by the entry setting the barrier. As operations are popped from the channel queue 550, newer instructions can be shifted towards the head and the dependency field can discard one bit with each position (e.g., the head entry can have 0 dependency bits because it cannot depend on any other instruction).

[0081] FIG. 6 is a block diagram illustrating an example barrier manager according to some embodiments. In the illustrated example, the barrier manager 355 includes an active barrier counter (AFC) 610, a hit / miss array 620, a barrier check 630, and a pipelined barrier tracker 640.

[0082] In some embodiments, stalls are classified as short latency, medium latency, and long latency. For short latency stalls, the stage 2 scheduling circuitry 250 can attempt to fill the gap with instructions issued from other channels, with hazards defended by the scoreboard circuitry. For example, short latency stalls can be due to ALU latency. For medium latency stalls, the pipelined barrier tracker 640 can defend against hazards, and the stage 2 scheduling circuitry 250 can attempt to fill the gap with instructions issued from other channels. Medium latency stalls can be due to short accesses outside of the data path block 312, such as write ordering for UL1. For long latency stalls, the stage 2 scheduling circuitry 250 can disable the channel. For example, long latency stalls can include UL1 accesses due to DL0 misses or sample operations. For even longer stalls, work can be context switched out. Convertible barriers can wait to determine whether a stall is long latency or medium latency (e.g., whether a read on which the stall depends is a hit DL0 or requires access to UL1).

[0083] In some embodiments, the active barrier counter 610 is incremented for each long latency transaction and decremented when the long latency transaction is complete. The active barrier counter 610 can include a counter for each barrier of each SIMD group. Note that the type of transaction classified as a long latency transaction can be different in different embodiments. As one example, DL0 370 misses and sample operations can be classified as long latency transactions. The scheduling circuitry can not be able to cover the latency of these transactions by issuing instructions from other channels. In the illustrated embodiment, the counter is incremented and decremented based on information from the stage 2 scheduler.

[0084] In the illustrated embodiment, the pipelined fence tracker 640 is configured to control the active fence counters and stall the lane while waiting for a fence. In some embodiments, the hit / miss array 620 is configured to track whether a given fence counter represents the expected number of decrements due to a cache hit (in which case the lane can be stalled) or a cache miss or a non-pipelined set fence (in which case the lane can be disabled). Thus, the hit / miss array can include, for example, a bit per active fence counter.

[0085] In the illustrated embodiment, the fence check circuit 630 is configured to check for conflicts between waiting fences and set fences. In some embodiments, the fence check 630 is configured to check for uncommitted instructions as well as committed instructions. The fence check circuit 630 can fail the fence check and disable the SIMD lane in response to determining that the waiting fence check matches an active convertible fence from an older instruction.

[0086] The fence manager 355 can maintain a faling fence mask and a conv fence mask that tracks all outstanding convertible set fences for all instructions of the same SIMD lane that are waiting for a hit to UL1 or have not yet communicated their request to UL1. Newer instructions can communicate tag checks and pre-fetch data, while older instructions stall in pipeline stage 510 (which can be the last location at which the instructions can be tagged for disabling). These instructions can stall until a miss is returned (which can disable the lane) or all requests hit (in which case they can continue to stall until their data is available in DL0 370, then proceed to later pipeline stages that can access data in DL0).

[0087] FIG. 6 The circuit of FIG. 6 represents one example implementation of fence operations, but the disclosed stop / disable techniques based on instruction state information can be used with various memory ordering architectures.

[0088] FIG. 7A FIG. 7 is a diagram illustrating an example instruction sequence in which a lane is stalled, in accordance with some embodiments, while FIG. 7B FIG. 8 is a diagram illustrating an example instruction sequence in which a lane is disabled, in accordance with some embodiments.

[0089] In the illustrated example, the execution state over time is shown for the following instruction sequence:

[0090] Instruction i0: ld.32 xR r0.sF2, 0x100

[0091] Instruction i1 : copy r1, r0 wF2

[0092] where the first instruction i0 is a load targeting register r0, and the second instruction is a copy of the contents of r0 into r1. The ".sF2" operator sets fence 2 and the "wF2" operator waits on this fence.

[0093] In the two examples of FIG. 7A and FIG. 7B , instruction i0 continues through the decode stage, the fence check stage, the instruction scheduling stage (e.g., stage 1 scheduler), the operand fetch stage, and the address generation stage. Instruction i1 continues through the decode stage, and then stalls at the fence check stage while waiting for fence 2 to clear.

[0094] In the example of FIG. 7A , there is a cache hit for instruction i0, and instruction i1 is submitted in the next cycle. In the example of FIG. 7B , there is a cache miss for instruction i0 and the lane manager 335 stalls the lane. This can allow the lane to be used by another SIMD group instead of being blocked by i1 while it waits for data to be fetched into the cache.

[0095] Example Preemption Technique in the Context of Memory-Supported Registers

[0096] In some embodiments, various memory spaces, including general purpose registers, are memory backed. For example, referring back to FIG. 3 , register data can be stored in DL0 370 and the operand cache 348 instead of in a dedicated register file, and can be backed up according to a unified memory architecture with UL1 385 and one or more other memory hierarchy levels (e.g., ultimately backed up at system memory, which can be shared by non-graphics data, e.g., from I / O units, CPUs, etc.).

[0097] In various cases, an operating system or driver can determine to context switch out a particular job, e.g., to allow a higher priority job to be executed. This can involve saving the context of the job that has completed and restoring the context when the higher priority job is completed. For a given thread group, all of the executing SIMD groups included in the thread group can need to save their contexts, and additional context of the thread group can also be saved. In a memory backed context, it can be challenging to properly handle register data when a SIMD group is context switched out.

[0098] FIG. 8is a flowchart illustrating an example method for handling preemption notifications according to some embodiments. At 810, in the illustrated embodiment, control circuitry receives a preemption notification, e.g., from tile and thread group manager 320. In response, control circuitry can initiate context storage to save the context of the preempted SIMD group. This can include saving their general purpose register data. Some instructions can be allowed to complete execution and save state, while others can be flushed and re-executed later.

[0099] At 820, in the illustrated embodiment, control circuitry flushes the cache line in DL0 370 used to store GPRs (e.g., to UL1 385). Control circuitry also saves a pointer to the memory page associated with the cache line. This can ensure that control circuitry is able to correctly locate those pages for context restoration (e.g., at 840, discussed in detail below).

[0100] Note that flushing the cache line can move data from the memory structure tagged with a private memory identifier (e.g., GPR data in an operand cache) to a higher level of the memory hierarchy tagged with a system memory address. This can advantageously facilitate context saving of that data so that it can later be correctly restored without loss.

[0101] Note that while GPRs are discussed herein for illustrative purposes, similar techniques can also be used for other architectural registers, such as special registers, stack registers, etc. Similarly, various private memory spaces can also be memory supported, such as thread group private memory, local image block private memory, etc.

[0102] At 830, in the illustrated embodiment, control circuitry invalidates the flushed cache line. Note that in some embodiments, flushing and invalidating can be specified as a single operation.

[0103] At 840, in the illustrated embodiment, for context restoration, control circuitry retrieves the saved memory pages used to track GPRs so that GPR data is available for continued execution.

[0104] FIG. 9A is a block diagram illustrating example circuitry associated with cache flushing / invalidation for context switching according to some embodiments. In the illustrated example, register data storage circuitry 920 (e.g., a register file, DL0 370, or operand cache 348) is configured to flush register data to cache / memory hierarchy 910 and invalidate it.

[0105] FIG. 9Bis a more detailed block diagram illustrating example circuitry associated with cache flush / invalidate for context switch, in accordance with some embodiments. In the illustrated example, the device includes memory 930, memory cache 940, LI cache 950, data L0 cache 960, and operand cache 970. In some embodiments, the device can omit one or more of the illustrated cache levels, include one or more other cache levels (not shown), or both. In the illustrated example, control circuitry can flush GPR data from data L0 cache 960 to LI cache 950 and invalidate it. Page management control circuitry can record these pages in memory 930, which support those cache lines for eventual recovery, as discussed in detail below.

[0106] FIG. 10 is a block diagram illustrating example control circuitry configured to process page information for context switch, in accordance with some embodiments. In the illustrated embodiment, the processor includes shader 160, memory management unit (MMU) 1010, memory allocator 1020, and page manager 1030, and is configured to access global memory 1065. As shown, MMU 1010 includes page information handler circuitry 1015, and memory allocator 1020 includes virtual hardware ID handler circuitry 1025. Circuitry 1015 and 1025 can correspond to the control circuitry discussed above.

[0107] In some embodiments, MMU 1010 is configured to communicate with global memory (GM) 1065 for access by shader circuitry 160. MMU 1010 can handle translations between address spaces (e.g., from private address space to virtual address, from virtual address to physical address, etc.). MMU 1010 can receive address mapping requests and page reservation requests, and respond with virtual addresses and page grants. MMU 1010 can communicate with memory allocator 1020 to obtain memory pages to respond to page reservation requests. In some embodiments, MMU 1010 maintains a page table cache.

[0108] In some embodiments, memory allocator circuitry 1020 is configured to communicate with global memory 1065 via page manager circuitry 1030 to obtain pages to satisfy requests from MMU 1010. Memory allocator 1020 can include a page descriptor cache and a page directory cache. Page manager circuitry 1030 can provide virtual addresses of allocated pages to memory allocator 1020.

[0109] In some embodiments, page manager circuit 1030 includes a main component and a distributed component. The main page manager can interact with software to manage page pools, while the distributed page manager can work with the MMU hierarchy for graphics shader hardware. For example, the main page manager can map a set of one or more graphics launches to a given memory pool. Thus, page manager circuit 1030 can store page table descriptor information in global memory 1065 and can also cache page table descriptor entries.

[0110] A number of "launches" can be performed to render a frame of graphics data. In some embodiments, a launch is a unit of work from a single context, which can include a number of threads to be executed (and potentially other types of graphics work not executed by a shader). A launch can not provide any guarantees regarding memory synchronization between threads (other than specified by the threads themselves), concurrency between threads, or order of launches between threads. In some embodiments, a launch can be identified in terms of the results of another launch, which can allow for memory synchronization without requiring hardware memory coherency support. Typically, graphics firmware or hardware programs are configured to register each launch before sending the work to the pipeline for processing. Typically, once a launch is started, it does not access more than a certain level of the memory hierarchy until the launch is complete (at which point the results can be written to another level in the hierarchy). Information for a given launch can include state information needed to complete a corresponding graphics operation, location of a shader program to be executed, buffer information, location of texture data, available address space, etc. For example, when a launch is complete, graphics firmware or hardware can schedule the launch and detect an interrupt. In some embodiments, portions of a graphics unit are configured to work on a single launch at a time. This group of resources can be referred to as a "launch slot." Thus, in some embodiments, any data needed for a given launch is read from memory that is shared among multiple processing elements at the start of the launch, and results are written back to the shared memory at the end of the launch. Thus, until the launch is complete, the results of the launch can not be visible to other hardware, at which point the results can be in shared memory and accessible by other launches (including launches from other data masters). A launch can include a set of one or more rendering commands, which can include commands to draw procedural geometry, commands to set up shadow sampling methods, commands to draw meshes, commands to retrieve textures, commands to perform generation computations, etc. A launch can be performed at one of various stages during frame rendering. Examples of rendering stages include, but are not limited to: camera rendering, light rendering, projection, texturing, fragment shading, etc. For example, a launch can be scheduled for compute work, vertex work, or pixel work.

[0111] In some embodiments, the graphics driver shoots a new launch into one of a plurality of launch slots. Each launch slot can include a set of configuration registers and can have a context ID that indicates a mapping between virtual addresses and physical addresses for that launch.

[0112] Pages can be reserved from a page pool before a corresponding launch is allowed to launch on shader hardware. When a shader program ends, the pages can be released back to the page pool (e.g., using a dedicated page reservation / release interface in some embodiments). Pages can also be allocated to the MMU (and in some embodiments, only reserved pages can be allocated) on demand and released when the MMU is done using them. Thus, in some embodiments, the page manager implements separate reservation / release interfaces and allocation / release interfaces. For allocation, the system can use a credit-based mechanism to push virtual pages to the MMU hierarchy. The page manager can also interact with one or more data controller units (e.g., vertex data controller, pixel data controller, and compute data controller) to coordinate launch start activities and launch end activities and support logical launch slots.

[0113] In some embodiments, the page information handler circuit 1015 is configured to save a directory base address (e.g., per page manager in a distributed embodiment with multiple page managers 1030). The backing cache lines for GPRs can be on pages managed by the MMU 1010, and thus restoring the directory base address can allow dynamic restoration when context restoration occurs. For example, rather than actually retrieving GPR data, dynamic restoration means that a miss will refill caches at various levels, such as page-related caches (such as page directory / directory / table caches) and data caches (such as the data cache shown in FIG. 1 1 1 ). While the directory base address is saved in this example, other base address information (or other identification information for page table levels) can be saved and restored in embodiments with other paging topologies. FIG. 9B

[0114] In some embodiments, the shader private memory address can contain an embedded hardware identifier, such as a SIMD group ID, a tile ID, an execution state ID, etc. The identifier can be encoded or hashed into the given address using various suitable techniques. In some embodiments, a given hardware ID is only allocated for work within a single launch slot at a time.

[0115] ​In some embodiments, the virtual hardware ID handler circuit 1025 is configured to virtualize private addresses to provide more virtual hardware IDs than the number of hardware IDs supported by the actual hardware. For example, the hardware IDs provided as part of the private addresses can be replaced by virtual hardware IDs within the memory allocator 1020. This can allow more virtual IDs than the number of hardware IDs, allow for early release of hardware IDs (e.g., release virtual IDs as background processes while making hardware IDs available for the next shader program), and facilitate preemption.

[0116] When a launch is preempted, the token resolver 315 can release the preempted hardware ID and trigger writing of the associated virtual hardware ID to a data structure in memory. Thus, the virtual hardware ID handler circuit 1025 is configured to save tracking information for virtualization when context switching out and retrieve the tracking information when context resuming in order to properly handle the virtual pages managed by the MMU 1010. At the time of resumption, the token resolver 315 can assign a new hardware ID to the retrieved virtual hardware ID and communicate the mapping to the MMU 1010. In some embodiments, most of the tracking information can be saved and restored at the page table cache instance in the MMU 1010. For example, the tracking information can also include the associated kernel ID and virtual tile ID.

[0117] In various embodiments, flushing and invalidating the cache lines used to support GPRs, as well as storing various associated information such as page directory base address information and virtualization information, can facilitate efficient context switching in a unified memory architecture.

[0118] Lane Stall Based on Instruction Cache Misses Overview

[0119] Another source of different stop times for threads can be instruction cache misses. A given architecture can include multiple levels of instruction caches and a miss at one or more levels can substantially increase latency. Thus, in some embodiments, when a lane misses at one or more instruction cache levels, the lane can be stalled. However, it can be desirable to guarantee forward progress (e.g., to ensure that a thread is not stalled continuously because other work repeatedly causes eviction of instructions for that thread while the thread is stalled). Thus, in some embodiments, some cache operations can implement a stall upon an instruction cache miss, rather than allowing stalling of the corresponding thread.

[0120] FIG. 11is a block diagram illustrating an example pipeline configured to stall a lane based on an instruction cache miss, in accordance with some embodiments. In the illustrated embodiment, the pipeline includes stage 1 scheduling circuitry 210, a plurality of lane pipelines (which include respective instruction cache tag check circuitry 1135 and decode circuitry 230), instruction cache 1110, and stage 2 scheduling circuitry 250.

[0121] Decode stage 230, as well as scheduling circuitries 210 and 250, can operate as described above with reference to FIG. 2

[0122] In some embodiments, instruction cache 1110 corresponds to one or both of IL0 cache 375 and IL1 cache 376. Given instruction cache tag check stage 1135 is configured to check whether the requested data is present in the instruction cache. If not, in some cases this stage can stall the SIMD group / lane that missed at one or more instruction cache levels. For example, the lane can generally be stalled, but ultimately can proceed with an instruction cache request that needs to return (indicating that the SIMD group should not be stalled) to provide forward progress. Detailed state diagrams for handling different types of instruction cache requests are discussed below with reference to FIG. 13

[0123] FIG. 12 is a block diagram illustrating an example two-level instruction cache implementation, in accordance with some embodiments. Although two levels are shown for purposes of illustration, in other embodiments, the disclosed lane stalling techniques can be used with additional cache levels or a single cache level.

[0124] In the illustrated embodiment, the processor includes lane manager 335, IL0 cache 375, and IL1 cache 376 (which can generally operate as discussed above with reference to FIG. 4

[0125] ​​​In the illustrated embodiment, when an instruction for a lane misses in the instruction buffer (not shown), the lane manager 335 issues an instruction buffer miss request. The IL0 cache 375 checks the IL0 tag 1220 based on the request. For a miss, the request goes into the IL0 miss queue 1210, which then issues to the IL1 cache 376. The IL1 cache 376 in turn checks the IL1 tag 1222, and the request goes into the IL1 miss queue 1212 on a miss. The request goes to the cache / memory hierarchy 1205 as an L1 miss request. When the data finally returns, it propagates via the IL1 cache storage 1232, the IL0 cache storage 1230, and back to the lane manager 335 for execution.

[0126] In some embodiments, when a request misses in the IL1 cache 376, the control circuitry deactivates the corresponding lane. When the cache fill data arrives in the IL1 cache 376 from the cache / memory hierarchy 1205, the control circuitry can signal that the lane is ready to arbitrate. Once the SIMD group is reactivated, it can request instruction data, which can then be fetched into the IL0 and instruction buffer.

[0127] As discussed above, a thread can be deactivated because of an instruction cache miss, reactivated, but deactivated again because another thread causes its desired instruction data to be evicted in the interim. Thus, in some embodiments, the cache control circuitry supports two types of instruction cache requests, "no return" requests, which allow deactivation for instruction cache misses, and "return" requests, which do not cause deactivation for misses in IL0 and IL1, but will wait until the data is available in IL0.

[0128] FIG. 13 An example state diagram for instruction cache control is illustrated in accordance with some embodiments. As shown, the transitions are different in certain cases for misses with return and no return. In the illustrated example, "M" denotes a miss, "R" denotes reading data from IL0, and "W" denotes a write from IL1 to IL0.

[0129] As shown, from INVALID, a non-returning miss will transition to PEND_WR, while a returning miss will transition to PEND_WRR, while the instruction is being fetched. Note that the "w_rtn" field can indicate whether the request is returning or non-returning. If a second miss occurs with a different w_rtn, the state changes to PEND_2WR. Because a non-returning miss can not return the instruction, a read with no write will change the state back to INVALID. If the instruction is returned, a write will change the state to PEND_RD. Once in the PEND_RD state, a read will change the state to VALID. From the VALID state, as before, a miss replacement will move the state to PEND_WR or PEND_WRR based on w_rtn, and a cache flush invalidate (CFI) will change the state to INVALID.

[0130] Example Method

[0131] FIG. 14 FIG. 1 is a flow diagram illustrating an example method for scheduling threads based on backpressure information, in accordance with some embodiments. FIG. 14 The illustrated methods can be used in connection with any of the computer circuitry, systems, devices, elements, or components, etc. disclosed herein. In various embodiments, some of the method elements shown can be performed concurrently, in a different order than shown, or omitted. Additional method elements can also be performed as desired.

[0132] At 1410, in the illustrated embodiment, the computing device performs operations for a plurality of threads. In some embodiments, the performing utilizes a plurality of lane pipelines for a plurality of lanes, where a lane pipeline for a given lane includes a decode stage; and a plurality of execution pipelines shared by the lane pipeline and configured to perform different types of operations provided by the lane pipeline.

[0133] In some embodiments, the thread is included in a single instruction multiple data (SIMD) group, and the scheduler circuit is configured to assign the SIMD group to a lane. In some embodiments, the plurality of execution pipelines includes an integer pipeline, a floating point pipeline, and a load pipeline.

[0134] At 1420, in the illustrated embodiment, the computing device (e.g., using the stage 1 scheduler 210) arbitrates between threads to assign the threads to lanes.

[0135] At 1430, in the illustrated embodiment, the computing device (e.g., using stage 2 scheduler 250) arbitrates among lanes to assign the operation from a given lane to a given execution pipeline based on the decoding of the operation by the decoding stage. In some embodiments, stage 2 scheduler 250 is configured to arbitrate based on lane priority, hardware resources requested by the lane, and hardware resource availability information.

[0136] At 1440, in the illustrated embodiment, the computing device (e.g., using pipeline 260) provides backpressure information based on execution state.

[0137] At 1450, in the illustrated embodiment, the computing device adjusts priority of threads assigned to lanes based on the backpressure information.

[0138] In some embodiments, the priority adjustment is based on compiler hint information indicating a type of execution pipeline targeted by a thread. For example, a scheduler circuit can increase priority of threads targeting an execution pipeline with backpressure below a threshold (or decrease priority of threads targeting a busy execution pipeline). In some embodiments, stage 1 scheduler 210 implements age-based arbitration among lanes, and is configured to adjust an initial age-based priority of a thread based on the backpressure information.

[0139] In some embodiments, stage 1 scheduler 210 is configured to, prior to completion of a first thread assigned to a first lane, perform a preemption procedure to remove the first thread from the first lane and assign a second thread to the first lane, where the second thread has a higher priority than the first thread. In some embodiments, dependency circuit is configured to, for a first operation that depends on a previous operation using one of the execution pipelines, determine whether to stall the first operation or deactivate a thread including the first operation from a lane to which it is assigned based on state information from the previous operation in the one of the execution pipelines.

[0140] FIG. 15 is a flow diagram illustrating an example method for dependency management in accordance with some embodiments. FIG. 15 The illustrated methods can be used in conjunction with any of the computer circuitry, systems, devices, elements, or components disclosed herein. In various embodiments, some of the method elements can be performed concurrently, in a different order than shown, or can be omitted. Additional method elements can also be performed as desired.

[0141] At 1510, in the illustrated embodiment, the computing device executes the instructions using: a plurality of lane pipelines for a plurality of lanes, where a lane pipeline for a given lane includes a decode stage; and a plurality of execution pipelines shared by the lane pipelines and configured to perform different types of operations provided by the lane pipelines.

[0142] At 1520, in the illustrated embodiment, the computing device (e.g., using stage 1 scheduler 210) assigns threads to lanes.

[0143] At 1530, in the illustrated embodiment, the computing device (e.g., using stage 2 scheduler 250) assigns operations from a given lane to a given execution pipeline based on a decode of the operations for the given lane.

[0144] At 1540, in the illustrated embodiment, the computing device (e.g., using fence manager 355) determines, for a first operation that depends on a previous operation of one of the execution pipelines, whether to stall the first operation or deactivate a thread assigned from a lane including the first operation based on state information from the previous operation of the one of the execution pipelines.

[0145] In some embodiments, the plurality of execution pipelines are configured to operate on input operands stored in memory-backed registers, where the state information includes information about a data cache access that missed a memory-backed register. In some embodiments, the dependency circuit stalls the first operation in response to all input operands of the previous operation being available in registers or a hit in a first level data cache. In some embodiments, the dependency circuit deactivates a thread including the first operation in response to a miss of the previous operation in the first level data cache.

[0146] In some embodiments, the state information indicates a current pipeline stage of the previous operation. In some embodiments, the dependency circuit is configured to resume the first operation (e.g., if it was stalled rather than deactivated) in response to one or more conditions associated with the previous operation.

[0147] In some embodiments, the dependency circuit detects the dependency of the first operation on the previous operation based on a compiler-inserted fence, and the dependency circuit includes a set of active fence counters. In some embodiments, the dependency circuit includes circuitry that tracks whether a given fence counter represents a decremented number expected due to a cache hit.

[0148] In some embodiments, the first scheduler circuit is configured to consider a thread that is stalled for reassignment to a lane and is configured to adjust a priority of the thread based on a compiler hint indicating a target execution pipeline and state information of the target execution pipeline.

[0149] FIG. 16 is a flowchart illustrating an example method for preemption in the context of a memory backed register according to some embodiments. FIG. 16 The illustrated methods can be used in connection with any of the computer circuitry, systems, devices, elements, or components, etc. disclosed herein. In various embodiments, some of the method elements shown can be performed concurrently, in a different order than shown, or omitted. Additional method elements can also be performed as desired.

[0150] At 1610, in the illustrated embodiment, the computing device operates on operands in an architectural register to execute an instruction of a thread, where data of the architectural register is stored and backed up by a memory hierarchy including one or more cache levels and one or more memory circuits.

[0151] At 1620, in the illustrated embodiment, the computing device flushes and invalidates a set of architectural register data from a first cache level in response to a context switch indication for a given thread.

[0152] At 1630, in the illustrated embodiment, the computing device stores memory page information associated with the set of architectural register data in response to the context switch indication. In some embodiments, the memory page information includes a page directory base address.

[0153] In some embodiments, the memory allocator circuit allocates one or more pages for the set of architectural register data, and recovers the one or more pages based on the stored memory page information in response to a resumption of the thread. In some embodiments, the memory management circuit retrieves and caches one or more pages based on the stored memory page information.

[0154] In some embodiments, the set of architectural register data is stored in a private memory space for the thread that is addressed based at least in part on hardware identifier information. In some embodiments, the control circuit stores virtual hardware identifier information for the thread in response to the context switch indication, and recovers the virtual hardware identifier information for the thread in response to a resumption of the thread.

[0155] In some embodiments, an operand cache (e.g., cache 348 of FIG. 3) stores operand data at a level closer to the execution circuit in the memory hierarchy than the first cache level. FIG. 3 In some embodiments, the operand cache (e.g., cache 348 of FIG. 3) stores operand data at a level closer to the execution circuit in the memory hierarchy than the first cache level.

[0156] FIG. 17 is a flowchart illustrating an example method for deactivating a thread from a lane based on an instruction cache miss, in accordance with some embodiments. FIG. 17 The illustrated methods can be used in conjunction with any of the computer circuitry, systems, devices, elements, or components disclosed herein. In various embodiments, some of the method elements can be performed concurrently, in a different order than illustrated, or can be omitted. Additional method elements can also be performed as desired.

[0157] At 1710, in the illustrated embodiment, the computing device (e.g., using stage 1 scheduler 210) assigns a thread to a lane corresponding to a plurality of lane pipelines, where a given lane pipeline includes a decode stage and the lane pipeline shares a plurality of execution pipelines.

[0158] At 1720, in the illustrated embodiment, the computing device (e.g., using stage 2 scheduler 250) assigns an operation from a given lane to an execution pipeline based on a decode of the operation for the lane.

[0159] At 1730, in the illustrated embodiment, the computing device (e.g., using thread replacement control circuitry) deactivates a first thread from a first lane in response to an instruction cache miss for an operation of the first thread assigned to the first lane. The control circuitry can support a control field indicating whether a given operation is discardable or non-discardable, and in response to detecting an instruction cache miss for a non-discardable operation of a given thread, the control circuitry can cause the thread to stall in its assigned lane until corresponding instruction data is available (rather than deactivating the thread).

[0160] In some embodiments, the first scheduler circuitry is configured to promote an operation from the first thread to a non-discardable request in response to a threshold number of misses of one or more discardable requests of the first thread. In some embodiments, the control circuitry notifies the first scheduler circuitry (e.g., using scheduler 210) in response to retrieving instruction data for the first thread into the instruction cache.

[0161] In some embodiments, the instruction cache circuitry includes at least a first hierarchical level and a second hierarchical level, and the thread replacement circuitry is configured to deactivate the first thread from the first lane in response to a miss in the first and second levels. In some embodiments, the control circuitry is configured to notify the first scheduler circuitry in response to retrieving instruction data into the first level, but the control circuitry is not configured to fetch the instruction data into the second level until requested by the first thread after the first thread is reactivated.

[0162] At 1740, in the illustrated embodiment, the computing device executes the assigned instructions using the execution pipeline (e.g., using pipeline 260). This can include executing the instructions of the first thread, followed by the instructions of another thread after the first thread is deactivated.

[0163] In some embodiments, the miss queue circuitry is configured to store information for instruction cache misses, and the cache control circuitry is configured to send a single fetch request to the same instruction cache entry in response to multiple instruction cache misses, and to update multiple corresponding miss queue entries in response to completion of the single fetch request.

[0164] The concept of “execution” is broad and can refer to 1) the processing of an instruction through the entire execution pipeline (e.g., through the fetch stage, the decode stage, the execution stage, and the retirement stage), and 2) the processing of an instruction at an execution unit or execution subsystem of such a pipeline (e.g., an integer execution unit or a load store unit). The latter meaning can also be referred to as “performing” an instruction. Thus, “performing” an add instruction refers to adding two operands to produce a result, which in some embodiments can be implemented by circuitry at the execution stage of the pipeline (e.g., an execution unit). In contrast, “executing” an add instruction can refer to the entire operation that occurs through the pipeline as a result of the add instruction. Similarly, “performing” a “load” instruction can include retrieving a value (e.g., from a register, memory, or a storage result of another instruction) and storing the retrieved value into a register or other location.

[0165] As used herein, in the context of an instruction, the term “completion” refers to the commitment of the result of the instruction to the architectural state of the processor or processing element. For example, the completion of an add instruction includes writing the result of the add instruction to a target register. Similarly, the completion of a load instruction includes writing a value (e.g., retrieved from a register or memory) to a target register or its representation.

[0166] The concept of a processor “pipeline” is well known and refers to the concept of dividing the “work” that a processor does on an instruction into multiple stages. In some embodiments, the decoding, dispatching, execution (i.e., performance), and retirement of an instruction can be examples of different pipeline stages. Many different pipeline architectures can have different ordering of elements / sections. The various pipeline stages perform such steps on an instruction during one or more processor clock cycles, then pass the instruction or operations associated with the instruction to other stages for further processing.

[0167] Example Device

[0168] Reference is now made to FIG. 18FIG. 18 illustrates a block diagram showing an example implementation of a device 1800. In some embodiments, the elements of device 1800 can be included within a system on a chip. In some embodiments, device 1800 can be included in a mobile device that can be battery powered. Thus, power consumption of device 1800 can be an important design consideration. In the illustrated embodiment, device 1800 includes fabric 1810, compute complex 1820, input / output (I / O) bridge 1850, cache / memory controller 1845, graphics unit 1875, and display unit 1865. In some embodiments, device 1800 can include other components (not shown) in addition to or instead of the illustrated components, such as video processor encoders and decoders, image processing or recognition elements, computer vision elements, etc.

[0169] Fabric 1810 can include various interconnects, buses, MUXs, controllers, etc., and can be configured to facilitate communication between various elements of device 1800. In some embodiments, portions of fabric 1810 can be configured to implement various different communication protocols. In other embodiments, fabric 1810 can implement a single communication protocol, and elements coupled to fabric 1810 can translate from the single communication protocol to other communication protocols internally.

[0170] In the illustrated embodiment, compute complex 1820 includes bus interface unit (BIU) 1825, cache 1830, and cores 1835 and 1840. In various embodiments, compute complex 1820 can include various numbers of processors, processor cores, and caches. For example, compute complex 1820 can include 1, 2, or 4, or any other suitable number of processor cores. In one embodiment, cache 1830 is a set-associative L2 cache. In some embodiments, cores 1835 and 1840 can include internal instruction and data caches. In some embodiments, a coherency unit (not shown) of fabric 1810, cache 1830, or elsewhere in device 1800 can be configured to maintain coherency between various caches of device 1800. BIU 1825 can be configured to manage communications between compute complex 1820 and other elements of device 1800. Processor cores, such as cores 1835 and 1840, can be configured to execute instructions of a particular instruction set architecture (ISA) that can include operating system instructions and user application instructions.

[0171] Note that while various GPU implementations have been discussed herein, various disclosed techniques, such as multi-stage scheduling, compiler hints, fence management, register handling for memory support for preemption, and processing instruction cache misses, can be implemented in other types of processors, such as compute complex 1820.

[0172] Cache / memory controller 1845 can be configured to manage the transfer of data between fabric 1810 and one or more caches and memories. For example, cache / memory controller 1845 can be coupled to an L3 cache, which in turn can be coupled to system memory. In other embodiments, cache / memory controller 1845 can be directly coupled to memory. In some embodiments, cache / memory controller 1845 can include one or more internal caches.

[0173] As used herein, the term“coupled to” can indicate one or more connections between elements, and a coupling can include intervening elements. For example, in FIG. 18 , graphics unit 1875 can be described as“coupled to” memory through fabric 1810 and cache / memory controller 1845. In contrast, in FIG. 18 , graphics unit 1875 is“directly coupled” to fabric 1810 because there are no intervening elements.

[0174] Graphics unit 1875 can include one or more processors, such as one or more graphics processing units (GPUs). For example, graphics unit 1875 can receive graphics-oriented instructions, such as Metal or instructions. Graphics unit 1875 can execute specialized GPU instructions or perform other operations based on the received graphics-oriented instructions. Graphics unit 1875 can generally be configured to process large blocks of data in parallel and can construct images in a frame buffer for output to a display, which can be included in a device or can be a separate device. Graphics unit 1875 can include transform, lighting, triangle, and rendering engines in one or more graphics processing pipelines. Graphics unit 1875 can output pixel information for displayed images. In various embodiments, graphics unit 1875 can include programmable shader circuitry, which can include highly parallel execution cores configured to execute graphics programs, which can include pixel tasks, vertex tasks, and compute tasks (which can or can not be graphics-related).

[0175] In various embodiments, the disclosed technology can improve the performance of operations performed by graphics unit 1875, reduce its power consumption, or both.

[0176] Display unit 1865 can be configured to read data from frame buffers and provide a stream of pixel values for display. In some embodiments, display unit 1865 can be configured as a display pipeline. Additionally, display unit 1865 can be configured to blend multiple frames to produce an output frame. Further, display unit 1865 can include one or more interfaces (e.g., a high-definition multimedia interface (HDMI), a mobile industry processor interface (MIPI), or an embedded display port (eDP)) to couple to a user display (e.g., a touchscreen or an external display).

[0177] I / O bridge 1850 can include various elements configured to enable, for example, universal serial bus (USB) communications, security, audio, and low-power always-on functionality. I / O bridge 1850 can also include interfaces such as, for example, pulse width modulation (PWM), general purpose input / output (GPIO), serial peripheral interface (SPI), and inter-integrated circuit (I2C). Various types of peripheral devices and equipment can be coupled to device 1800 via I / O bridge 1850.

[0178] In some embodiments, device 1800 includes network interface circuitry (not shown explicitly) that can be connected to fabric 1810 or I / O bridge 1850. The network interface circuitry can be configured to communicate over various networks, which can be wired, wireless, or both. For example, the network interface circuitry can be configured to communicate over a wired local area network, a wireless local area network (e.g., via WiFi), or a wide area network (e.g., the Internet or a virtual private network). In some embodiments, the network interface circuitry is configured to communicate via one or more cellular networks using one or more radio access technologies. In some embodiments, the network interface circuitry is configured to communicate using device-to-device communication (e.g., Bluetooth or WiFi Direct), among other possibilities. In various embodiments, the network interface circuitry can provide device 1800 with connectivity to various types of other devices and networks.

[0179] Example Application

[0180] Turning now to FIG. 19 , various types of systems that can include any of the circuitry, devices, or systems discussed above are shown. Systems or devices 1900, which can incorporate or otherwise utilize one or more of the techniques described herein, can be used in a wide range of fields. For example, systems or devices 1900 can be used as part of the hardware of a system such as a desktop computer 1910, a laptop computer 1920, a tablet computer 1930, a cellular or mobile telephone 1940, or a television 1950 (or a set-top box coupled to a television).

[0181] ​Similarly, the disclosed elements can be used in a wearable device 1960, such as a smart watch or health monitoring device. In many embodiments, a smart watch can implement a variety of different functions— e.g., access to email, cellular service, a calendar, health monitoring, etc. Wearable devices can also be designed to perform only health monitoring functions, such as monitoring a user’s vital signs, performing epidemiological functions such as contact tracing, providing communication to emergency medical services, etc. Other types of devices are also contemplated, including devices worn on the neck, devices implantable in the human body, glasses or headsets designed to provide computer-generated reality experiences, such as those based on augmented reality and / or virtual reality, etc.

[0182] The system or device 1900 can also be used in a variety of other contexts. For example, the system or device 1900 can be used in the context of a server computer system, such as a dedicated server, or on shared hardware that implements a cloud-based service 1970. Still further, the system or device 1900 can be implemented in a wide range of specialized everyday devices, including devices commonly found in the home 1980, such as refrigerators, thermostats, security cameras, etc. The interconnection of such devices is often referred to as the “Internet of Things” (IoT). The elements can also be implemented in various modes of transportation. For example, the system or device 1900 can be used in control systems, guidance systems, entertainment systems, etc. of various types of vehicles 1990.

[0183] FIG. 19 The applications shown in FIG. 16 are merely exemplary and are not intended to suggest any limitation as to the scope of use or functionality of the disclosed system or device. Other example applications include, without limitation, portable gaming devices, music players, data storage devices, unmanned aerial vehicles, etc.

[0184] Example Computer-Readable Medium

[0185] The present disclosure has described in detail various example circuits above. It is intended that the present disclosure encompass not only embodiments comprising such circuits, but also computer readable storage media comprising design information specifying such circuits. Thus, the present disclosure is intended to support claims not only encompassing apparatuses comprising the disclosed circuits, but also storage media specifying circuits in a format recognized by a manufacturing system configured to produce hardware (e.g., integrated circuits) comprising the disclosed circuits. Claims to such storage media are intended to encompass, for example, entities that generate circuit designs but do not themselves manufacture the designs.

[0186] FIG. 20is a block diagram illustrating an example non-transitory computer-readable storage medium storing circuit design information, in accordance with some embodiments. In the illustrated embodiment, semiconductor fabrication system 2020 is configured to process design information 2015 stored on non-transitory computer-readable medium 2010 and to fabricate integrated circuit 2030 based on design information 2015.

[0187] Non-transitory computer-readable storage medium 2010 can include any of a variety of appropriate types of memory devices or storage devices. Non-transitory computer-readable storage medium 2010 can be an installation medium, such as a CD-ROM, floppy disks, or tape device; computer system memory or random access memory such as DRAM, DDR RAM, SRAM, EDO RAM, Rambus RAM, etc.; non-volatile memory such as flash memory, magnetic media, e.g., a hard disk drive or optical storage; registers, or other like type of storage elements, and the like. Non-transitory computer-readable storage medium 2010 can also include other types of non-transitory memory or combinations thereof. Non-transitory computer-readable storage medium 2010 can include two or more memory media, which can be resident in the same or different locations as the computer system that is using these memory media. In some embodiments, non-transitory computer-readable storage medium 2010 can be encoded with one or more programs that, when executed, instruct a computer system to fabricate integrated circuit 2030 based on design information 2015.

[0188] Design information 2015 can be specified using any of a variety of appropriate computer languages, including hardware description languages such as, but not limited to, VHDL, Verilog, SystemC, SystemVerilog, RHDL, M, MyHDL, etc. Design information 2015 can be usable by semiconductor fabrication system 2020 to fabricate at least a portion of integrated circuit 2030. The format of design information 2015 can be recognized by at least one semiconductor fabrication system 2020. In some embodiments, design information 2015 can also include one or more cell libraries specifying a synthesis, a layout, or both, of integrated circuit 2030. In some embodiments, design information is specified in whole or in part in the form of a netlist specifying cell library elements and their connectivity. Design information 2015 taken separately can or can not include sufficient information to fabricate a corresponding integrated circuit. For example, design information 2015 can specify circuit elements to be fabricated, but not their physical layout. In this case, design information 2015 can need to be combined with layout information to actually fabricate the specified circuit.

[0189] In various implementations, the integrated circuit 2030 may include one or more custom macrocells, such as memory, analog or mixed-signal circuitry, etc. In this case, design information 2015 may include information associated with the included macrocells. Such information may include, but is not limited to, schematic capture databases, mask design data, behavioral models, and device or transistor-level netlists. As used herein, mask design data may be formatted according to a Graphical Data System (GDSII) or any other suitable format.

[0190] The semiconductor manufacturing system 2020 may include any of the various suitable elements configured to manufacture integrated circuits. This may include, for example, elements for depositing semiconductor material (e.g., on a wafer that may include a mask), removing material, changing the shape of the deposited material, modifying the material (e.g., by doping the material or by using ultraviolet treatment to modify the dielectric constant), etc. The semiconductor manufacturing system 2020 may also be configured to perform various tests on the manufactured circuits for proper operation.

[0191] In various implementations, integrated circuit 2030 is configured to operate according to a circuit design specified by design information 2015, which may include performing any of the functionalities described herein. For example, integrated circuit 2030 may include FIG. 1B , FIG. 2 to FIG. 6 Figure 9 to FIG. 12 and FIG. 18 Any of the various elements shown herein. Additionally, integrated circuit 2030 can be configured to perform the various functions described herein in conjunction with other components. Furthermore, the functionality described herein can be performed by multiple interconnected integrated circuits.

[0192] As used herein, a phrase in the form of “design information specifying the design of a circuit configured to…” does not imply that the circuit in question must be manufactured in order to satisfy this element. Rather, the phrase indicates that the design information describes a circuit that, when manufactured, will be configured to perform the indicated action or will include the specified components.

[0193] ***

[0194] This disclosure includes references to “implementation scheme” or groups of “implementation schemes” (e.g., “some implementation schemes” or “various implementation schemes”). An implementation scheme is a different specific implementation or instance of the disclosed concepts. References to “implementation scheme,” “an implementation scheme,” “a particular implementation scheme,” etc., do not necessarily refer to the same implementation scheme. A large number of possible implementation schemes are contemplated, including those specifically disclosed, as well as modifications or alternatives that fall within the substance or scope of this disclosure.

[0195] This disclosure may discuss potential advantages that may arise from the disclosed embodiments. Not all specific implementations of all these embodiments will necessarily exhibit any or all of the potential advantages. Whether a particular embodiment achieves an advantage depends on many factors, some of which are outside the scope of this disclosure. In fact, there are many reasons why an embodiment falling within the scope of the claims may not exhibit some or all of any of the disclosed advantages. For example, a particular embodiment may include other circuitry outside the scope of this disclosure, in conjunction with an embodiment of the disclosed embodiments, which negates or diminishes one or more of the disclosed advantages. Furthermore, suboptimal design execution of a particular embodiment (e.g., the implementing technique or tool) may also negate or diminish the disclosed advantages. Even assuming an implementation of the technique, the realization of advantages may still depend on other factors, such as the environmental circumstances in which the implementation is deployed. For example, the inputs provided to a particular embodiment may prevent one or more problems addressed in this disclosure from occurring in a particular context, and as a result, the benefits of its solution may not be realized. In view of the existence of possible factors outside this disclosure, it is hereby expressed that any potential advantage described herein should not be construed as a claim limitation that must be satisfied in order to prove infringement. Rather, the identification of such potential advantages is intended to illustrate the types of improvements available to designers who benefit from this disclosure. Describing such advantages permanently (e.g., stating that a particular advantage "may occur") is not intended to convey a question about whether such advantages can actually be realized, but rather to recognize that the realization of such advantages often depends on the technological reality of additional factors.

[0196] Unless otherwise stated, the embodiments are non-limiting. That is, the disclosed embodiments are not intended to limit the scope of the claims drafted based on this disclosure, even where only a single example is described for a particular feature. The disclosed embodiments are intended to be illustrative and not restrictive, without any statement to the contrary in this disclosure. Therefore, this application is intended to allow for claims covering the disclosed embodiments, as well as such alternatives, modifications, and equivalents, which will be apparent to those skilled in the art upon which the beneficial effects of this disclosure are known.

[0197] For example, features in this application can be combined in any suitable manner. Therefore, new claims may be made for any such combination of features during the filing of this application (or an application claiming priority thereto). Specifically, referring to the appended claims, features of dependent claims may be combined with features of other dependent claims, including claims dependent on other independent claims, where appropriate. Similarly, features from the respective independent claims may be combined where appropriate.

[0198] Accordingly, although the appended claims are drafted in the singular and plural, this is done to embrace all the different standalone implementations of the disclosed application. Thus, the appended claims can be drafted in the singular and plural alternatively depending on the context. For example, a claim drafted in the singular sense can be used to support a claim drafted in the plural sense, and vice versa.

[0199] Where appropriate, a claim drafted in one format or style can also serve as a disclaimer of a corresponding claim drafted in the alternative format or style (e.g., a method claim can serve as a disclaimer of a corresponding apparatus claim and vice versa).

[0200] ***

[0201] As the present disclosure is a legal document, various terms and phrases can be subject to regulatory and judicial interpretation. It is hereby given notice that the following paragraphs, as well as the definitions provided throughout the present disclosure, will be used to determine how claims drafted based on the present disclosure are interpreted.

[0202] Unless the context clearly dictates otherwise, a reference to an item in the singular (i.e., a noun or noun phrase preceded by “a,” “an,” or “the”) is intended to mean “one or more.” Thus, a reference in the claims to an “item” does not exclude additional instances of the item. A plurality of items is a collection of two or more items.

[0203] The word “may” is used herein in the permissive sense (i.e., having the potential to), rather than the mandatory sense (i.e., must). The term “include” and variations thereof, such as “includes” and “including,” mean “including but not limited to.”

[0204] The terms “comprise” and “comprising,” and variations thereof, are open-ended, and mean “including but not limited to.”

[0205] When the term “or” is used in the present disclosure to link a list of items, it is intended that at least one, but not necessarily all, of the items be present in the list. Thus, the statement “x or y” is equivalent to “x or y, or both” and thus covers the cases where at least one of x or y is present, but not necessarily both. On the other hand, the phrase “one of x or y, but not both” makes it clear that either x or y is to be present, but not both.

[0206] The expressions "w, x, y, or z, or any combination thereof" or "at least one of w, x, y, and z" are intended to cover all possibilities involving up to the total number of elements in the set. For example, given the set [w, x, y, z], these phrases cover any single element in the set (e.g., w but not x, y, or z), any two elements (e.g., w and x but not y or z), any three elements (e.g., w, x, and y but not z), and all four elements. The phrase "at least one of w, x, y, and z" thus refers to at least one element in the set [w, x, y, z] thereby encompassing all possible combinations in the list of elements. The phrase should not be interpreted to require the presence of at least one instance of w, at least one instance of x, at least one instance of y, and at least one instance of z.

[0207] In this disclosure, various "labels" can precede a noun or noun phrase. Unless the context provides otherwise, different labels used for a feature (e.g., "first circuit," "second circuit," "particular circuit," "given circuit," etc.) refer to different instances of the feature. Additionally, unless otherwise noted, the labels "first," "second," and "third" when applied to a feature do not imply any type of ordering (e.g., spatial, temporal, logical, etc.).

[0208] The phrase "based on" or "based upon" is used to describe one or more factors that affect a determination. This term does not foreclose additional factors that can affect a determination. That is, a determination can be solely based on specified factors or based, at least in part, on specified factors and other, unspecified factors. Consider the phrase "determine A based on B." This phrase specifies that B is a factor that affects the determination of A. This phrase does not foreclose the determination of A from also being based on some other factor, such as C. This phrase also allows for a determination of A to be based on B solely, in an embodiment. As used herein, the phrase "based on" is synonymous with the phrase "based at least in part on."

[0209] The phrases "in response to" and "in response" describe one or more factors that trigger an effect. This phrase does not foreclose additional factors that can affect or otherwise trigger the effect, either jointly or severally, with the specified factors. That is, an effect can be triggered solely in response to the specified factors, or an effect can be triggered in response to the specified factors and some other, unspecified factors. Consider the phrase "perform A in response to B." This phrase specifies that B is a factor that triggers the performance of A or the triggering of a particular result of A. This phrase does not foreclose the performance of A from also being triggered in response to some other factor, such as C. This phrase also allows for the performance of A to be triggered in response to both B and C jointly. This phrase also allows for an embodiment in which A is performed solely in response to B. As used herein, the phrase "in response" is synonymous with the phrase "in response at least in part to." Similarly, the phrase "in response to" is synonymous with the phrase "in response at least in part to."

[0210] ***

[0211] Within the present disclosure, different entities (which can variously be referred to as “units,” “circuits,” other components, etc.) can be described or claimed as “configured” to perform one or more tasks or operations. This formulation—[entity] configured to [perform one or more tasks]—is used herein to convey structural determinism. Thus, within the context of this specification, “configured” describes a structural relationship. Where first and second entities are described or claimed as “configured” to perform a task or operation, such description is structurally sound, even if the first and second entities are not currently operational. Thus, such structural

[0212] In some cases, various units / circuits / components can be described herein as performing a set of tasks or operations. It should be understood that these entities “configured” to perform those tasks / operations are capable of so doing, even if not currently operational.

[0213] The term “configured” is used herein to convey structural determinism. Thus, a device or structure that is configured to perform a task is a structural determination that the device or structure is to perform the task. In some cases, a device or structure that is configured to perform a task is not capable of performing the task until so configured.

[0214] For purposes of United States patent applications filed on or after September 8, 2012, the term “consisting essentially of’ is always to be construed in accordance with 35 U.S.C. § 112(f) and 21 C.F.R. § 1.552(b)(3). Not 35 U.S.C. § 112(f). If the applicant desires to invoke § 112(f) for purposes of United States patent applications filed on or after September 8, 2012, the applicant will use the “means for” construction in the claims.

[0215] Different “circuits” can be described in the present disclosure. These circuits constitute hardware that includes various types of circuit elements, such as combinational logic, clocked storage devices (e.g., flip-flops, registers, latches, etc.), finite state machines, memory (e.g., random access memory, embedded dynamic random access memory), programmable logic arrays, etc. The circuits can be custom-designed, or taken from a standard library. In various implementations, the circuits can include digital components, analog components, or a combination of both, as appropriate. Certain types of circuits can be commonly referred to as “units” (e.g., a decode unit, an arithmetic logic unit (ALU), a function unit, a memory management unit (MMU), etc.). Such units also refer to circuits or circuitry.

[0216] Thus, the disclosed circuits / units / components and other elements illustrated in the drawings and described herein include hardware elements, such as those described in the preceding paragraph. In many cases, the internal arrangement of a hardware element within a particular circuit can be specified by describing the function of the circuit. For example, a particular “decode unit” can be described as performing the function of “processing the operation code of an instruction and routing the instruction to one or more of a plurality of functional units,” which means that the decode unit is “configured to” perform this function. This functional specification is sufficient to imply a set of possible structures for the circuit to a person of skill in the computer arts.

[0217] In various embodiments, circuits, units, and other elements can be defined by the functions or operations that they are configured to implement, as discussed in the preceding paragraph. The arrangement of relative to one another and the manner in which such circuits / units / components interact, as well as the way they are programmed, define the microarchitecture of the hardware, which is ultimately fabricated in integrated circuits or programmed into FPGAs to form the physical instantiation of the microarchitecture definition. The microarchitecture definition is therefore considered by those skilled in the art to be a structure from which many physical instantiations can be derived, all falling within the broader structure described by the microarchitecture definition. That is, a skilled artisan provided with the microarchitecture definition according to the present disclosure could, without undue experimentation, and with the application of ordinary skill, implement the structure by translating the description of the circuits / units / components into a hardware description language (HDL) such as Verilog or VHDL. The HDL description is often expressed in a way that can appear functional. But to the skilled artisan, the HDL description is a way to convert the structure of the circuits, units, or components into a next level of instantiation details. Such HDL descriptions can take the form of behavioral code (which is often not synthesizable), register transfer language (RTL) code (which is often synthesizable compared to behavioral code), or structural code (e.g., a netlist specifying logic gates and their connectivity). The HDL description can be synthesized sequentially against a library of cells designed for a given integrated circuit fabrication technology, and can be modified for timing, power, and other reasons to arrive at a final design database that is sent to a foundry to generate masks and ultimately produce integrated circuits. Some hardware circuits or portions thereof can also be custom designed in a schematic editor and captured into the integrated circuit design along with the synthesized circuits. The integrated circuits can include transistors and other circuit elements (e.g., passive elements such as capacitors, resistors, inductors, etc.), as well as interconnectors between the transistors and circuit elements. Some embodiments can implement multiple integrated circuits coupled together to implement the hardware circuits, and / or can use discrete elements in some embodiments. Alternatively, the HDL design can be synthesized for a programmable logic array such as a field-programmable gate array (FPGA), and can be implemented in the FPGA. This decoupling between the design of a set of circuits and the subsequent lower-level instantiations of those circuits often results in a situation in which the circuit or logic designer never specifies a particular set of structures for the lower-level instantiations beyond a description of what the circuits are configured to do, as this process is performed at a different stage of the circuit implementation process.

[0218] The fact that the same functionality of a circuit can be implemented using many different low-level combinations of circuit elements results in a large number of equivalent structures of that circuit. As noted, these low-level circuit implementations can vary according to variations in manufacturing technology, foundry chosen for fabricating the integrated circuit, cell library provided for a particular project, and the like. In many cases, the selection of these different implementations is arbitrary, made by different design tools or methods.

[0219] Moreover, for a given implementation, a single implementation of a particular functional specification of a circuit typically includes a large number of devices (e.g., millions of transistors). Thus, the sheer volume of this information makes it impractical to provide a complete recitation of the low-level structures used to implement a single implementation, much less the large number of equivalent possible implementations. To this end, the present disclosure describes the structure of a circuit using functional shorthand commonly used in the industry.

Claims

1. An apparatus comprising: a processor circuit configured to perform operations for a plurality of threads, wherein the processor circuit comprises: a plurality of lane pipelines for a plurality of lanes, wherein the lane pipeline for a given lane comprises a plurality of pipeline stages including a decode stage configured to identify one or more execution pipelines targeted by a given instruction; a plurality of execution pipelines shared by the lane pipelines and configured to perform different types of operations provided by the lane pipelines; a first scheduler circuit configured to arbitrate among threads prior to the decode stage of the lane pipelines to assign the threads to lanes based on priorities assigned to the threads; and a second scheduler circuit configured to arbitrate among lanes to assign operations from a given lane to a given execution pipeline, wherein the assignment is based on a decoding of the operations by the decode stage, and wherein the arbitration is among a greater number of the operations output by the lane pipelines than a number of available execution pipelines configured to perform operations; wherein one or more of the plurality of execution pipelines is configured to provide backpressure information to the first scheduler circuit based on an execution state; and wherein the first scheduler circuit is configured to adjust priorities of incoming threads based on the backpressure information for arbitration by the first scheduler circuit to assign to lanes.

2. The apparatus of claim 1, wherein the first scheduler circuit is configured to adjust the priorities of threads based on compiler hint information indicating types of execution pipelines targeted by threads.

3. The apparatus of claim 2, wherein the first scheduler circuit is configured to increase the priorities of threads targeting execution pipelines with backpressure below a threshold.

4. The apparatus of claim 1, wherein the threads are included in single instruction multiple data (SIMD) groups, and the first scheduler circuit is configured to assign SIMD groups to lanes.

5. The apparatus of claim 1, wherein the first scheduler circuit is configured to perform a pre-emption procedure to remove a first thread from a first lane and assign a second thread to the first lane before the first thread assigned to the first lane completes, wherein the second thread has a higher priority than the first thread.

6. The apparatus of claim 1, wherein the first scheduler circuit implements age-based arbitration among threads, and is configured to adjust initial age-based priorities of threads based on the backpressure information.

7. The apparatus of claim 1, wherein the plurality of execution pipelines comprises an integer pipeline, a floating point pipeline, and a load pipeline.

8. The apparatus of claim 1, wherein the second scheduler circuit is configured to arbitrate based on lane priority, hardware resources requested by a lane, and hardware resource availability information.

9. The apparatus of claim 1, wherein the apparatus further comprises: a dependency circuit configured to, for a first operation that depends on a prior operation using one of the execution pipelines, determine whether to stall the first operation or deactivate a thread comprising the first operation from a lane to which it is assigned based on state information from the prior operation of the one of the execution pipelines.

10. The apparatus of claim 1, further comprising: a fixed function circuit configured to control the execution pipelines to perform operations for at least one of the following types of programs: a graphics shader program; and a machine learning program.

11. The apparatus of claim 1, wherein the apparatus is a computing device, the computing device further comprising: a display; a central processing unit; and a network interface.

12. A method comprising: performing, by a computing device, operations for a plurality of threads using: a plurality of lane pipelines for a plurality of lanes, wherein the lane pipeline for a given lane comprises a plurality of pipeline stages, including a decode stage that identifies one or more execution pipelines targeted by a given instruction; and a plurality of execution pipelines shared by the lane pipelines and configured to perform different types of operations provided by the lane pipelines; arbitrating, by the computing device, among threads prior to the decode stage of a lane pipeline to assign the threads to lanes based on priority assigned to a thread; arbitrating, by the computing device, among lanes to assign an operation from a given lane to a given execution pipeline based on a decode of the operation by the decode stage, wherein the arbitration is among a greater number of the operations output by the lane pipeline than a number of available execution pipelines configured to perform operations; providing, by the computing device, backpressure information based on execution state; and adjusting, by the computing device, priority of incoming threads based on the backpressure information for arbitration to assignment to a lane.

13. The method of claim 12, wherein the adjusting is further based on compiler hint information indicating a type of execution pipeline targeted by a thread.

14. The method of claim 13, wherein the adjusting comprises increasing the priority of threads targeting an execution pipeline whose backpressure is below a threshold.

15. The method of claim 12, further comprising: prior to completion of a first thread assigned to a first lane, performing a reclamation procedure to remove the first thread from the first lane and assign a second thread to the first lane, wherein the second thread has a higher priority than the first thread. ​ 16. The method of claim 12, wherein arbitrating among threads implements age-based arbitration among lanes, and the adjusting is an adjustment to an initial age-based priority of a thread.

17. The method of claim 12, wherein arbitrating among lanes is based on lane priority, hardware resources requested by a lane, and hardware resource availability information.

18. A non-transitory computer readable storage medium having design information stored thereon, the design information specifying a design of at least a portion of a hardware integrated circuit in a format recognized by a semiconductor manufacturing system configured to use the design information to produce the circuit according to the design, wherein the design information specifies that the circuit comprises: a processor circuit configured to perform operations for a plurality of threads, wherein the processor circuit comprises: a plurality of lane pipelines for a plurality of lanes, wherein the lane pipeline for a given lane comprises a plurality of pipeline stages including a decode stage configured to identify one or more execution pipelines targeted by a given instruction; a plurality of execution pipelines shared by the lane pipelines and configured to perform different types of operations provided by the lane pipelines; a first scheduler circuit configured to arbitrate among threads prior to the decode stage of the lane pipelines to assign the threads to lanes based on priorities assigned to threads; and a second scheduler circuit configured to arbitrate among lanes to assign operations from a given lane to a given execution pipeline, wherein the assignment is based on a decoding of the operations by the decode stage, and wherein the arbitration is among a greater number of the operations output by the lane pipelines than a number of available execution pipelines configured to perform operations; wherein one or more of the plurality of execution pipelines is configured to provide backpressure information to the first scheduler circuit based on an execution state; and wherein the first scheduler circuit is configured to adjust a priority of an incoming thread based on the backpressure information for use in arbitration of the incoming thread by the first scheduler circuit to assignment to a lane.

19. The non-transitory computer readable storage medium of claim 18, wherein the first scheduler circuit is configured to adjust the priority of a thread based on compiler hint information indicating a type of execution pipeline targeted by a thread.

20. The non-transitory computer readable storage medium of claim 18, wherein the first scheduler circuit is configured to perform a usurpation procedure to remove a first thread from a first lane and assign a second thread to the first lane prior to completion of the first thread assigned to the first lane, wherein the second thread has a higher priority than the first thread.

Images