List-based single-instruction, multiple-thread computing architectures

Abstract

Systems and methods are described herein for a list-based single-instruction, multiple-thread (SIMT) processing system. The list-based SIMT system executes a program across many threads by organizing threads into variable-length thread lists. Each thread list includes thread contexts and a program counter common to the thread list. The system includes a scheduler to select thread lists, an instruction handler to fetch and decode using the program counter, and execution hardware to apply the decoded instruction one thread context at a time. A list manager handles divergence by dividing diverging thread contexts of a thread list into two or more child thread lists for different next program counters. The list manager also performs reconvergence by merging thread lists that satisfy a reconvergence condition.

Description

LIST-BASED SINGLE-INSTRUCTION, MULTIPLE-THREAD COMPUTING ARCHITECTURESTECHNICAL FIELD

[0001] This disclosure relates to single-instruction, multiple-thread (SIMT) computing architectures. More generally, this disclosure relates to hardware architectures for thread-level computing parallelism for use in massively parallel computing applications.BRIEF DESCRIPTION OF THE DRAWINGS

[0002] FIG. 1 illustrates a block diagram of an example single-instruction, multiple-thread (SIMT) microarchitecture with fixed-width warps.

[0003] FIG. 2 illustrates a block diagram of an example divergence branch in an SIMT architecture, according to various examples.

[0004] FIG. 3 illustrates a table of list elements that are part of a thread list, according to one embodiment.

[0005] FIG. 4 illustrates a block diagram of a list-based SIMT microarchitecture, according to one embodiment.

[0006] FIG. 5 illustrates a block diagram of an example of a thread list data structure with a list array and a thread array, according to one embodiment.

[0007] FIG. 6 illustrates a block diagram of a cold context table (CCT) in which program counters are mapped to a list array index, according to one embodiment.

[0008] FIG. 7 illustrates a graph of the minimum list count with respect to the list length, according to one embodiment.

[0009] FIG. 8 illustrates a group of independent list-based SIMT engines with independent schedulers and register files, according to one embodiment.DETAILED DESCRIPTION

[0010] Single-instruction, multiple-thread (SIMT) processors are designed to efficiently process parallel workloads that execute the same instruction sequence over many independent threads. Instead of provisioning a complete front end for every thread, a SIMT engine shares an instruction fetch and decode path across a group of threads while replicating the execution resources that operate on per-thread data. Many existing SIMT engines operate with a shared front1NEU 1500end organized around “warps” of threads having fixed widths that advance through the program together. Conventional SIMT microarchitectures maintain explicit state for each active warp and schedule warps each cycle. A warp table maintains a set of warps. Each warp includes a warp identifier, an active mask indicating which threads of the warp are currently participating, and a program counter (PC) that identifies the next instruction to fetch. A traditional SIMT architecture includes front-end circuitry (e.g., hardware, firmware, microcode, etc.) that fetches and decodes an instruction using the program counter of the selected warp. A register file supplies per-thread operands, and a set of parallel execution lanes applies the decoded instruction across the active threads of the warp. Memory operations are served through a memory access unit into private or global caches, and results are written back to the register file.

[0011] In the context of SIMT engines, spatial (or “horizontal”) SIMT refers to executing a single decoded instruction in parallel across multiple threads of a warp (e g., with a fixed width) by replicating execution units as “lanes.” The instruction is fetched and decoded once, then broadcast to the warp, and an active mask determines which thread lanes participate for that instruction. Temporal (or “vertical”) SIMT executes the same instruction sequentially for multiple threads over successive cycles on a single execution unit or a smaller number of units. From a performance perspective, temporal execution is effectively a transpose in time of spatial execution. That is, rather than using N lanes operating in parallel in one cycle, one lane may operate for N dispatch cycles. Parallelism is obtained by executing multiple warps across multiple lanes in parallel. A hybrid spatio-temporal architecture consists of several lanes designed for spatial parallelism, with each lane capable of processing multiple threads over time. An SIMT engine may be configured with a spatial architecture, a temporal architecture, or a hybrid spatio-temporal architecture, depending on the target design trade-offs for throughput, area, power, cost, and other factors.

[0012] FIG. 1 illustrates a block diagram of an example single-instruction, multiple-thread (SIMT) microarchitecture 100 with fixed-width warps. As illustrated, the microarchitecture 100 includes a scheduler 102 coupled to a warp definition table 104. The example warp definition table contains entries for a Warp ID, an Active mask, and a program counter (PC). The microarchitecture 100 includes an instruction cache (I-cache) 106 in communication with a fetch unit 108 and a decoder 109. The register files 120 are used to store per-thread data, and a set of parallel execution lanes 130 and 170 interacts with a memory access unit 160. The memory access unit 160 services 2NEU 1500requests to a global memory data cache 140 and a private memory 150. Register results are returned through the writeback module 180. Programming models allow a programmer to create thousands of threads, each executing the same code. Prior to execution, threads are grouped into fixed-size batches (warps).

[0013] The conventional SIMT microarchitecture 100 implements a processing flow in which the scheduler 102 selects a warp from the warp definition table 104, the front-end circuitry (instruction cache 106, fetch unit 108, decoder 109, etc.) fetches and decodes using the selected program counter, the register files 120 supply operands, and the execution lanes 130 and 170 apply the instruction across active threads indicated by the active mask of the selected warp from the warp definition table 104. The register files 120 store per-thread registers. Thus, if the warps are N threads wide, the engine supports a maximum of M active warps, and each thread has up to R registers, then the total register capacity is M x N x R registers. The execution units (collectively referred to as lanes 130 and 170) provide the execution resources to apply the decoded instructions.

[0014] In existing SIMT architectures, the warp width is fixed, including, for example, 32 or 64 threads, and the active mask in the warp entry determines which of those threads participate in a given instruction issue. For instance, if all bits in the active mask are set (e.g., “1”), all threads execute the instruction in lock-step. However, when some bits are clear (e.g., “0”), the corresponding lanes skip that instruction. The scheduler 102 sequentially advances the warps from the warp definition table 104 by consulting the per- warp program counter, fetching and decoding the next instruction, and supplying the available operands to the execution lanes 130 and 170.

[0015] FIG. 2 is a block diagram 290 illustrating an example of branch divergence in an SIMT architecture, according to various examples. The example is oversimplified with warps including only four threads for illustration purposes, but the same concept is applicable to warps having larger warp widths. As illustrated, the program counter PC: A 292 begins with an active mask “1111,” indicating that all four threads of the warp are participating. A branch at PC: A 292 diverges the warp so that three threads take PC: B 295 (mask “1011”) while another thread takes PC: C 297 (mask “0100”). After the distinct paths are complete, the control flow reconverges at PC: D 299, with the mask again showing “1111.”

[0016] Branch divergence is a significant performance impediment in existing SIMT engines. Most existing SIMT engines serialize the distinct paths of a diverged warp, which results in a decrease in throughput. The decrease in throughput is on average 50% for warp divergence 3NEU 1500involving a two-way split. Prior work and advancements in SIMT microarchitecture have focused on techniques to lessen the penalty of branch divergence, while generally relying on the same overall warp structures with fixed widths. Some SIMT architectures have been proposed that execute both sides of a branch simultaneously; however, these architectures are generally impractical, overly complex, inefficient for other reasons, and / or too costly.

[0017] Accordingly, while existing SIMT engines can achieve high performance on coherent workloads, there remains a need for improved architectures that reduce or eliminate the throughput losses caused by divergent branches, without incurring significant complexity or changes to the scheduler, front-end circuitry, or memory system.

[0018] Systems, methods, and architectures are described herein for various embodiments of list-based SIMT engines. The list-based SIMT engine provides a processing system for the parallel execution of a program across a plurality of threads, while efficiently handling divergent branches with minimal reductions in throughput losses. In contrast with traditional architectures that use fixed-width warps, the presently described systems and methods group threads into variable-length lists (i.e., thread lists having a variable number of thread contexts in each respective thread list).

[0019] In various embodiments, a list-based SIMT engine organizes threads into variablelength thread lists. Each thread list includes a set of thread contexts that specify per-thread local data and a program counter that identifies the next instruction, which is common to all thread contexts in that thread list. A scheduler maintains and selects among multiple thread lists, an instruction handler fetches and decodes the instruction identified by the program counter of the selected thread list, and execution hardware applies the decoded instruction. In various embodiments, the execution hardware includes one or more execution lanes to apply the decoded instruction temporally. In such embodiments, the engine applies the decoded instruction one thread context at a time and produces per-thread results. In some embodiments, the scheduler dispatches different selected thread lists to multiple lanes so that several thread lists can progress in parallel. In some embodiments, the instruction handler includes multiple parallel fetch and / or decode pipelines to increase the dispatch bandwidth. Additional details, variations, and examples are provided below in the context of figures.

[0020] As described herein, a list manager may handle control transfer instructions (CTIs), such as calls, returns, and branches. The list manager monitors outcomes when the thread contexts of a list evaluate a CTI, such as a branch instruction. If the evaluation selects different next program4NEU 1500counters, the list manager divides the existing thread list into one or more child thread lists. Each child thread list is associated with one of the next program counters. The list manager moves each thread context from the original (parent) list into one of the new child thread lists based on the selected next program counter. For example, for a set of thread contexts in the parent thread list, a first subset of the thread contexts may be moved to a first child thread list, and a second subset of the thread contexts may be moved to a second child thread list.

[0021] The scheduler is updated (e.g., configured) to independently schedule the child thread lists. For example, a branch instruction may result in two different next program counters that correspond to different branch targets. The list manager creates a child thread list for each unique branch target and sets the program counter of each child thread list to that target. Thus, for any existing thread list, including all created child thread lists, all the thread contexts of that thread list are converged at the program counter of that thread list when the instruction is executed. Again, additional details, variations, and examples are provided below in the context of figures.

[0022] The list manager may also detect when two or more thread lists satisfy a reconvergence condition and merge them together. As described herein, reconvergence and scheduling may be guided by program counter values and call stack depth indicators, with priority given to lists having numerically smaller call stack depths. In some embodiments, thread lists may be maintained with compact data structures. For example, a list array may be maintained that, for each thread list, stores a thread count, a head pointer, and a tail pointer. Additionally, a thread array may be maintained with thread descriptors that have a link or index to the next thread. As such, each thread list may be maintained as a linked list. In some embodiments, a doubly linked list may be used in which the thread descriptors have a link to a previous thread. In some embodiments, the list manager may restrict the length of a thread list to a predetermined or predefined maximum. This may result in multiple thread lists with the same program counter. In some embodiments, the list manager may maintain the thread lists under the maximum length by rebalancing the thread contexts among thread lists. Again, additional details, variations, and examples are provided below in the context of figures.

[0023] Some of the examples described herein implement list reconvergence algorithms and approaches that are adapted from stackless convergence algorithms for existing, fixed-width warp SIMT engines. For example, the list manager may utilize a reconvergence condition based on program counter values and call stack depth indicators. Additionally, the list manager may 5NEU 1500implement a thread group mechanism in which child thread lists inherit a thread group identifier from a parent thread list. The list manager and / or scheduler may maintain a hot context table (HCT) that, for each thread group, identifies a thread list having the lowest pair of program counter and call stack depth values. The scheduler may prioritize thread lists that have numerically smaller call stack depth indicators and, if they are equal, the smaller program counter.

[0024] In some embodiments, the scheduler and / or list manager use the hot context table (HCT) together with a cold context table (CCT). The HCT holds thread lists that are eligible for immediate scheduling on execution lanes. The CCT identifies thread lists eligible for reconvergence but not immediately available for selection by the scheduler for execution. Additional details of the HCT and CCT are described in the context of FIG. 6 below. Each HCT entry and each CCT entry identifies at least a program counter. In some embodiments, each HCT and CCT entry further includes a thread group identifier (groupID), a call stack depth indicator (SP), and the program counter. No two thread lists share the same stack depth, program counter, and groupID across the HCT and CCT, because any such thread lists would be merged instead of co-existing.

[0025] The scheduler selects the next thread list from the HCT to execute based on a priority policy that sorts first by lowest stack depth and then, in the event of a tie, by the numerically smallest program counter. The instruction handler fetches and decodes the program code of the selected thread list. After the decoded instruction is evaluated for the thread list (including any call / return activity that changes the depth indicator), the program counter of the thread list (and the depth indicator, if applicable) is advanced. The list manager probes the CCT for a candidate to reconverge by searching for a matching key (groupID, depth indicator, program counter). If the probe hits, the list manager merges the just-executed HCT list with the matching CCT list by, for example, concatenating their descriptor chains. The merged list remains hot in the HCT and continues to be eligible for scheduling.

[0026] If the CCT probe misses, no merge occurs. The executing list remains in the HCT with its updated depth indicator and program counter and can be selected again according to the policy above. The CCT thus functions as a reconvergence scavenger rather than a general ready queue for thread lists. In various implementations, a swap between HCT and CCT occurs only when an HCT entry becomes hard-stalled (e.g., waiting at a barrier, a long-latency event, or a resource condition that prevents execution). In such situations, the stalled thread list may be moved from 6NEU 1500the HCT and “parked” in the CCT under its current (groupID, depth indicator, and program counter) key, and a compatible CCT entry may be promoted to the HCT. Otherwise, the scheduler continues selecting from the remaining HCT entries. In some embodiments, the groupID field is initialized when a program spawns its initial thread lists, with child thread lists inheriting the parent groupID upon divergence. Restricting reconvergence to lists that share the same groupID prevents uncontrolled growth into a single, long list across independent groups and enables scalable parallelism when multiple groups are active.

[0027] Additional understanding and details relating to the thread group concepts, hot context tables, and stackless convergence algorithms are described in the following publications, each of which is hereby incorporated by reference in its entirety: Damien Collange and G. E. T. et al. Stack-less SIMT reconvergence at low cost. Tech. Rep. INRIA, Sept. 2011. url: https: / / hal.science / hal-00622654vl / file / ; Sylvain Collange, Nicolas Sergent, and Jean-Daniel Fekete. “Simty: generalized SIMT execution on RISC-V” In: Proceedings of the Workshop on Computer Architecture Research with RISC-V (CARRV). 2017, pp. 1-6.

[0028] As used herein, the term “list manager” is used to refer to control logic that maintains the plurality of thread lists (e g., the list array and thread array) and that performs list operations, including creating child thread lists, moving thread descriptors between thread lists, merging thread lists on reconvergence, updating per-list program counters, updating thread counts, and updating the scheduler state. The list manager may be implemented within the scheduler, within the thread list maintaining circuitry (e.g., the list state), or as a separate module, circuitry, or subsystem. The functions may be realized in hardware logic, microcode, or firmware on one or more embedded control processors.

[0029] In some embodiments, the scheduler and / or the list manager are implemented as software, firmware, or microcode executed by one or more embedded control processors. Instructions are executed to perform thread list creation, movement, and merging by updating entries in, for example, the list array and thread array, and by setting per-list program counters, while the instruction handler and execution lanes operate as described herein. For example, the scheduler and / or the list manager may be implemented, at least in part, within a non-transitory computer-readable storage medium as instructions that, when executed by one or more processors embedded in or operably coupled to a processing system having an instruction handler and7NEU 1500execution hardware, cause the processing system to perform the scheduling and / or list management operations described herein.

[0030] Many existing computing systems, methods, and devices can be used in combination with the systems and methods described herein. Some of the infrastructure that can be utilized with embodiments disclosed herein is already available, including general-purpose computers, computer programming tools and techniques, digital storage media, and communication links. A computing device or controller may include a processor, such as a microprocessor, a microcontroller, or logic circuitry.

[0031] A processor, as used herein, may include a special-purpose processing device, such as application-specific integrated circuits (ASIC), programmable array logic (PAL), programmable logic array (PLA), programmable logic device (PLD), field programmable gate array (FPGA), or other customizable and / or programmable devices. The computing device may also include a machine-readable storage device, such as non-volatile memory, static RAM, dynamic RAM, ROM, CD-ROM, disk, tape, magnetic, optical, flash memory, or other machine-readable storage medium. Various aspects of certain embodiments may be implemented using hardware, software, firmware, or a combination thereof.

[0032] The components of the disclosed embodiments, as generally described and illustrated in the figures herein, could be arranged and designed in a wide variety of different configurations. Furthermore, the features, structures, and operations associated with one embodiment may be applicable to or combined with the features, structures, or operations described in conjunction with another embodiment. In many instances, well-known structures, materials, or operations are not shown or described in detail to avoid obscuring aspects of this disclosure.

[0033] FIG. 3 illustrates a table of list elements that are part of a thread list 300, according to one embodiment. As illustrated, the thread list 300 includes a program counter 310 that points to the next instruction to execute, a count 320 of the number of threads in the thread list, and a list 330 with an array of thread contexts. Each thread context specifies, for example, private, per-thread local data (e.g., private, per-thread register files). The program counter 310, count 320, and list 330 together provide a logical state for list-based execution. In some embodiments, the list elements of a thread list 300 may include only the program counter 310 and the list 330. Unlike warp-based SIMT engines, no active mask is utilized since all thread contexts in a given thread8NEU 1500list are converged at the program counter 310. That is, the program counter 310 is common to every thread context of the thread list 300.

[0034] The program counter 310 is used by front-end circuitry (e.g., instruction cache, a fetch unit, a decoder, etc.) to fetch and decode the next instruction to be executed for the thread list. All thread contexts are converged when the instruction is executed. However, evaluating the instruction may cause different per-thread outcomes that select different next program counters, such as different branch targets. In response, the list manager creates child thread lists and assigns the program counter 310 of each child thread list to a corresponding next program counter. This ensures that each resulting child thread list remains converged at its own program counter 310. As described herein, when a reconvergence condition is satisfied, two or more thread lists may be merged so that their thread contexts execute at a common program counter 310.

[0035] In some embodiments, including the illustrated example, the thread list includes a count 320 that tracks the current number of thread contexts in the thread list 300. The count 320 may be updated automatically as thread contexts are added or removed as a result of divergence, reconvergence, thread termination, or rebalancing. The count 320 can be used by a list manager to enforce a maximum length limit and / or by a scheduler to implement a fairness policy. For example, the scheduler may bias selection toward longer lists so that threads make uniform forward progress.

[0036] In various embodiments, the list 330 identifies the membership and / or order of thread contexts in the thread list 300. The simplified embodiment of the thread list 300 depicted only has three list elements. However, it is appreciated that some embodiments may associate additional metadata with each thread list 300, such as per-list identifiers, thread group identifiers, call stack depth indicators, and the like, as described herein.

[0037] FIG. 4 illustrates a block diagram of a list-based SIMT microarchitecture 400, according to one embodiment. A scheduler 402 manages a plurality of thread lists whose state, including the depicted list heads and program counters, is maintained in list state 404. The scheduler 402 selects a thread list for execution and provides the program counter of the selected thread list to the front-end subsystems, modules, or circuitry, including an instruction cache 406, a fetch unit 408, and a decoder 409. These components fetch and decode the instruction identified by the program counter of the selected thread list. Again, the list state 404 does not include an active mask because, by definition, all the thread contexts in any given thread list are converged at the program counter of that thread list.9NEU 1500

[0038] Operands for the decoded instruction are read from register banks 420. As illustrated and according to various embodiments, the register banks 420 may be arranged in columns and rows to enable parallel operand access for multiple selected thread lists. Operands are transported through a register crossbar 425 to the execution hardware, which includes one or more execution lanes 430 and 470. The illustrated architecture and scheduler functionality allow for multiple thread lists to be in process while operands are read in parallel and routed to the appropriate lanes for per-thread execution.

[0039] The execution hardware applies the decoded instruction temporally, one thread context at a time per lane, and produces per-thread results. Memory operations for the executing thread contexts are serviced by a memory access unit 460 coupled to a global-memory data cache 440 and / or a private memory 450. Results are provided to the writeback module 480. In some embodiments, the list-based SIMT microarchitecture 400 may include more than one scheduler 402 and any number of fetch units 408, decoders 409, etc., to increase the dispatch bandwidth.

[0040] The list manager, as described herein, may be implemented inside the scheduler 402 or as a separate subsystem coupled to the list state 404. As described herein, the list manager monitors outcomes when the thread contexts of a selected thread list evaluate the decoded instruction. If the evaluation selects different next program counters, such as different branch targets, the list manager creates one or more child thread lists. The list manager sets the program counter of each child thread list to the corresponding target and moves the appropriate thread contexts into the respective child thread lists. The list manager may also handle thread list reconvergence by evaluating lists to determine if they satisfy a reconvergence condition. If the reconvergence condition is met, the list manager may merge two or more thread lists so that their thread contexts converge at a common program counter.

[0041] The list manager may use various reconvergence policies and reconvergence condition criteria. For example, the scheduler 402 and / or list manager may evaluate reconvergence and scheduling priority using program counter values and call stack depth indicators (e.g., the stack pointer depth). The system may give priority to thread lists with smaller call stack depths and / or smaller program counters. In some embodiments, a thread group mechanism is used in which child thread lists inherit a thread group identifier from the parent. For each thread group, the system may use a “hot context table” that tracks the thread list having the lowest stack-depth value for next execution and, in the event of a tie, the thread list with the lowest program counter.10NEU 1500

[0042] The execution hardware may issue the next decoded instruction for a selected thread list without waiting for the prior instruction to be completed by all the other thread contexts of the thread list. The system may use modulo scheduling to statically schedule basic blocks onto each execution unit. This increases the net throughput of the engine but also increases pressure on the memory system because the dispatch rate, Amem, scales linearly with the total instructions per clock (IPC). In some embodiments, the system may use a scalar execution optimization to detect when one or more operands are uniform across all thread contexts of the selected thread list. In such instances, the system may execute the instruction once and propagate the result to every thread context in that thread list. These optimizations, along with other known techniques in the art of SIMT engines, can be used to reduce latency and improve overall throughput in the presently described list-based SIMT architectures.

[0043] In various embodiments, the list-based SIMT engine supports superscalar execution in addition to or instead of the scalar execution optimization described herein. In such embodiments, the instruction handler and scheduler cooperate to dispatch two or more decoded instructions per cycle to disjoint execution resources. Superscalar execution may be combined with multiple fetch / decode pipelines and with modulo scheduling so that independent instructions from a selected thread list can overlap in flight while other thread lists continue to make forward progress on separate lanes.

[0044] FIG. 5 illustrates a block diagram of an example of a thread list data structure 500 with a list array 510 and a thread array 520, according to one embodiment. In various embodiments, the list array 510 stores a thread count field NTH 512, a head pointer THead 514, and a tail pointer TTail 516 for each thread list. The thread array 520 stores thread descriptors <desc> 522 and 526 in conjunction with corresponding thread index fields TIDX 524 and 528 that link to the next thread in the list. In some embodiments, an end-of-list condition may be encoded with a reserved value or a dedicated end bit. In some embodiments, a doubly linked list is used in which each thread descriptor also includes a link to a previous thread, which may facilitate deletions.

[0045] The system may determine that the evaluation of an instruction causes the thread contexts of a thread list to select different next program counters. In such instances, the list manager allocates a new entry in the list array 510 for each child thread list, assigns the program counter of the child thread to the corresponding next program counter, initializes THead 514 and11NEU 1500TTail 516, and moves the thread descriptors from the parent thread list to the child thread lists by updating TIDX 524 and 528.

[0046] The system also monitors for satisfaction of a reconvergence condition by two or more thread lists. When identified, the list manager merges the linked lists of two or more thread lists by updating the THead 514, TTail 516, updating the TIDX values 524 and 528, and increasing the NTH value 512. In various embodiments, pipeline pointer update operations may include list collectors to coalesce and reorder descriptor movements between lists. As described herein, in some embodiments, the list manager enforces a maximum thread list length. When the limit is reached, additional thread lists sharing the same program counter may be maintained. In some examples, rebalancing logic may be used to redistribute or otherwise rebalance thread contexts among different lists that have the same program counter.

[0047] FIG. 6 illustrates a block diagram 600 of a cold context table (CCT) 610 in which program counters are mapped to a list array index, according to one embodiment. As illustrated, the CCT 610 associates a program counter field 612 with a list array index (“LAI”) 614. The CCT 610 identifies thread lists eligible to reconverge with thread lists in a just-executed, hot context table (HCT), as previously described. The list array index 614 identifies which entry of a list array 620 is associated with, at least, a specific program counter 612. In some embodiments, in addition to the program counter 612, the CCT 610 may further identify a groupID and / or a stack pointer depth (SP) (e.g., a “stack depth”). For example, in some embodiments, the CCT 610 may include a staker pointer depth (SP) that can be used as a sorting index. Each record of the list array 620 includes a thread count field NTH 622, a head pointer THead 624, and a tail pointer TTail 626, as described in conjunction with FIG. 5. The list array 620 includes references to a thread array 630, which stores per-thread descriptors 632 and 636 and respective thread indices TIDX 634 and 638, as described in conjunction with FIG. 5. Each thread list can be traversed, extended, merged, or split by pointer manipulation of the THead 624, TTail 626, and the TIDX links 634 and 638.

[0048] In various embodiments, the scheduler maintains a HCT that includes thread lists eligible for immediate scheduling. In contrast, the CCT serves as location to store thread lists that are eligible and / or likely candidates for reconvergence, but not immediately eligible for scheduling. In various embodiments, the scheduler and / or list manager implement a reconvergence algorithm using the HCT and the CCT. For example, the scheduler may select a thread list from the HCT (e.g., based on the thread list with the lowest stack depth and, in the case of thread lists 12NEU 1500having equal stack depths, based on the smallest program counter). After execution of the decoded instruction, the list manager updates the program counter and the stack depth if it has changed. The scheduler continues to select thread lists from the HCT for execution. A thread list from the CCT is moved into the HCT when HCT entries are hard-stalled or otherwise not executable anymore (e.g., at a barrier). Otherwise, the scheduler continues to select thread lists from only the HCT for execution.

[0049] During the continuous selection and execution of thread lists from the HCT, the system evaluates the CCT 610 for a thread list with a matching program counter (and, optionally in some embodiments, a matching groupID). If a match is found, the list manager merges the thread list from the HCT and the identified matching thread list from the CCT into a single thread list with a common program counter by concatenating or otherwise combining their respective descriptor chains in the respective thread arrays 630, updating the thread count field NTH 622, a head pointer THead 624, and a tail pointer TTail 626 in the list array 620, and removing the CCT entry. The merged thread list remains “hot” in the HCT and continues execution.

[0050] In some embodiments, each thread list is further associated with a groupID. A program may be initiated with some number of thread lists, each of which has a unique groupID. As a thread list diverges and is split into multiple child thread lists, each child thread list is populated with the same groupID. In such embodiments, the reconvergence algorithm evaluates thread lists within the CCT for reconvergence with a thread list in the HCT to identify marching program counters and a matching groupID. Thus, reconvergence is constrained by the groupID, such that only thread lists that share the same groupID are eligible to merge, even if they share the same program counter. Restricting reconvergence by groupID prevents unbounded growth into a single long list while preserving parallism across multiple thread groups.

[0051] Accordingly, stackless reconvergence techniques for fixed-width warps can be adapted for use in the presently described list-based SIMT architectures by applying the general principles to thread lists. Reconvergence is framed as merging entire variable-length thread lists rather than simply toggling the masks on a per-warp basis. Using a CCT ensures reconvergence occurs opportunistically (and locally in embodiments using a groupID) and without disrupting execution progress of thread lists on the HCT.

[0052] In the context of a quantitative model of a list-based SIMT processor, if an instruction for a thread is sent every cycle, a list of length N will take N cycles to send. If the execution latency 13NEU 1500per thread is Lecycles, the net execution time from the first thread dispatch to the last thread completion time is N + Lecycles. Assuming no overlap in the execution of instructions, the perlist throughput averages:

[0054] In Equation 1, the throughput, A, is measured in thread-instructions per cycle. With E execution units, the aggregate throughput is:

[0056] To feed the E execution units, an excess of threads is needed to cover memory latency and / or other types of per-thread stall. For each list executed at a rate of A^, an average 8LDof them will incur a memory access and stall for Lmemcycles, where Lmemrepresents the average memory latency and 6LDrepresents the average fraction of instructions that are load instructions that cause memory stalls. Assuming fully pipelined memory loads, the per-list throughput of these lists is:

[0058] A list ready queue includes thread lists that are ready for execution. Depending on the instruction class, the thread lists are sent to the execution units (density = (1 — A5) or to the memory pipeline (density = A5LD). Once delivered, a thread list takes time N + Le, or N + Lmem, to enter and then exit their respective pipelines. From there, they return to the list ready queue.

[0059] Complete occupancy of the execution units can be achieved if the list ready queuehas enough ready lists that a ready list can be dispatched after N cycles from the last dispatch. For E units, a thread list must be sent for every - cycle. Inverting this, the list throughput to the execution units, measured in lists per cycle, is as follows:

[0061] The number of lists “in flight” is governed by Little’s Law, N = AL, where N is the number in flight, A is the throughput, and L is the latency of the server process. For the execution units, this is expressible as:

[0063] LCerepresents the count of the lists allocated to the execution units. The implication of Equation 5 is that as N approaches infinity, we need E execution unit-ready lists to keep the14NEU1500execution units busy. Conversely, as lists become short, Lebecomes dominant, and we need more than E X ELeexecution-unit-ready lists.

[0064] With respect to keeping the memory pipeline occupied, the ratio of memory instructions to non-memory instructions is:

[0066] With lists sent to execution units at a rate A(e, lists can be sent to the memory unit at a rate Amem, as follows:

[0068] Reapplying Little’s Law with memory system latency as the delay, the number of lists in flight in the memory system is computed as:

[0070] The summation of the two occupancies is expressible as:

[0072] FIG. 7 illustrates a graph 700 of the minimum list count with respect to the list length for various memory latencies, according to one embodiment. The curves are generated from Equation 9 for a representative load-density SLDof 0.15 and a one execution unit-ready list (£ = 1). The trend indicates that increasing list length reduces the minimum number of lists that the scheduler must maintain to saturate the engine. In contrast, short lists dramatically increase the number of execution ready lists needed to saturate the engine, especially with increased memory latency. In various embodiments, the system is operated with a target mean list length between approximately 64 and 128, since smaller list lengths do not dramatically affect the total thread count TC, but rapidly increase the number of lists LC.

[0073] FIG. 8 illustrates a group 800 of independent list-based SIMT engines 810 and 811 with independent schedulers 402, register banks 420, and a local set of execution lanes 430 according to one embodiment. Each of the K independent list-based SIMT engines 810 and 811 maintains its own CCT 610, list array 620, and thread array 630 as described in conjunction with FIG. 6. Each list-based SIMT engine is separate from the others and employs its own private data lanes. Assuming each list-based SIMT engine has many fewer lanes than the overall machine, the grouping of independent list-based SIMT engines 810 and 811 simplifies routing data and threads.15NEU 1500

[0074] The presently described systems and methods of list-based STMT engines can be used in various applications. Examples of applications include irregular, control-intensive data-parallel tasks such as string and pattern search, parsing, and general “branchy” kernels, machine learning inference, high-throughput general-purpose compute kernels, and graphics workloads. As a specific example, a list-based STMT engine can be applied to massively parallel graphics workloads, such as fragment / pixel shading workloads, and / or as a general-purpose graphics processing unit (GPU). In some such embodiments, a scheduler maintains many ready thread lists corresponding to pixels or fragments executing the same shader at a common program counter, while execution lanes apply decoded instructions temporally to one thread context at a time. When the shader control flow diverges for conditional texturing or material -dependent branches, the list manager partitions the parent thread list into one or more child thread lists, with one child thread list for each branch target. As described herein, each child thread list remains converged at its own program counter. Because the engine does not idle lanes pursuant to an active mask, divergent paths proceed independently and reconverge when a reconvergence condition is met. Accordingly, the list-based STMT engine mitigates or eliminates the throughput losses exhibited by STMT engines that use fixed-width warps on “branchy” graphics code.

[0075] This disclosure has been made with reference to various exemplary embodiments, including the best mode. However, those skilled in the art will recognize that changes and modifications may be made to the exemplary embodiments without departing from the scope of the present disclosure. While the principles of this disclosure have been shown in various embodiments, many modifications of structure, arrangements, proportions, elements, materials, and components may be adapted for a specific environment and / or operating requirements without departing from the principles and scope of this disclosure. These and other changes or modifications are intended to be included within the scope of the present disclosure.

[0076] This disclosure is to be regarded in an illustrative rather than a restrictive sense, and all such modifications are intended to be included within the scope thereof. Likewise, benefits, other advantages, and solutions to problems have been described above with regard to various embodiments. However, benefits, advantages, solutions to problems, and any element(s) that may cause any benefit, advantage, or solution to occur or become more pronounced are not to be construed as a critical, required, or essential feature or element.16NEU 1500

Claims

What is claimed is:

1. A processing system for parallel execution of a program across a plurality of threads, the system comprising: a scheduler configured to maintain and select from a plurality of thread lists, each thread list including: a variable number of thread contexts, each of which specifies per-thread local data, and a program counter that identifies an instruction to execute next, where the program counter is common to all the thread contexts of the thread list; an instruction handler configured to fetch and decode, for each thread list selected by the scheduler, the instruction identified by the program counter of that thread list; execution hardware with a plurality of execution lanes for spatio-temporal execution of decoded instructions of the selected thread lists to produce per-thread results; and a list manager configured to: detect that evaluation of the instruction by the thread contexts of the selected thread list selects different next program counters and, in response, (i) create two or more child thread lists, each child thread list being associated with a respective one of the selected next program counters and a subset of thread contexts that selected the corresponding next program counter, and (ii) update the scheduler such that the child thread lists are scheduled independently of one another, and detect that two or more thread lists satisfy a reconvergence condition and, in response, merge the two or more thread lists into a single thread list for subsequent execution at a common program counter.

2. The system of claim 1, further comprising: a register array organized as a plurality of banks arranged in columns and rows to enable parallel operand access for multiple scheduled thread lists, and a register crossbar configured to route read operands to corresponding execution lanes.

3. The system of claim 1, wherein the list manager is implemented as one of:17NEU 1500a subsystem of the scheduler, and a subsystem separate from the scheduler.

4. The system of claim 1, wherein for any one of the thread lists, all thread contexts of that thread list are converged at the program counter of that thread list when the instruction is executed.

5. The system of claim 1, wherein, responsive to the instruction handler decoding a control transfer instruction (CTI) for which evaluation of the CTI by the respective thread contexts resolve to different next program counters that correspond to different control flow targets, the list manager is configured to: create a child thread list for each unique next program counter, set a program counter of each child thread list to a respective next program counter, and include, in each respective child thread list, the subset of thread contexts for which the CTI resolved to the corresponding next program counter.

6. The system of claim 1, wherein the instruction handler comprises a plurality of parallel fetch and decode pipelines and the scheduler is configured to distribute selected thread lists among the pipelines.

7. The system of claim 1, further comprising a scalar execution subsystem configured to determine that one or more operands of the decoded instruction are uniform across all thread contexts of the selected thread list and, in response, execute the decoded instruction once and propagate a common result to all thread contexts of the selected thread list.

8. The system of claim 1, wherein the execution hardware is configured to issue a next decoded instruction for the selected thread list without waiting for completion of execution of the decoded instruction by all thread contexts of that thread list.

9. The system of claim 8, wherein the execution hardware employs modulo scheduling.18NEU 150010. The system of claim 1, wherein the reconvergence condition is evaluated using program counter values and call stack depth indicators associated with thread lists, and wherein the scheduler prioritizes thread lists having numerically smaller call stack depth indicators.

11. The system of claim 10, wherein the list manager is further configured to: implement a thread group mechanism in which child thread lists inherit a thread group identifier from a parent thread list, and maintain a hot context table that, for each thread group, identifies a thread list having a lowest stack depth value and, in the case of a equal stack depth values, a thread list having a smallest program counter.

12. The system of claim 1, wherein the list manager is configured to represent each thread list as a linked list by maintaining: a list array that stores, for each thread list, a thread count, a head pointer, and a tail pointer, and a thread array that stores thread descriptors corresponding to the thread contexts, each thread descriptor including a link identifying a next thread in the thread list.

13. The system of claim 12, wherein the list manager is configured to represent each thread list as a doubly linked list, with each thread descriptor further including a link to a previous thread in the thread list.

14. The system of claim 1, wherein the list manager restricts a length of a thread list to a predefined maximum, and wherein the list manager is configured to handle multiple thread lists that share a common program counter.

15. The system of claim 14, wherein the list manager rebalances thread contexts among thread lists to maintain thread list lengths less than the predefined maximum.

16. A method for parallel execution of a program across a plurality of threads, comprising:19NEU 1500maintaining, by a scheduler, a plurality of thread lists, each thread list including a variable number of thread contexts that specify per-thread local data and a program counter that identifies an instruction to execute next, the program counter being common to all thread contexts of the thread list; selecting, by the scheduler, at least one thread list; fetching and decoding, by an instruction handler, the instruction identified by the program counter of each selected thread list; executing, by execution hardware with a plurlatiy of execution lanes for spatio-temporal execution, the decoded instruction for each selected thread list to produce per-thread results; detecting that evaluation of the instruction by the thread contexts of the selected thread list selects different next program counters and, in response: creating, via a list manager, two or more child thread lists, each associated with a respective one of the selected next program counters and including a subset of the thread contexts, and updating the scheduler such that the child thread lists are scheduled independently of one another; and detecting that two or more thread lists satisfy a reconvergence condition and, in response, merging, via the list manager, the two or more thread lists into a single thread list for subsequent execution at a common program counter.

17. The method of claim 16, further comprising: dispatching different selected thread lists to different execution lanes such that multiple thread lists are executed contemporaneously.

18. The method of claim 16, further comprising: accessing operands from a register array organized as a plurality of banks arranged in columns and rows to enable parallel operand access for multiple scheduled thread lists, and routing read operands to corresponding execution lanes via a register crossbar.20NEU 150019. The method of claim 16, wherein executing the decoded instruction for any one of the thread lists is performed with all thread contexts of that thread list converged at the program counter of that thread list.

20. The method of claim 16, wherein, responsive to the instruction handler decoding a control transfer instruction (CTI) for which evaluation of the CTI by the respective thread contexts resolve to different next program counters that correspond to different control flow targets, the list manager operates to: create a child thread list for each unique next program counter, set a program counter of each child thread list to a respective next program counter, and include, in each respective child thread list, the subset of thread contexts for which the CTI resolved to the corresponding next program counter.

21. The method of claim 16, wherein fetching and decoding comprises using a plurality of parallel fetch and decode pipelines and distributing selected thread lists among the pipelines.

22. The method of claim 16, further comprising determining that one or more operands of the decoded instruction are uniform across all thread contexts of the selected thread list and, in response, executing the decoded instruction once and propagating a common result to all thread contexts of the selected thread list.

23. The method of claim 16, further comprising issuing a next decoded instruction for the selected thread list without waiting for completion of execution of the decoded instruction by all thread contexts of that thread list.

24. The method of claim 23, wherein the issuing employs modulo scheduling.

25. The method of claim 16, wherein evaluating the reconvergence condition uses program counter values and call stack depth indicators associated with thread lists, and comprises prioritizing scheduling of thread lists having numerically smaller call stack depth indicators.21NEU 150026. The method of claim 25, further comprising implementing a thread group mechanism in which child thread lists inherit a thread group identifier from a parent thread list, and maintaining a hot context table that, for each thread group, identifies a thread list having a lowest pair of program counter and call stack depth values.

27. The method of claim 16, wherein maintaining the plurality of thread lists comprises representing each thread list as a linked list by maintaining: a list array that stores, for each thread list, a thread count, a head pointer, and a tail pointer, and a thread array that stores thread descriptors corresponding to the thread contexts, each thread descriptor including a link identifying a next thread in the thread list.

28. The method of claim 27, wherein maintaining the plurality of thread lists comprises representing each thread list as a doubly linked list, with each thread descriptor further including a link to a previous thread in the thread list.

29. The method of claim 16, further comprising: restricting a length of a thread list to a predefined maximum, and handling, via the list manager, multiple thread lists that share a common program counter.

30. The method of claim 29, further comprising: rebalancing, via the list manager, thread contexts among thread lists to maintain threadlist lengths less than the predefined maximum.

31. A non-transitory computer-readable storage medium storing instructions that, when executed by one or more processors of a processing system having an instruction handler and execution hardware, cause the processing system to perform operations for: maintaining, by a scheduler module, a plurality of thread lists, each thread list including a variable number of thread contexts that specify per-thread local data and a program counter that identifies an instruction to execute next, the program counter being common to all thread contexts of the thread list;22NEU 1500selecting, by the scheduler, a thread list; fetching and decoding, by the instruction handler, the instruction identified by the program counter of the selected thread list; executing, by the execution hardware with a plurality of execution lanes for spatiotemporal execution, the decoded instruction for the thread contexts of the selected thread list in successive cycles to produce per-thread results; creating, by a list manager module, in response to detecting that evaluation of the instruction by the thread contexts of the selected thread list selects different next program counters, two or more child thread lists that are each associated with a respective one of the selected next program counters and include a subset of the thread contexts that selected the corresponding next program counter; and merging, by the list manager module, in response to detecting that two or more thread lists satisfy a reconvergence condition, the two or more thread lists into a single thread list for subsequent execution at a common program counter.

32. The non-transitory computer-readable storage medium of claim 31, wherein the instruction comprises a branch instruction and the different next program counters correspond to different branch targets, and wherein creating the child thread lists comprises creating a child thread list for each unique branch target, setting a program counter of each child thread list to the corresponding branch target, and including, in each child thread list, the subset of thread contexts that selected the corresponding branch target.

33. The non-transitory computer-readable storage medium of claim 31, wherein evaluating the reconvergence condition uses program counter values and call stack depth indicators associated with thread lists, and comprises prioritizing scheduling of thread lists having numerically smaller call stack depth indicators.

34. The non-transitory computer-readable storage medium of claim 31, wherein maintaining the plurality of thread lists comprises representing each thread list as a linked list by: maintaining a list array that stores, for each thread list, a thread count, a head pointer, and a tail pointer, and23NEU 1500maintaining a thread array that stores thread descriptors corresponding to the thread contexts, each thread descriptor including a link or index identifying a next thread in the thread list.

35. The non-transitory computer-readable storage medium of claim 31, further comprising instructions that, when executed, cause the processing system to perform additional operations of: determining that one or more operands of the decoded instruction are uniform across all thread contexts of the selected thread list and, in response, executing the decoded instruction once and propagating a common result to all thread contexts of the selected thread list.24NEU 1500

Images