Scalable Sparse Matrix Multiplication Acceleration Using Systolic Arrays with Feedback Inputs

Abstract

To provide devices, systems and methods to enable scalable sparse matrix multiplication.SOLUTION: In a graphics processor 320, a compute accelerator 330 includes a host interface 328, a fabric interconnect 324 coupled with the host interface, and one or more compute engine tiles coupled with the fabric interconnect. The one or more compute engine tiles include sparse matrix multiply acceleration hardware including a systolic array with feedback input.SELECTED DRAWING: Figure 3C

Description

[Background technology]

[0001] Related Applications This application claims priority to Indian Provisional Patent Application No. 202041019059, filed May 5, 2020, which is incorporated herein by reference.

[0002] Background technology Systolic matrix multiplications used in machine learning workloads have a significant proportion of zeros (sparse data workloads). These zero multiplications can be skipped, thus improving overall performance. Current systolic architectures may offer support for sparsity within workloads, but such architectures may not scale well. [Brief explanation of the drawings]

[0003] So that the above-described features of the present embodiments can be understood in detail, a more particular description of the embodiments briefly summarized here will be made by reference to embodiments, some of which are illustrated in the accompanying drawings. It should be noted, however, that the accompanying drawings illustrate only typical embodiments and therefore should not be construed as limiting the scope thereof.

[0004] [Figure 1] FIG. 1 is a block diagram of a processing system according to one embodiment.

[0005] [Figure 2A] 1 illustrates a computing system and graphics processor provided by embodiments described herein. [Figure 2B] 1 illustrates a computing system and graphics processor provided by embodiments described herein. [Figure 2C]1 illustrates a computing system and graphics processor provided by embodiments described herein. [Figure 2D] 1 illustrates a computing system and graphics processor provided by embodiments described herein.

[0006] [Figure 3A] 1 illustrates a block diagram of an additional graphics processor and computation accelerator architecture provided by embodiments described herein. [Figure 3B] 1 illustrates a block diagram of an additional graphics processor and computation accelerator architecture provided by embodiments described herein. [Figure 3C] 1 illustrates a block diagram of an additional graphics processor and computation accelerator architecture provided by embodiments described herein.

[0007] [Figure 4] FIG. 4 is a block diagram of a graphics processing engine 410 of a graphics processor according to some embodiments.

[0008] [Figure 5A] 1 illustrates thread execution logic including an array of processing elements used in a graphics processor core according to embodiments described herein. [Figure 5B] 1 illustrates thread execution logic including an array of processing elements used in a graphics processor core according to embodiments described herein.

[0009] [Figure 6] 1 illustrates an additional execution unit according to one embodiment.

[0010] [Figure 7] FIG. 1 is a block diagram illustrating a graphics processor instruction format according to some embodiments.

[0011] [Figure 8] FIG. 10 is a block diagram of a graphics processor according to another embodiment.

[0012] [Figure 9A] 1 illustrates a command format and command sequence for a graphics processor according to some embodiments. [Figure 9B] 1 illustrates a command format and command sequence for a graphics processor according to some embodiments.

[0013] [Figure 10] 1 illustrates an exemplary graphics software architecture for a data processing system according to some embodiments.

[0014] [Figure 11A] FIG. 1 is a block diagram illustrating an IP core development system according to an embodiment.

[0015] [Figure 11B] 1 illustrates a cross-sectional side view of an integrated circuit package assembly according to some embodiments described herein.

[0016] [Figure 11C] 1 shows a package assembly containing multiple units of hardware logic chiplets connected to a substrate.

[0017] [Figure 11D] 1 illustrates a package assembly including replaceable chiplets according to an embodiment.

[0018] [Figure 12] FIG. 1 is a block diagram illustrating an exemplary system on a chip integrated circuit that can be manufactured using one or more IP cores according to an embodiment.

[0019] [Figure 13A] FIG. 1 is a block diagram illustrating an exemplary graphics processor for use in an SoC according to embodiments described herein. [Figure 13B] FIG. 1 is a block diagram illustrating an exemplary graphics processor for use in an SoC according to embodiments described herein.

[0020] [Figure 14] 1 is a block diagram of a data processing system according to an embodiment.

[0021] [Figure 15] 1 illustrates a matrix operation performed by an instruction pipeline according to an embodiment.

[0022] [Figure 16] A systolic array of multiplier / adder circuits organized in a pipelined fashion is shown.

[0023] [Figure 17A] We show the use of a 4-deep systolic array to compute an equivalent array of 8 systolic stages. [Figure 17B] We show the use of a 4-deep systolic array to compute an equivalent array of 8 systolic stages.

[0024] [Figure 18A] Shows a time diagram of the systolic architecture. [Figure 18B] Shows a time diagram of the systolic architecture.

[0025] [Figure 19] We present a two-path matrix multiplication accelerator, where each path has a depth of four stages. [Figure 20] We present a four-path matrix multiplication accelerator, where each path has a depth of two stages. [Figure 21]We present a scalable sparse matrix multiplication accelerator that utilizes a systolic array with feedback input. [Figure 22] Indicates Src2 input containing sparse data. [Figure 23] We present a scalable sparse matrix multiplication accelerator that uses systolic arrays with feedback inputs and outputs at each stage. [Figure 24] 1 illustrates how the operations are performed on the scalable sparse matrix multiplication accelerator described herein. [Figure 25] We demonstrate how to perform a matrix multiplication operation using a sparse Src2 input matrix. [Figure 26] FIG. 1 is a block diagram of a computing device including a graphics processor according to an embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0026] Described herein are devices, systems, and methods that enable scalable sparse matrix multiplication acceleration utilizing systolic arrays with feedback inputs.

[0027] For purposes of explanation, numerous specific details are set forth to provide a thorough understanding of the various embodiments described below. However, it will be apparent to those skilled in the art that embodiments may be practiced without some of these specific details. In other instances, well-known structures and devices are shown in block diagram form to avoid obscuring underlying principles and to provide a more thorough understanding of the embodiments. Some of the following embodiments are described with reference to a graphics processor; however, the techniques and teachings described herein may be applied to various types of circuit or semiconductor devices, including general-purpose processing devices or graphics processing devices. References herein to "one embodiment" or "an embodiment" indicate that a particular feature, structure, or characteristic described in connection with or associated with the embodiment may be included in at least one of the embodiments. However, appearances of the phrase "in one embodiment" in various places throughout the specification do not necessarily all refer to the same embodiment.

[0028] In the following specification and claims, the terms "coupled" and "connected," along with their derivatives, may be used. It should be understood that these terms are not intended as synonyms for each other. "Coupled" is used to indicate that two or more elements, which may or may not be in direct physical or electrical contact with each other, co-operate or interact with each other. "Connected" is used to indicate the establishment of communication between two or more elements that are coupled with each other.

[0029] In the following description, Figures 1 through 13A-13B provide overviews of exemplary data processing systems and graphics processor logic incorporating or associated with various embodiments. Figures 14-26 provide specific details of various embodiments. Some aspects of the following embodiments are described in the context of a graphics processor, while other aspects are described in the context of a general-purpose processor, such as a central processing unit (CPU). Similar techniques and teachings can be applied to other types of circuits or semiconductor devices, including, but not limited to, one or more instances of a multi-core processor, a GPU cluster, or a field-programmable gate array (FPGA). In general, the teachings are applicable to any processor or machine that manipulates or processes images (e.g., samples, pixels), vertex data, or geometry data, or performs parallel processing operations for machine learning and high-performance computing applications.

[0030] System Overview 1 is a block diagram of a processing system 100 according to an embodiment. System 100 may be used in a single-processor desktop system, a multiprocessor workstation system, or a server system having multiple processors 102 or processor cores 107. In one embodiment, system 100 is a processing platform embedded in a system-on-chip (SoC) integrated circuit for use in mobile, handheld, or embedded devices, such as in Internet of Things (IoT) devices with wired or wireless connectivity to local or wide area networks.

[0031] In one embodiment, system 100 may include, be coupled to, or be integrated into a server-based gaming platform; a game console, including a game and media console; a mobile game console, a handheld game console, or an online game console. In some embodiments, system 100 is part of a mobile internet-connected device, such as a mobile phone, a smart phone, a tablet computing device, or a laptop with limited internal storage capacity. Processing system 100 may also include, be coupled to, or be integrated into a wearable device, such as a smart watch wearable device; smart eyewear or clothing enhanced with augmented reality (AR) or virtual reality (VR) functionality to provide visual, auditory, or tactile output to augment real-world visual, auditory, or tactile experiences or to provide other text, audio, graphic, video, holographic images or video, or haptic feedback; other augmented reality (AR) devices; or other virtual reality (VR) devices. In some embodiments, processing system 100 includes or is part of a television or set-top box device. In one embodiment, system 100 can include, be coupled to, or be integrated into an autonomous vehicle, such as a bus, a tractor-trailer, an automobile, a motor or electric bicycle, an airplane, or a glider (or any combination thereof). The autonomous vehicle can use system 100 to process the sensed environment around the vehicle.

[0032] In some embodiments, the one or more processors 102 each include one or more processor cores 107 for processing instructions that, when executed, perform operations on the system or perform user software. In some embodiments, at least one of the one or more processor cores 107 is configured to process a particular instruction set 109. In some embodiments, the instruction set 109 may facilitate complex instruction set computing (CISC), reduced instruction set computing (RISC), or very long instruction word (VLIW) computing. One or more processor cores 107 may process different instruction sets 109, which may include instructions to facilitate emulation of other instruction sets. The processor cores 107 may also include other processing devices, such as a digital signal processor (DSP).

[0033] In some embodiments, processor 102 includes cache memory 104. Depending on the architecture, processor 102 may have a single internal cache or multiple levels of internal cache. In some embodiments, cache memory is shared among various components of processor 102. In some embodiments, processor 102 also uses an external cache (e.g., a level 3 (L3) cache or last level cache (LLC)) (not shown), which may be shared within processor core 107 using known cache coherency techniques. A register file 106 may additionally be included in processor 102 and may include different types of registers (e.g., integer registers, floating-point registers, status registers, and instruction pointer registers) for storing different types of data. Some registers may be general-purpose registers, while others may be specific to the design of processor 102.

[0034] In some embodiments, one or more processors 102 are coupled to one or more interface buses 110 to carry communication signals, such as address, data, or control signals, between the processors 102 and other components in the system 100. The interface bus 110, in one embodiment, may be a processor bus such as a version of a Direct Media Interface (DMI) bus. However, the processor bus is not limited to a DMI bus and may include one or more peripheral component interconnect buses (e.g., PCI, PCI Express), memory buses, or other types of interface buses. In one embodiment, the processor 102 includes an integrated memory controller 116 and a platform controller hub 130. The memory controller 116 facilitates communication between memory devices and other components of the system 100, and the platform controller hub (PCH) 130 provides connectivity to I / O devices via a local I / O bus.

[0035] Memory device 120 may be a dynamic random access memory (DRAM) device, a static random access memory (SRAM) device, a flash memory device, a phase-change memory device, or any other memory device with suitable performance to function as process memory. In one embodiment, memory device 120 may operate as system memory for system 100, storing data 122 and instructions 121 for use when one or more processors 102 execute applications or processes. Memory controller 116 also couples to an optional external graphics processor 118, which communicates with one or more graphics processors 108 within processor 102 and may perform graphics and media operations. In some embodiments, graphics, media, and / or computation operations may be assisted by accelerator 112, a co-processor that may be configured to perform a specialized set of graphics, media, or computation operations. For example, in one embodiment, accelerator 112 is a matrix multiplication accelerator used to optimize machine learning or computational operations. In one embodiment, accelerator 112 is a ray tracing accelerator that may be used to perform ray tracing operations in cooperation with graphics processor 108. In one embodiment, external accelerator 119 may be used instead of or in cooperation with accelerator 112.

[0036] In some embodiments, a display device 111 may be connected to the processor 102. The display device 111 may be one or more internal display devices, such as a mobile electronic device, a laptop device, or an external display device attached via a display interface (e.g., DisplayPort, etc.). In one embodiment, the display device 111 may be a head-mounted display (HMD), such as a stereoscopic display device for use in virtual reality (VR) or augmented reality (AR) applications.

[0037] In some embodiments, the platform controller hub 130 allows peripherals to connect to the memory device 120 and the processor 102 via a high-speed I / O bus. The I / O peripherals include, but are not limited to, an audio controller 146, a network controller 134, a firmware interface 128, a wireless transceiver 126, a touch sensor 125, and a data storage device 124 (e.g., non-volatile memory, volatile memory, hard disk drive, flash memory, NAND, 3D NAND, 3D XPoint, etc.). The data storage device 124 can be connected via a storage interface (e.g., SATA) or via a peripheral bus such as a peripheral component interconnect bus (e.g., PCI, PCI Express). The touch sensor 125 can include a touch screen sensor, a pressure sensor, or a fingerprint sensor. Wireless transceiver 126 may be a Wi-Fi transceiver, a Bluetooth transceiver, or a mobile network transceiver such as a 3G, 4G, 5G, or Long Term Evolution (LTE) transceiver. Firmware interface 128 enables communication with system firmware and may be, for example, a unified extensible firmware interface (UEFI). Network controller 134 may enable network connectivity to a wired network. In some embodiments, a high-performance network controller (not shown) couples to interface bus 110. Audio controller 146, in one embodiment, is a multi-channel high-definition audio controller. In one embodiment, system 100 includes an optional legacy I / O controller 140 for coupling legacy (e.g., Personal System 2 (PS / 2)) devices to the system.The platform controller hub 130 may also be connected to one or more universal serial bus (USB) controllers 142 to which input devices such as a keyboard and mouse 143 combination, a camera 144, or other USB input devices may be connected.

[0038] It will be understood that the illustrated system 100 is exemplary and not limiting, as other types of data processing systems configured differently may be used. For example, instances of platform controller hub 130 and memory controller 116 may be integrated into a separate external graphics processor, such as external graphics processor 118. In one embodiment, platform controller hub 130 and / or memory controller 116 may be external to one or more processors 102. For example, system 100 may include external memory controller 116 and platform controller hub 130, which may be configured as a memory controller hub and a peripheral controller hub within a system chipset that communicates with processor 102.

[0039] For example, the circuit board (sled) on which components such as the CPU, memory, and other components are located is designed for increased thermal characteristics. In some examples, processing components such as the processor are located on the top side of the sled, while memory components such as DIMMs are located on the bottom side of the sled. As a result of the increased airflow provided by this design, the components can operate at higher frequencies and power levels than in typical systems, thereby improving performance. Furthermore, the sled is configured to seamlessly mate with power and data communication cables in the rack, thereby enhancing the ability to quickly remove, upgrade, reinstall, and / or replace the sled. Similarly, the individual components located on the sled, such as the processor, accelerator, memory, and data storage drives, are configured to be easily upgraded due to the increased spacing between them. In an exemplary embodiment, the components further include hardware authentication features to prove their authenticity.

[0040] Data centers can utilize a single network architecture ("fabric") that supports multiple other network architectures, including Ethernet and Omni-Path. Threads can be coupled to switches via optical fiber, which offers higher bandwidth and lower latency than typical twisted-pair cabling (e.g., Category 5, Category 5e, Category 6, etc.). Due to the high-bandwidth, low-latency interconnect and network architecture, data centers can pool resources such as memory, accelerators (e.g., GPUs, graphics accelerators, FPGAs, ASICs, neural network and / or artificial intelligence accelerators), and physically disaggregated data storage drives, providing them to compute resources (e.g., processors) as needed, allowing the compute resources to access the pooled resources as if they were local.

[0041] A power supply or power source can provide voltage and / or current to system 100 or any component or system described herein. In one example, the power source includes an AC-DC (alternating current-to-direct current) adapter for plugging into a wall outlet. Such AC power can be a renewable energy (e.g., solar power) power source. In one example, the power source includes a DC power source, such as an external AC-DC converter. In one example, the power source or power supply includes wireless charging hardware for charging by proximity to a charging field. In one example, the power source can include an internal battery, an AC power source, a motion-based power source, a solar-powered power source, or a fuel cell power source.

[0042] 2A-2D illustrate a computing system and graphics processor provided by embodiments described herein. Elements in FIG. 2A-2D having the same reference number (or name) as elements in any other figure herein may operate or function in some manner similar to that described elsewhere herein, but are not limited to such.

[0043] 2A is a block diagram of an embodiment of a processor 200 having one or more processor cores 202A-202N, an integrated memory controller 214, and an integrated graphics processor 208. Processor 200 includes, and can include, additional cores up to additional core 202N, represented by a dashed box. Each of processor cores 202A-202N includes one or more internal cache units 204A-204N. In some embodiments, each processor core also has access to one or more shared cache units 206. The internal cache units 204A-204N and the shared cache unit 206 represent a cache memory hierarchy within processor 200. The cache memory hierarchy may include at least one level of instruction and data cache within each processor core, one or more levels of shared mid-level cache, such as level 2 (L2), level 3 (L3), level 4 (L4), and other cache levels, where the highest level of cache before external memory is classified as LLC. In some embodiments, cache coherence logic maintains coherency between the various cache units 206 and 204A-204N.

[0044] In some embodiments, processor 200 may include one or more bus controller units 216 and a set of system agent cores 210. One or more bus controller units 216 manage a set of peripheral buses, such as one or more PCI or PCI Express buses. System agent core 210 provides management functions for various processor components. In some embodiments, system agent core 210 includes one or more integrated memory controllers 214 to manage access to various external memory devices (not shown).

[0045] In some embodiments, one or more of the processor cores 202A-202N include support for simultaneous multithreading. In such embodiments, the system agent core 210 includes components for coordinating and operating the cores 202A-202N during multithreaded processing. The system agent core 210 may further include a power control unit (PCU) that includes logic and components for coordinating the power states of the processor cores 202A-202N and the graphics processor 208.

[0046] In some embodiments, processor 200 further includes a graphics processor 208 for performing graphics processing operations. In some embodiments, graphics processor 208 couples to a set of shared cache units 206, including one or more integrated memory controllers 214, and to a system agent core 210. In some embodiments, system agent core 210 also includes a display controller 211 that drives the graphics processor output to one or more coupled displays. In some embodiments, display controller 211 may also be a separate module coupled to the graphics processor via at least one interconnect, or may be integrated within graphics processor 208.

[0047] In some embodiments, a ring-based interconnect unit 212 is used to couple the internal components of processor 200. However, other interconnect units, such as point-to-point interconnects, switched interconnects, or other techniques, including techniques known in the art, may also be used. In some embodiments, graphics processor 208 couples to ring interconnect 212 via I / O link 213.

[0048] The exemplary I / O link 213 represents at least one of several types of I / O interconnects, including an on-package I / O interconnect that facilitates communication between various processor components and a high-performance embedded memory module 218, such as an eDRAM module. In some embodiments, each of the processor cores 202A-202N and the graphics processor 208 may use the embedded memory module 218 as a shared last-level cache.

[0049] In some embodiments, processor cores 202A-202N are homogeneous cores that execute the same instruction set architecture. In another embodiment, processor cores 202A-202N are heterogeneous with respect to instruction set architecture (ISA), where one or more of processor cores 202A-202N execute a first instruction set and at least one of the other cores executes a subset of the first instruction set or a different instruction set. In one embodiment, processor cores 202A-202N are heterogeneous with respect to microarchitecture, where one or more cores with relatively high power consumption are combined with one or more cores with lower power consumption. In one embodiment, processor cores 202A-202N are heterogeneous with respect to computational capability. Furthermore, processor 200 can be implemented as one or more chips or SoC integrated circuits having the illustrated components, in addition to other components.

[0050] FIG. 2B is a block diagram of the hardware logic of a graphics processor core 219 according to some embodiments described herein. Elements in FIG. 2B having the same reference number (or name) as elements in any other figure herein may operate or function in any manner similar to that described elsewhere herein, including, but not limited to, the above. The graphics processor core 219, sometimes referred to as a core slice, may be one or more graphics cores within a modular graphics processor. The graphics processor core 219 is an example of a graphics core slice, and a graphics processor as described herein may include multiple graphics core slices based on target power and performance envelopes. Each graphics processor core 219 may include a fixed function block 230 coupled with multiple sub-cores 221A-221F, also referred to as sub-slices, which include modular blocks of general-purpose and fixed-function logic.

[0051] In some embodiments, fixed function block 230 includes a geometry / fixed function pipeline 231 that can be shared by all sub-cores within graphics processor core 219, e.g., in lower performance and / or lower power graphics processor implementations. In various embodiments, geometry / fixed function pipeline 231 includes a 3D fixed function pipeline (e.g., 3D pipeline 312 as in FIGS. 3A and 4 ) that includes a video front end unit, a thread spawner and thread dispatcher, and a unified return buffer manager that manages a unified return buffer (e.g., unified return buffer 418 of FIG. 4 , described below).

[0052] In one embodiment, fixed function block 230 also includes graphics SoC interface 232, graphics microcontroller 233, and media pipeline 234. Graphics SoC interface 232 provides an interface between graphics processor core 219 and other processor cores within a system on chip integrated circuit. Graphics microcontroller 233 is a programmable sub-processor that can be configured to manage various functions of graphics processor core 219, including thread dispatch, scheduling, and preemption. Media pipeline 234 (e.g., media pipeline 316 in FIGS. 3A and 4) includes logic that facilitates decoding, encoding, pre-processing, and / or post-processing of multimedia data, including image and video data. Media pipeline 234 performs media processing via requests for computation or sampling logic within sub-cores 221-221F.

[0053] In one embodiment, SoC interface 232 enables graphics processor core 219 to communicate with general-purpose application processor cores and / or other components within the SoC, including memory hierarchy elements such as shared last-level cache memory, system RAM, and / or embedded on-chip or on-package DRAM. SoC interface 232 also enables communication with fixed-function devices within the SoC, such as a camera imaging pipeline, and enables and / or implements the use of global memory atomics that can be shared between a CPU and graphics processor core 219 within the SoC. SoC interface 232 can also implement power management controls for graphics processor core 219 and enable interfacing between the clock domain of graphics core 219 and other clock domains within the SoC. In one embodiment, SoC interface 232 enables receipt of command buffers from a command streamer and global thread dispatcher configured to provide commands and instructions to each of one or more graphics cores within the graphics processor. Commands and instructions can be dispatched to the media pipeline 234 if a media operation is to be performed, or to the geometry and fixed function pipelines (e.g., to the geometry and fixed function pipeline 231, the geometry and fixed function pipeline 237) if a graphics processing operation is to be performed.

[0054] Graphics microcontroller 233 can be configured to perform various scheduling and management tasks for graphics processor core 219. In one embodiment, graphics microcontroller 233 can perform graphics and / or compute workload scheduling among the various graphics parallel engines within execution unit (EU) arrays 222A-222F, 224A-224F within sub-cores 221A-221F. In this scheduling model, host software executing on a CPU core of an SoC that includes graphics processor core 219 can submit a workload to one of multiple graphics processor arrays, which initiates a scheduling operation on the appropriate graphics engine. Scheduling operations include determining which workload to run next, submitting the workload to the command streamer, preempting existing workloads running in the engines, monitoring the progress of the workload, and notifying host software when the workload is complete. In one embodiment, graphics microcontroller 233 may also facilitate low power or idle states for graphics processor core 219 and may provide graphics processor core 219 with the ability to save and restore registers within graphics processor core 219 across low power state transitions independent of the system's operating system and / or graphics driver software.

[0055] Graphics processor core 219 may have more or fewer sub-cores 221A-221F than shown, and may have up to N modular sub-cores. For each set of N sub-cores, graphics processor core 219 may also include shared function logic 235, shared and / or cache memory 236, geometry / fixed function pipeline 237, and additional fixed function logic 238 for accelerating various graphics and compute processing operations. Shared function logic 235 may include logic units (e.g., samplers, math, and / or inter-thread communication logic) related to shared function logic 420 of FIG. 4 that may be shared by each of the N sub-cores in graphics processor core 219. Shared and / or cache memory 236 may be a last-level cache for the set of N sub-cores 221A-221F in graphics processor core 219 and may also function as a shared memory accessible by multiple sub-cores. Geometry / fixed function pipeline 237 may be included in place of geometry / fixed function pipeline 231 in fixed function block 230 and may include the same or similar logical units.

[0056] In one embodiment, graphics processor core 219 includes additional fixed-function logic 238, which may include various fixed-function acceleration logic used by graphics processor core 219. In one embodiment, additional fixed-function logic 238 includes an additional geometry pipeline for use in position-only shading. In position-only shading, there are two geometry pipelines: a full geometry pipeline within geometry / fixed-function pipeline 238, 231, and a cull pipeline, an additional geometry pipeline that may be included within additional fixed-function logic 238. In one embodiment, the cull pipeline is a trimmed-down version of the full geometry pipeline. The full pipeline and the cull pipeline can run different instances of the same application, with each instance having a separate context. Position-only shading can hide long cull runs of discarded triangles, potentially allowing shading to complete earlier. For example, in one embodiment, cull pipeline logic in additional fixed-function logic 238 can execute position shaders in parallel with the main application. Generally, the cull pipeline generates critical results faster than the full pipeline because it fetches and shades only vertex position attributes without rasterizing and rendering pixels to the frame buffer. The cull pipeline can use the generated critical results to calculate visibility information for all triangles, regardless of whether those triangles are culled. The full pipeline (which in this example may be referred to as the reproduction pipeline) can use the visibility information to skip culled triangles in order to occlude only visible triangles that are ultimately passed to the rasterization phase.

[0057] In one embodiment, additional fixed function logic 238 may also include machine learning acceleration logic, such as fixed function matrix multiplication logic, for implementations that include optimizations for machine learning training or inference.

[0058] Each graphics sub-core 221A-221F includes a set of execution resources that can be used to perform graphics, media, and computation operations as requested by the graphics pipeline, media pipeline, or shader programs. The graphics sub-cores 221A-221F include a plurality of EU arrays 222A-222F, 224A-224F, thread dispatch and inter-thread communication (TD / IC) logic 223A-223F, 3D (e.g., texture) samplers 225A-225F, media samplers 206A-206F, shader processors 227A-227F, and shared local memories (SLMs) 228A-228F. EU arrays 222A-222F, 224A-224F each contain multiple execution units, which are general-purpose graphics processing units capable of executing floating-point and integer / fixed-point logic operations, including graphics, media, or computational shader programs, in service of graphics, media, or computational operations. TD / IC logic 223A-223F performs local thread dispatch and thread control operations for the execution units within a sub-core and facilitates communication between threads running on the sub-core's execution units. 3D samplers 225A-225F can load textures or other 3D graphics-related data into memory. The 3D samplers can load texture data differently based on the configured sample state and the texture format associated with a given texture. Media samplers 206A-206F can perform similar load operations based on the type and format associated with the media data. In one embodiment, each graphics sub-core 221A-221F may alternatively include a unified 3D and media sampler. Threads executing in execution units within each sub-core 221A-221F may use shared local memory 228A-228F within each sub-core, allowing threads executing within a thread group to execute using a common pool of on-chip memory.

[0059] 2C illustrates a graphics processing unit (GPU) 239 that includes a dedicated set of graphics processing resources arranged into multi-core groups 240A-240N. While details of only a single multi-core group 240A are provided, it should be understood that other multi-core groups 240B-240N may include the same or similar sets of graphics processing resources.

[0060] As shown, multi-core group 240A may include a set of graphics cores 243, a set of tensor cores 244, and a set of ray tracing cores 245. Scheduler / dispatcher 241 schedules and dispatches graphics threads for execution on the various cores 243, 244, and 245. A set of register files 242 stores operand values ​​used by cores 243, 244, and 245 when executing the graphics threads. These may include, for example, integer registers for storing integer values, floating-point registers for storing floating-point values, vector registers for storing packed data elements (integer and / or floating-point data elements), and tile registers for storing tensor / matrix values. In one embodiment, the tile registers are implemented as a combined set of vector registers.

[0061] One or more combined level 1 (L1) cache and shared memory units 247 store graphics data, such as texture data, vertex data, pixel data, ray data, and bounding volume data, locally within each multi-core group 240A. One or more texture units 247 may also be used to perform texturing operations, such as texture mapping and sampling. A level 2 (L2) cache 253, shared by all or a subset of the multi-core groups 240A-240N, stores graphics data and / or instructions for multiple simultaneous graphics threads. As shown, the L2 cache 253 may be shared across multiple multi-core groups 240A-240N. One or more memory controllers 248 couple the GPU 239 to memory 249, which may be system memory (e.g., DRAM) and / or dedicated graphics memory (e.g., GDDR6 memory).

[0062] Input / output (I / O) circuitry 250 couples GPU 239 to one or more input / output devices 252, such as a digital signal processor (DSP), a network controller, or a user input device. An on-chip interconnect may be used to couple I / O devices 252 to GPU 239 and memory 249. One or more I / O memory management units (IOMMUs) 251 of I / O circuitry 250 directly couple I / O devices 252 to system memory 249. In one embodiment, IOMMU 251 manages multiple sets of page tables to map virtual addresses to physical addresses in system memory 249. In this embodiment, I / O devices 252, CPU 246, and GPU 239 may share the same virtual address space.

[0063] In some implementations, IOMMU 251 supports virtualization. In this case, IOMMU 251 may manage a first set of page tables for mapping guest / graphics virtual addresses to guest / graphics physical addresses and a second set of page tables for mapping guest / graphics physical addresses to system / host physical addresses (e.g., in system memory 249). The base addresses of each of the first and second sets of page tables are stored in control registers and may be swapped in a context switch (e.g., so that a new context is provided for accessing the associated set of page tables). Although not shown in FIG. 2C, cores 243, 244, 245 and / or multi-core groups 240A-240N may each include a translation lookaside buffer (TLB) for caching guest-virtual-to-guest-physical, guest-physical-to-guest-virtual, and guest-virtual-to-host-physical translations.

[0064] In one embodiment, CPU 246, GPU 239, and I / O devices 252 are integrated into a single semiconductor chip and / or chip package. The illustrated memory 249 may be integrated on the same chip or may be coupled to memory controller 248 via an off-chip interface. In one implementation, memory 249 includes GDDR6 memory that shares the same virtual address space as other physical system-level memory, although the underlying principles of the present invention are not limited to this particular implementation.

[0065] In one embodiment, tensor cores 244 include multiple execution units specifically designed to perform matrix operations, which are fundamental computational operations used to perform deep learning operations. For example, simultaneous matrix multiplication operations can be used for neural network training and inference. Tensor cores 244 can perform matrix operations using a variety of operand precisions, including single-precision floating-point (e.g., 32 bits), half-precision floating-point (e.g., 16 bits), integer words (16 bits), bytes (8 bits), and half-bytes (4 bits). In one embodiment, neural network implementations potentially combine details from multiple frames to extract features for each rendered scene and construct a high-quality final image.

[0066] In deep learning implementations, parallel matrix multiplication operations may be scheduled for execution on tensor cores 244. Neural network training, in particular, requires a significant number of matrix dot product operations. To process the dot product formula for N x N x N matrix multiplication, tensor cores 244 may contain at least N dot product processing elements. Before the matrix multiplication begins, one entire matrix is ​​loaded into a tile register, and at least one column of the second matrix is ​​loaded in each of the N cycles. There are N dot products processed per cycle.

[0067] Matrix elements can be stored in different precisions depending on the specific implementation, including 16-bit words, 8-bit bytes (e.g., INT8), and 4-bit half-bytes (e.g., INT4). Different precision modes may be specified for tensor core 244 to ensure that the most efficient precision is used for various workloads (e.g., inference workloads that can tolerate quantization to bytes and half-bytes).

[0068] In one embodiment, ray tracing core 245 accelerates ray tracing operations for both real-time and non-real-time ray tracing implementations. In particular, ray tracing core 245 includes ray traverse / intersection circuitry for performing ray traversals using bounding volume hierarchies (BVHs) and identifying intersections between rays and primitives enclosed by the BVH volumes. Ray tracing core 245 may also include circuitry for performing depth testing and culling (e.g., using a Z-buffer or similar construct). In one embodiment, ray tracing core 245 performs traversal and intersection operations in coordination with the image denoising techniques described herein, at least some of which may be performed by tensor core 244. For example, in one embodiment, tensor core 244 implements a deep learning neural network to perform denoising of frames produced by ray tracing core 245. However, the CPU 246, graphics core 243, and / or ray tracing core 245 may also implement all or part of the denoising and / or deep learning algorithms.

[0069] Additionally, as noted above, a distributed approach to denoising may be used when the GPU 239 is in a computing device coupled to other computing devices via a network or high-speed interconnect. In this embodiment, the interconnected computing devices share neural network learning / training data to improve the speed at which the overall system learns to perform denoising on different types of image frames and / or different graphics applications.

[0070] In one embodiment, ray tracing core 245 handles all BVH traversals and ray-primitive intersections, preventing graphics core 243 from becoming overloaded with thousands of instructions per ray. In one embodiment, each ray tracing core 245 includes a first set of specialized circuitry for performing bounding box tests (e.g., traversal operations) and a second set of specialized circuitry for performing ray-triangle intersection tests (e.g., intersecting rays). Thus, in one embodiment, multi-core group 240A can simply initiate ray probes, and ray tracing core 245 independently performs ray traversals and intersections and returns hit data (e.g., hit, no hit, multiple hits, etc.) to the thread context. The other cores 243, 244 are freed to perform other graphics or compute other work when ray tracing core 245 performs traversal and intersection operations.

[0071] In one embodiment, each ray tracing core 245 includes a traversal unit for performing BVH test operations and an intersection unit for performing ray-primitive intersection tests. The intersection unit generates a "hit," "no hit," or "multi-hit" response and provides it to the appropriate thread. Between traversal and intersection operations, the execution resources of other cores (e.g., graphics core 243 and tensor core 244) are freed to perform other forms of graphics work.

[0072] In one particular embodiment described below, a hybrid rasterization / ray tracing approach is used in which work is distributed between the graphics core 243 and the ray tracing core 245.

[0073] In one embodiment, ray tracing core 245 (and / or other cores 243, 244) includes hardware support for Microsoft's DirectX Ray Tracing (DXR), including the DispatchRays command, and ray tracing instruction sets such as ray generation, nearest hit, any hit, and miss shaders, which enable the assignment of a unique set of shaders and textures to each object. Another ray tracing platform that may be supported by ray tracing core 245, graphics core 243, and tensor core 244 is Vulkan 1.1.85. However, the underlying principles of the present invention are not limited to any particular ray tracing ISA.

[0074] In general, the various cores 245, 244, 243 may support a ray tracing instruction set that includes instructions / functions for ray generation, nearest hit, any hit, ray-primitive intersection, primitive and hierarchy related bounding box construction, misses, visits, and exceptions. More specifically, one embodiment includes ray tracing instructions for performing the following functions:

[0075] ray generation Ray generation commands can be executed for each pixel, sample, or other user-defined work allocation.

[0076] Nearest Hit A nearest hit instruction can be executed to find the closest intersection of a ray with a primitive in the scene.

[0077] Any hit - Any hit instruction identifies multiple intersection points between rays and primitives in the scene and potentially identifies a new closest intersection point.

[0078] Intersection - The intersection command performs a ray-primitive intersection test and outputs the result.

[0079] Primitive-related bounding box construction - This command builds a bounding box around a given primitive or group of primitives (e.g., when building a new BVH or other acceleration data structure).

[0080] Miss - Indicates that the ray misses all geometry in the scene or a specific area of ​​the scene.

[0081] Visit - Indicates the child volume that the ray will traverse.

[0082] exception - Contains different types of exception handling (called for different error conditions).

[0083] FIG. 2D is a block diagram of a general-purpose graphics processing unit (GPGPU) 270, which can be configured as a graphics processor and / or computer accelerator, according to embodiments described herein. The GPGPU 270 can be interconnected with a host processor (e.g., one or more CPUs 246) and memories 271 and 272 via one or more system and / or memory buses. In one embodiment, memory 271 is system memory, which may be shared with one or more CPUs 246, and memory 272 is device memory dedicated to the GPGPU 270. In one embodiment, components within the GPGPU 270 and device memory 272 may be mapped to memory addresses accessible to one or more CPUs 246. Access to memories 271 and 272 can be facilitated by a memory controller 268. In one embodiment, the memory controller 268 can include an internal direct memory access (DMA) controller 269 or logic to perform operations that would otherwise be performed by the DMA controller.

[0084] GPGPU 270 includes multiple cache memories, including L2 cache 253, L1 cache 254, instruction cache 255, and shared memory 256, at least some of which may be designated as cache memories. GPGPU 270 also includes multiple compute units 260A-260N. Each compute unit 260A-260N includes a set of vector registers 261, scalar registers 262, a vector logic unit 263, and a scalar logic unit 264. Compute units 260A-260N may also include local shared memory 265 and a program counter 266. Compute units 260A-260N may be coupled with constant cache 267, which may be used to store constant data that does not change during the execution of a kernel or shader program running on GPGPU 270. In one embodiment, constant cache 267 is a scalar data cache, and cached data can be fetched directly into scalar registers 262 .

[0085] During operation, one or more CPU(s) 246 can write commands to registers or memory within GPGPU 270 that are mapped to an accessible address space. Command processor 257 can read commands from registers or memory and determine how these commands are processed within GPGPU 270. Thread dispatcher 258 can then dispatch threads to compute units 260A-260N to execute these commands. Each compute unit 260A-260N can execute threads independently of the other compute units. Furthermore, each compute unit 260A-260N can be independently configured for conditional computation and can conditionally output computation results to memory. Command processor 257 can interrupt one or more CPUs 246 when a submitted command completes.

[0086] 3A-3C illustrate block diagrams of additional graphics processor and computational accelerator architectures provided by embodiments described herein. Elements in FIG. 3A-3C having the same reference numbers (or names) as elements in other figures herein may operate or function in a manner similar to that described elsewhere in this application, but are not limited to such.

[0087] 3A is a block diagram of a graphics processor 300, which may be, but is not limited to, a discrete graphics processing unit or a graphics processor integrated with multiple processing cores or other semiconductor devices, such as memory devices or network interfaces. In some embodiments, the graphics processor communicates via a memory-mapped I / O interface to registers on the graphics processor and by commands located in processor memory. In some embodiments, the graphics processor 300 includes a memory interface 314 for accessing memory. The memory interface 314 may be an interface to local memory, one or more internal caches, one or more shared external caches, and / or system memory.

[0088] In some embodiments, graphics processor 300 also includes a display controller 302 that drives display output data to a display device 318. Display controller 302 includes hardware for one or more overlay planes for displaying and composing multiple layers of video or user interface elements. Display device 318 can be an internal or external display device. In one embodiment, display device 318 is a head-mounted display device, such as a virtual reality (VR) display device or an augmented reality (AR) display device. In some embodiments, graphics processor 300 includes a video codec engine 306 that encodes, decodes, or transcodes media to, from, or between one or more media encoding formats, including, but not limited to, Moving Picture Experts Group (MPEG) formats such as MPEG-2, Advanced Video Coding (AVC) formats such as H.264 / MPEG-4 AVC, H.265 / HEVC, Alliance for Open Media (AOMedia) VP8, VP9, ​​and the Society of Motion Picture & Television Engineers (SMPTE) 421 M / VC-1, and Joint Photographic Experts Group (JPEG) formats such as JPEG and Motion JPEG (MJPEG).

[0089] In some embodiments, graphics processor 300 includes a block image transfer (BLIT) engine 304 to perform two-dimensional (2D) rasterizer operations, including, for example, bit-boundary block transfer. However, in one embodiment, 2D graphics operations are performed using one or more components of a graphics processing engine (GPE) 310. In some implementations, GPE 310 is a computation engine for performing graphics operations, including three-dimensional (3D) graphics operations and media operations.

[0090] In some embodiments, GPE 310 includes a 3D pipeline 312 for performing 3D processing, such as drawing three-dimensional images and scenes using processing functions that operate on 3D primitive shapes (e.g., rectangles, triangles, etc.). 3D pipeline 312 includes programmable and fixed function elements that perform various tasks within the element and / or spawn execution threads in 3D / media subsystem 315. While 3D pipeline 312 can be used to perform media processing, embodiments of GPE 310 also include a media pipeline 316 that is specifically used to perform media processing, such as video post-processing and image enhancement.

[0091] In some embodiments, the media pipeline 316 includes a fixed function or programmable logic unit for performing one or more specialized media operations, such as video decode acceleration, video deinterlacing, and video encode acceleration, in place of or in lieu of the video codec engine 306. In some embodiments, the media pipeline 316 additionally includes a thread generation unit that generates threads for execution in the 3D / media subsystem 315. The generated threads perform computations for the media operations on one or more graphics execution units included in the 3D / media subsystem 315.

[0092] In some embodiments, the 3D / media subsystem 315 includes logic for executing threads generated by the 3D pipeline 312 and the media pipeline 316. In one embodiment, the pipelines send thread execution requests to the 3D / media subsystem 315, which includes thread dispatch logic for arbitrating the various requests and dispatching them to available thread execution resources. The execution resources include an array of graphics execution units for processing the 3D and media threads. In some embodiments, the 3D / media subsystem 315 includes one or more internal caches for thread instructions and data. In some embodiments, the subsystem also includes shared memory, including registers and addressable memory, for sharing data between threads and storing output data.

[0093] FIG. 3B illustrates a graphics processor 320 having a tiled architecture according to an embodiment described herein. In one embodiment, the graphics processor 320 includes a graphics processing engine cluster 322 having multiple instances of the graphics processing engine 310 of FIG. 3A within graphics engine tiles 310A-310D. Each graphics engine tile 310A-310D can be interconnected via a set of tile interconnects 323A-323F. Each graphics engine tile 310A-310D can also be connected to memory modules or memory devices 326A-326D via memory interconnects 325A-325D. The memory devices 326A-326D can use any graphics memory technology. For example, the memory devices 326A-326D can be graphics double data rate (GDDR) memory. The memory devices 326A-326D, in one embodiment, are high-bandwidth memory (HBM) modules that may be on-die with their respective graphics engine tiles 310A-310D. In one embodiment, the memory devices 326A-326D are stacked memory devices that may be stacked on top of their respective graphics engine tiles 310A-310D. In one embodiment, each graphics engine tile 310A-310D and associated memory 326A-326D resides on a separate chiplet bonded to a base die or base substrate, as described in further detail in Figures 11B-11D.

[0094] Graphics processor 320 can be configured with a non-uniform memory access (NUMA) system in which memory devices 326A-326D are coupled to associated graphics engine tiles 310A-310D. A given memory device may be accessed by graphics engine tiles other than the tile to which it is directly connected. However, access latency to memory devices 326A-326D will be lowest when accessing the local tile. In one embodiment, a cache-coherent NUMA (ccNUMA) system uses tile interconnects 323A-323F to enable communication between cache controllers within graphics engine tiles 310A-310D to maintain a consistent memory image when multiple caches store the same memory location.

[0095] The graphics processing engine cluster 322 can be connected to an on-chip or on-package fabric interconnect 324. The fabric interconnect 324 can enable communication between the graphics engine tiles 310A-310D and components such as the video codec 306 and one or more copy engines 304. The copy engine 304 can be used to move data to, from, and between the memory devices 326A-326D and memory external to the graphics processor 320 (e.g., system memory). The fabric interconnect 324 can also be used to interconnect the graphics engine tiles 310A-310D. The graphics processor 320 may optionally include a display controller 302 to enable connection to an external display device 318. The graphics processor can also be configured as a graphics or computation accelerator. In an accelerator configuration, the display controller 302 and display device 318 may be omitted.

[0096] Graphics processor 320 may connect to a host system via host interface 328. Host interface 328 may enable communication between graphics processor 320, system memory, and / or other system components. Host interface 328 may be, for example, a PCI Express bus or other type of host system interface.

[0097] FIG. 3C illustrates a computational accelerator 330 according to an embodiment described herein. The computational accelerator 330 may include an architecture similar to the graphics processor 320 of FIG. 3B and is optimized for computational acceleration. The computational engine cluster 332 may include a set of computational engine tiles 340A-340D, each of which includes execution logic optimized for parallel or vector-based general-purpose computation. In some embodiments, the computational engine tiles 340A-340D do not include fixed-function graphics processing logic, but in one embodiment, one or more of the computational engine tiles 340A-340D may include logic for performing media acceleration. The computational engine tiles 340A-340D may be connected to memories 326A-326D via memory interconnects 325A-325D. The memories 326A-326D and memory interconnects 325A-325D may be of similar technology to the graphics processor 320, or may be different. Graphics compute engine tiles 340A-340D may also be interconnected via a set of tile interconnects 323A-323F, which may be connected to and / or interconnected by fabric interconnect 324. In one embodiment, compute accelerator 330 includes a large L3 cache 336, which may be configured as a device-wide cache. Compute accelerator 330 may also be connected to a host processor and memory via host interface 328, in a manner similar to graphics processor 320 of FIG. 3B.

[0098] Graphics Processing Engine FIG. 4 is a block diagram of a graphics processing engine 410 of a graphics processor according to some embodiments. In one embodiment, the graphics processing engine (GPE) 410 is a version of the GPE 310 shown in FIG. 3A and may represent the graphics engine tiles 310A-310D of FIG. 3B. Elements in FIG. 4 having the same reference number (or name) as elements in any other figure herein may operate or function in a manner similar to that described elsewhere herein, but are not limited to such. For example, the 3D pipeline 312 and media pipeline 316 of FIG. 3A are shown. The media pipeline 316 is optional in some embodiments of the GPE 410 and may not be explicitly included within the GPE 410. For example, in at least one embodiment, a separate media and / or image processor is coupled to the GPE 410.

[0099] In some embodiments, the GPE 410 is coupled to or includes a command streamer 403 that provides a command stream to the 3D pipeline 312 and / or the media pipeline 316. In some embodiments, the command streamer 403 is coupled to a memory, which may be system memory or one or more of an internal cache memory and a shared cache memory. In some embodiments, the command streamer 403 receives commands from memory and sends the commands to the 3D pipeline 312 and / or the media pipeline 316. The commands are directives fetched from a ring buffer that stores commands for the 3D pipeline 312 and the media pipeline 316. In one embodiment, the ring buffer may additionally include a batch command buffer that stores batches of multiple commands. Commands for the 3D pipeline 312 may also include references to data stored in memory, such as, but not limited to, vertex and geometry data for the 3D pipeline 312 and / or image data and memory objects for the media pipeline 316. 3D pipeline 312 and media pipeline 316 process commands and data by executing operations through logic within their respective pipelines or by dispatching one or more threads of execution to graphics core array 414. In one embodiment, graphics core array 414 includes one or more blocks of graphics cores (e.g., graphics core 415A, graphics core 415B), each block including one or more graphics cores. Each graphics core includes a set of graphics execution resources, including general-purpose and graphics-specific execution logic for performing graphics and computational processing, as well as fixed-function texture processing and / or machine learning and artificial intelligence acceleration logic.

[0100] In various embodiments, the 3D pipeline 312 may include fixed-function and programmable logic to process one or more shader programs, such as vertex shaders, geometry shaders, pixel shaders, fragment shaders, compute shaders, or other shader programs, by processing instructions and dispatching execution threads to the graphics core array 414. The graphics core array 414 provides a unified block of execution resources for use in processing these shader programs. The multipurpose execution logic (e.g., execution units) within the graphics cores 415A-414B of the graphics core array 414 includes support for various 3D API shader languages ​​and is capable of running multiple concurrent threads of execution associated with multiple shaders.

[0101] In some embodiments, graphics core array 414 includes execution logic for performing media functions such as video and / or image processing. In one embodiment, the execution units include general-purpose logic that is programmable to perform parallel general-purpose computing operations in addition to graphics processing operations. The general-purpose logic can perform processing operations in parallel with or in conjunction with general-purpose logic in processor core 107 of FIG. 1 or cores 202A-202N, such as those in FIG. 2A.

[0102] Output data generated by threads executing on graphics core array 414 can output data to memory in unified return buffer (URB) 418. URB 418 can store data for multiple threads. In some embodiments, URB 418 may be used to send data between different threads executing on graphics core array 414. In some embodiments, URB 418 may additionally be used for synchronization between threads on the graphics core array and fixed function logic in shared function logic 420.

[0103] In some embodiments, graphics core array 414 is scalable such that the array includes a variable number of graphics cores, each with a variable number of execution units based on the target power and performance level of GPE 410. In one embodiment, execution resources are dynamically scalable such that execution resources may be enabled or disabled as needed.

[0104] Graphics core array 414 couples with shared function logic 420, which includes multiple resources shared among the graphics cores in the graphics core array. Shared functions within shared function logic 420 are hardware logic units that provide specialized complementary functionality to graphics core array 414. In various embodiments, shared function logic 420 includes, but is not limited to, sampler 421, math 422, and inter-thread communication (ITC) 423 logic. Additionally, some embodiments implement one or more caches 425 within shared function logic 420.

[0105] Shared functionality is implemented at least when demand for a given specialized function is insufficient to justify inclusion within graphics core array 414. Instead, a single instance of that specialized function is implemented as a stand-alone entity within shared functionality logic 420 and shared among execution resources within graphics core array 414. The exact set of functionality shared among and contained within graphics core array 414 varies between embodiments. In some embodiments, certain shared functionality within shared functionality logic 420 that is widely used by graphics core array 414 may be included within shared functionality logic 416 within graphics core array 414. In various embodiments, shared functionality logic 416 within graphics core array 414 may include some or all of the logic within shared functionality logic 420. In one embodiment, all logic elements within shared functionality logic 420 may be duplicated within shared functionality logic 416 of graphics core array 414. In one embodiment, shared function logic 420 is omitted in favor of shared function logic 416 within graphics core array 414 .

[0106] Execution Unit 5A-5B illustrate thread execution logic 500 including an array of processing elements for use in a graphics processor core, according to embodiments described herein. Elements in FIGS. 5A-5B having the same reference numbers (or names) as elements in other figures herein may operate or function in a manner similar to that described elsewhere herein, but are not limited to such. FIGS. 5A-5B illustrate an overview of thread execution logic 500, which may represent the hardware logic shown in each of sub-cores 221A-221F in FIG. 2B. FIG. 5A represents an execution unit in a general-purpose graphics processor, while FIG. 5B represents an execution unit that may be used in a computer accelerator.

[0107] 5A , in some embodiments, thread execution logic 500 includes a shader processor 502, a thread dispatcher 504, an instruction cache 506, a scalable execution unit including multiple execution units 508A-508N, a sampler 510, a shared local memory 511, a data cache 512, and a data port 514. In one embodiment, the scalable execution unit array is dynamically scalable by enabling or disabling one or more execution units (e.g., any of execution units 508A, 508B, 508C, 508D, through 508N-1 and 508N) based on the computational requirements of a workload. In one embodiment, the included components are interconnected via an interconnect structure that links each of the components. In some embodiments, the thread execution logic 500 includes one or more connections to memory, such as system memory or cache memory, via one or more of the instruction cache 506, the data port 514, the sampler 510, and the execution units 508A-508N. In some embodiments, each execution unit (e.g., 508A) is a standalone programmable general-purpose computational unit capable of executing multiple simultaneous hardware threads while processing multiple data elements in parallel for each thread. In various embodiments, the array of execution units 508A-508N is scalable to include any number of individual execution units.

[0108] In some embodiments, the execution units 508A-508N are primarily used to execute shader programs. The shader processor 502 processes various shader programs and can dispatch execution threads associated with the shader programs via the thread dispatcher 504. In one embodiment, the thread dispatcher includes logic that arbitrates thread initiation requests from the graphics and media pipelines and instantiates the requested threads in one or more execution units within the execution units 508A-508N. For example, the geometry pipeline can dispatch vertex, tessellation, or geometry shaders to thread execution logic for processing. In some embodiments, the thread dispatcher 504 can also handle runtime thread creation requests from executing shader programs.

[0109] In some embodiments, the execution units 508A-508N support an instruction set that includes native support for many standard 3D graphics shader instructions, allowing shader programs from graphics libraries (e.g., Direct3D and OpenGL) to execute with minimal translation. The execution units support vertex and geometry processing (e.g., vertex programs, geometry programs, vertex shaders), pixel processing (e.g., pixel shaders, fragment shaders), and general-purpose processing (e.g., compute and media shaders). Each of the execution units 508A-508N is capable of multi-issue single-instruction, multiple-data (SIMD) execution, and multi-threaded operation enables an efficient execution environment when facing higher-latency memory accesses. Each hardware thread within each execution unit has a dedicated high-bandwidth register file and associated independent thread state. Execution is multi-issue per clock for a pipeline capable of integer, single- and double-precision floating-point operations, SIMD branching capability, logical operations, transcendental operations, and other operations. While waiting for data from memory or one of the shared functions, dependency logic within the execution units 508A-508N causes the waiting thread to sleep until the requested data is returned. While the waiting thread is sleeping, hardware resources may be allocated to processing other threads. For example, during a delay associated with a vertex shader operation, the execution unit may execute operations of a pixel shader, fragment shader, or another type of shader program (including a vertex shader). Various embodiments may apply to the use of execution using single instruction multiple threads (SIMT) as an alternative to or in addition to using SIMD. References to SIMD cores or operations may also apply to SIMT or SIMD in combination with SIMT.

[0110] Each execution unit of execution units 508A-508N operates on an array of data elements. The number of data elements is the "execution size," or the number of channels of the instruction. An execution channel is a logical unit of execution for data element access, masking, and flow control within an instruction. The number of channels may be independent of the number of physical arithmetic logic units (ALUs) or floating-point units (FPUs) for a particular graphics processor. In some embodiments, execution units 508A-508N support integer and floating-point data types.

[0111] The execution unit instruction set includes SIMD instructions. Various data elements can be stored in registers as packed data types, and the execution unit processes the various elements based on the element's data size. For example, when operating on a 256-bit wide vector, the 256 bits of the vector are stored in a register, and the execution unit operates on the vector as four separate 54-bit packed data elements (quad-word (QW) size data elements), eight separate 32-bit packed data elements (double-word (DW) size data elements), 16 separate 16-bit packed data elements (word (W) size data elements), or 32 separate 8-bit data elements (byte (B) size data elements). However, different vector widths and register sizes are possible.

[0112] In one embodiment, one or more execution units may be combined into fused execution units 509A-509N with thread control logic (507A-507N) common to the fused EUs. Multiple EUs may be fused into EU groups. Each EU within a fused EU group may be configured to execute a separate SIMD hardware thread. The number of EUs within a fused EU group may vary depending on the embodiment. Additionally, various SIMD widths may be implemented per EU, including, but not limited to, SIMD8, SIMD16, and SIMD32. Each fused graphics execution unit 509A-509N includes at least two execution units. For example, fused execution unit 509A includes a first EU 508A, a second EU 508B, and thread control logic 507A common to the first EU 508A and the second EU 508B. Thread control logic 507A controls the threads executing on fused graphics execution unit 509A, allowing each EU in fused execution units 509A-509N to execute using a common instruction pointer register.

[0113] One or more internal instruction caches (e.g., 506) are included in the thread execution logic 500 for caching thread instructions for the execution units. In some embodiments, one or more data caches (e.g., 512) are included for caching thread data during thread execution. Threads executing on the execution logic 500 can also store explicitly managed data in shared local memory 511. In some embodiments, a sampler 510 is included to provide texture sampling for 3D processing and media sampling for media processing. In some embodiments, the sampler 510 includes specialized texture or media sampling functions for processing texture or media data during the sampling process before providing the sampled data to the execution units.

[0114] During execution, the graphics and media pipeline sends thread start requests to the thread execution logic 500 via thread creation and dispatch logic. Once a group of geometric objects has been processed and rasterized into pixel data, pixel processor logic (e.g., pixel shader logic, fragment shader logic, etc.) within the shader processor 502 is invoked to further compute output information and write the results to an output surface (e.g., a color buffer, a depth buffer, a stencil buffer, etc.). In some embodiments, a pixel shader or fragment shader computes values ​​for various vertex attributes to be interpolated across the rasterized object. In some embodiments, the pixel processor logic within the shader processor 502 then executes an application programming interface (API)-supplied pixel or fragment shader program. To execute a shader program, the shader processor 502 dispatches threads to execution units (e.g., 508A) via the thread dispatcher 504. In some embodiments, the shader processor 502 uses texture sampling logic in the sampler 510 to access texture data in texture maps stored in memory. Arithmetic operations on the texture data and input geometry data calculate pixel color data for each geometric piece or eliminate one or more pixels from further processing.

[0115] In some embodiments, data port 514 provides a memory access mechanism for thread execution logic 500 and outputs processed data to memory for further processing in the graphics processor output pipeline. In some embodiments, data port 514 includes or is coupled to one or more cache memories (e.g., data cache 512) for caching data for memory accesses via the data port.

[0116] In one embodiment, execution logic 500 may also include a ray tracer 505 capable of providing ray tracing acceleration functionality. Ray tracer 505 may support a ray tracing instruction set that includes instructions / functions for ray generation. The ray tracing instruction set may be similar to or different from the ray tracing instruction set supported by ray tracing core 245 of FIG. 2C.

[0117] 5B illustrates exemplary internal details of an execution unit 508 according to an embodiment. The graphics execution unit 508 may include an instruction fetch unit 537, a general-purpose register file array (GRF) 524, an architectural register file array (ARF) 526, a thread arbiter 522, a send unit 530, a branch unit 532, a set of SIMD floating-point units (FPUs) 534, and, in one embodiment, a set of dedicated integer SIMD ALUs 535. The GRF 524 and ARF 526 include a set of general-purpose register files and architectural register files associated with each concurrent hardware thread that may be active in the graphics execution unit 508. In one embodiment, per-thread architectural state is maintained within the ARF 526, and data used during thread execution is stored within the GRF 524. The execution state of each thread, including the instruction pointer for each thread, may be kept in thread-specific registers within the ARF 526.

[0118] In one embodiment, the graphics execution unit 508 has an architecture that is a combination of simultaneous multithreading (SMT) and finely interleaved multithreading (IMT). The architecture has a modular configuration that is tunable at design time based on the target number of simultaneous threads and the number of registers per execution unit, where execution unit resources are divided across the logic used to execute multiple simultaneous threads. The number of logical threads that can be executed by the graphics execution unit 508 is not limited to the number of hardware threads, and multiple logical threads can be assigned to each hardware thread.

[0119] In one embodiment, the graphics execution unit 508 can issue multiple instructions together, each of which may be a different instruction. The thread arbiter 522 of a graphics execution unit thread 508 can dispatch instructions to the send unit 530, the branch unit 532, or one of the SIMD FPUs for execution. Each execution thread can access 128 general-purpose registers in the GRF 524, each of which can store 32 bytes and is accessible as a SIMD 8-element vector of 32-bit data elements. In one embodiment, each execution unit thread has access to 4K bytes in the GRF 524, although embodiments are not so limited, and other embodiments may provide greater or lesser register resources. In one embodiment, the graphics execution unit 508 is divided into seven hardware threads capable of independently performing computational operations, although the number of threads per execution unit can vary depending on the embodiment. For example, in one embodiment, up to 16 hardware threads are supported. In an embodiment where seven threads may access 4K bytes, the GRF524 can store a total of 28K bytes. If 16 threads can access 4K bytes, the GRF524 can store a total of 64K bytes. Flexible addressing modes allow registers to be addressed together, effectively building wider registers or representing strided rectangular block data structures.

[0120] In one embodiment, memory operations, sampler operations, and other longer latency system communications are dispatched by "send" instructions executed by message passing send unit 530. In one embodiment, branch instructions are dispatched to dedicated branch unit 532 to facilitate SIMD diversity and eventual convergence.

[0121] In one embodiment, the graphics execution unit 508 includes one or more SIMD floating-point units (FPUs) 534 for performing floating-point operations. In one embodiment, the FPU(s) 534 also support integer calculations. In one embodiment, the FPUs 534 can SIMD up to M 32-bit floating-point (or integer) operations, or SIMD up to 2M 16-bit integer or 16-bit floating-point operations. In one embodiment, at least one of the FPUs provides extended math capabilities supporting high-throughput transcendental math functions and double-precision 54-bit floating-point. In some embodiments, a set of 8-bit integer SIMD ALUs 535 are also present and may be specially optimized for performing operations related to machine learning calculations.

[0122] In one embodiment, an array of multiple instances of graphics execution unit 508 can be instantiated in graphics sub-core groupings (e.g., sub-slices). For scalability, product architects can select the exact number of execution units per sub-core group. In one embodiment, execution unit 508 can execute instructions across multiple execution channels. In a further embodiment, each thread executing on a graphics execution unit 508 executes on a different channel.

[0123] FIG. 6 illustrates an additional execution unit 600 according to one embodiment. The execution unit 600 may be a computationally optimized execution unit for use in the compute engine tiles 340A-340D, such as those in FIG. 3C, but is not limited to such. Variations of the execution unit 600 may also be used in the graphics engine tiles 310A-310D, as shown in FIG. 3B. In one embodiment, the execution unit 600 includes a thread control unit 601, a thread state unit 602, an instruction fetch / prefetch unit 603, and an instruction decode unit 604. The execution unit 600 further includes a register file 606 that stores registers that can be allocated to hardware threads within the execution unit. The execution unit 600 additionally includes a send unit 607 and a branch unit 608. In one embodiment, the send unit 607 and the branch unit 608 may operate similarly to the send unit 530 and the branch unit 532 of the graphics execution unit 508 of FIG. 5B.

[0124] The execution unit 600 also includes a compute unit 610 that includes multiple different types of functional units. In one embodiment, the compute unit 610 includes an ALU unit 611 that includes an array of arithmetic logic units. The ALU unit 611 can be configured to perform 64-bit, 32-bit, and 16-bit integer and floating-point operations. The integer and floating-point operations may be performed simultaneously. The compute unit 610 can also include a systolic array 612 and a math unit 613. The systolic array 612 includes a W-wide and D-deep network of data processing units that can be used to perform vector or other data-parallel operations in a systolic fashion. In one embodiment, the systolic array 612 can be configured to perform matrix operations, such as matrix dot-product operations. In one embodiment, the systolic array 612 supports 16-bit floating-point operations and 8-bit and 4-bit integer operations. In one embodiment, the systolic array 612 can be configured to accelerate machine learning operations. In such an embodiment, systolic array 612 may be configured to support the bfloat 16-bit floating-point format. In one embodiment, math unit 613 may be included to perform a specific subset of mathematical operations in a more efficient and lower power manner than ALU unit 611. Math unit 613 may include a variation of math logic found in the shared function logic of a graphics processing engine provided by other embodiments (e.g., math logic 422 of shared function logic 420 of FIG. 4). In one embodiment, math unit 613 may be configured to perform 32-bit and 64-bit floating-point operations.

[0125] The thread control unit 601 contains logic that controls the execution of threads within the execution units. The thread control unit 601 may include thread arbitration logic that starts, stops, and preempts the execution of threads within the execution units 600. The thread state unit 602 may be used to store thread state for threads assigned to execute on the execution units 600. Storing thread state within the execution units 600 allows for rapid preemption of threads if they become blocked or idle. The instruction fetch / prefetch unit 603 may fetch instructions from an instruction cache of a higher-level execution logic (e.g., instruction cache 506, such as in FIG. 5A). The instruction fetch / prefetch unit 603 may also issue prefetch requests for instructions to be loaded into the instruction cache based on an analysis of the currently executing thread. The instruction decode unit 604 may be used to decode instructions to be executed by the compute units. In one embodiment, the instruction decode unit 604 may be used as a secondary decoder to decode complex instructions into their micro-operation components.

[0126] The execution unit 600 additionally includes a register file 606 that can be used by hardware threads executing on the execution unit 600. The registers in the register file 606 can be divided across logic used to execute multiple simultaneous threads within the compute units 610 of the execution unit 600. The number of logical threads that may be executed by the graphics execution unit 600 is not limited to the number of hardware threads, and multiple logical threads can be assigned to each hardware thread. The size of the register file 606 can vary depending on the embodiment based on the number of hardware threads supported. In one embodiment, register renaming can be used to dynamically allocate registers to hardware threads.

[0127] FIG. 7 is a block diagram illustrating a graphics processor instruction format 700 according to some embodiments. In one or more embodiments, a graphics processor execution unit supports an instruction set having instructions in multiple formats. Solid boxes indicate components that are typically included in execution unit instructions, while dashed lines include components that are optional or that are only included in a subset of instructions. In some embodiments, the instruction format 700 described and illustrated are macro-instructions, in that they are instructions supplied to the execution unit as the instruction is processed, as opposed to micro-operations that result from instruction decode.

[0128] In some embodiments, the graphics processor execution units natively support instructions in 128-bit instruction format 710. A 64-bit compact instruction format 730 is available for some instructions based on the selected instruction, instruction options, and number of operands. The native 128-bit instruction format 710 provides access to all instruction options, while some options and operations are limited in 64-bit format 730. The native instructions available in 64-bit format 730 vary depending on the embodiment. In some embodiments, instructions are compacted in part using a set of index values ​​in index field 713. The execution unit hardware looks up a set of compression tables based on the index values ​​and uses the compression table outputs to reconstruct the native instruction in 128-bit instruction format 710. Other sizes and formats of instructions are possible.

[0129] For each format, the instruction opcode 712 defines the operation the execution unit performs. The execution unit executes each instruction in parallel across multiple data elements of each operand. For example, in response to an add instruction, the execution unit performs a simultaneous add operation across each color channel representing a texture or picture element. By default, the execution unit executes each instruction across all data channels of the operand. In some embodiments, the instruction control field 714 allows control of certain execution options, such as channel selection (e.g., prediction) and data channel order (e.g., swizzle). For instructions in the 128-bit instruction format 710, the exec-size field 716 limits the number of data channels that are executed in parallel. In some embodiments, the exec-size field 716 is not available for use with the 64-bit compact instruction format 730.

[0130] Some execution unit instructions have up to three operands, including two source operands, src0 720 and src1 722, and one destination 718. In some embodiments, the execution units support dual-destination instructions, where one of the destinations is implicit. Data manipulation instructions may have a third source operand (e.g., SRC2 724), and the instruction opcode 712 determines the number of source operands. The last source operand of an instruction may be a direct (e.g., hard-coded) value passed by the instruction.

[0131] In some embodiments, the 128-bit instruction format 710 includes an access / address mode field 726 that specifies, for example, whether direct register addressing mode or indirect register addressing mode is used. When direct register addressing mode is used, the register addresses of one or more operands are provided directly by bits in the instruction.

[0132] In some embodiments, the 128-bit instruction format 710 includes an access / address mode field 726 that specifies the access mode and / or address mode of the instruction. In one embodiment, the access mode is used to define the data access alignment for the instruction. Some embodiments support access modes including a 16-byte aligned access mode and a 1-byte aligned access mode, where the byte alignment of the access mode determines the access alignment of the instruction operands. For example, when in a first mode, the instruction may use byte-aligned addressing for source and destination operands, and when in a second mode, the instruction may use 16-byte aligned addresses for all source and destination operands.

[0133] In one embodiment, the address mode portion of the access / address mode field 726 determines whether the instruction uses direct or indirect addressing. If direct register addressing mode is used, bits in the instruction directly provide the register addresses of one or more operands. If indirect register addressing mode is used, the register addresses of one or more operands may be calculated based on the address immediate field in the instruction and the address register value.

[0134] In some embodiments, instructions are grouped based on opcode 712 bit fields to simplify opcode decode 740. In an 8-bit opcode, bits 4, 5, and 6 allow the execution unit to determine the type of opcode. The distinct opcode groupings shown are merely illustrative. In some embodiments, the move and logical opcode group 742 includes data movement and logical instructions (e.g., move (mov), compare (cmp)). In some embodiments, the move and logical group 742 share five most significant bits (MSBs), where move (mov) instructions are of the form 0000xxxxb and logical instructions are of the form 0001xxxb. The flow control instruction group 744 (e.g., call, jump (jmp)) includes instructions of the form 0010xxxxb (e.g., 0x20). The other instruction group 746 contains a mix of instructions, including synchronization instructions (e.g., wait, send) in the format of 0011xxxxb (e.g., 0x30). The parallel math instruction group 748 contains component-wise arithmetic instructions (e.g., add, multiply) in the format of 0100xxxxb (e.g., 0x40). The parallel math group 748 performs arithmetic operations in parallel across data channels. The vector math group 750 contains arithmetic instructions (e.g., dp4) in the format of 0101xxxxb (e.g., 0x50). The vector math group performs operations such as dot product calculations of vector operands. The illustrated opcode decode 740, in one embodiment, can be used to determine which portion of the execution unit will be used to execute the decoded instruction. For example, some instructions may be designated as systolic instructions to be executed by a systolic array. Other instructions, such as ray tracing instructions (not shown), may be routed to a ray tracing core or ray tracing logic within a slice or partition of the execution logic.

[0135] Graphics Pipeline Figure 8 is a block diagram of another embodiment of a graphics processor 800. Elements of Figure 8 having the same reference number (or name) as elements of any other figure in this application may operate or function in a manner similar to that described elsewhere in this application, but are not limited to such.

[0136] In some embodiments, graphics processor 800 includes a geometry pipeline 820, a media pipeline 830, a display engine 840, thread execution logic 850, and a rendering output pipeline 870. In some embodiments, graphics processor 800 is a graphics processor within a multi-core processing system that includes one or more general-purpose processing cores. The graphics processor is controlled by register writes to one or more control registers (not shown) or by commands issued to graphics processor 800 via a ring interconnect 802. In some embodiments, ring interconnect 802 couples graphics processor 800 to other processing components, such as other graphics processors or general-purpose processors. Commands from ring interconnect 802 are interpreted by a command streamer 803, which provides instructions to individual components of geometry pipeline 820 or media pipeline 830.

[0137] In some embodiments, command streamer 803 directs the operation of vertex fetcher 805, which reads vertex data from memory and executes vertex processing commands provided by command streamer 803. In some embodiments, vertex fetcher 805 provides vertex data to vertex shader 807, which performs coordinate space transformations and lighting operations on each vertex. In some embodiments, vertex fetcher 805 and vertex shader 807 execute vertex processing instructions by dispatching execution threads to execution units 852A-852B via thread dispatcher 831.

[0138] In some embodiments, the execution units 852A-852B are arrays of vector processors with instruction sets for performing graphics and media operations. In some embodiments, the execution units 852A-852B have an attached L1 cache 851 that is unique to each array or shared between the arrays. The cache can be configured as a data cache, an instruction cache, or a single cache partitioned to contain data and instructions in different partitions.

[0139] In some embodiments, geometry pipeline 820 includes a tessellation component for performing hardware-accelerated tessellation of 3D objects. In some embodiments, programmable hull shader 811 configures the tessellation operation. Programmable domain shader 817 provides back-end evaluation of the tessellation output. Tessellator 813 operates at the direction of hull shader 811 and includes special-purpose logic for generating a set of detailed geometric objects based on a coarse geometric model provided as input to geometry pipeline 820. In some embodiments, the tessellation components (e.g., hull shader 811, tessellator 813, and domain shader 817) can be bypassed if tessellation is not used.

[0140] In some embodiments, complete geometric objects are processed by geometry shader 819 by one or more threads dispatched to execution units 852A-852B, or can proceed directly to clipper 829. In some embodiments, geometry shaders operate on entire geometric objects rather than vertices or patches of vertices like previous stages in the graphics pipeline. When tessellation is disabled, geometry shader 819 receives input from vertex shader 807. In some embodiments, geometry shader 819 is programmable by a geometry shader program to perform geometry tessellation when the tessellation unit is disabled.

[0141] Before rasterization, a clipper 829 processes the vertex data. The clipper 829 may be a fixed-function clipper or a programmable clipper with clipping and geometry shader functions. In some embodiments, a rasterizer and depth test component 873 in the rendering output pipeline 870 dispatches pixel shaders to convert geometric objects into pixel-by-pixel representations. In some embodiments, pixel shader logic is included in the thread execution logic 850. In some embodiments, an application can bypass the rasterizer and depth test component 873 and access unrasterized vertex data through the stream output unit 823.

[0142] Graphics processor 800 includes an interconnect bus, interconnect fabric, or some other interconnect mechanism that allows data and message transmission between major components of the processor. In some embodiments, execution units 852A-852B and associated logic units (e.g., L1 cache 851, sampler 854, texture cache 858, etc.) interconnect via data port 856 to perform memory accesses and communicate with the processor's rendering output pipeline components. In some embodiments, sampler 854, caches 851, 858, and execution units 852A-852B each have a separate memory access path. In one embodiment, texture cache 858 can be configured as a sampler cache.

[0143] In some embodiments, the rendering output pipeline 870 includes a rasterizer and depth test component 873 that converts vertex-based objects into associated pixel-based representations. In some embodiments, the rasterizer logic includes a window / mask unit for performing fixed-function triangle and line rasterization. An associated rendering cache 878 and depth cache 879 are also available in some embodiments. A pixel operations component 877 performs pixel-based operations on the data, although in some instances, pixel operations related to 2D operations (e.g., bit-block image transfers with blending) are performed by the 2D engine 841 or are replaced by the display controller 843 using overlay display planes at display time. In some embodiments, a shared L3 cache 875 is available to all graphics components, allowing data sharing without using main system memory.

[0144] In some embodiments, graphics processor media pipeline 830 includes a media engine 837 and a video front end 834. In some embodiments, video front end 834 receives pipeline commands from command streamer 803. In some embodiments, media pipeline 830 includes individual command streamers. In some embodiments, video front end 834 processes media commands before sending the commands to media engine 837. In some embodiments, media engine 837 includes a thread spawning function that spawns threads for dispatching by thread dispatcher 831 to thread execution logic 850.

[0145] In some embodiments, graphics processor 800 includes a display engine 840. In some embodiments, display engine 840 is external to processor 800 and couples to the graphics processor via ring interconnect 802 or some other interconnect bus or fabric. In some embodiments, display engine 840 includes a 2D engine 841 and a display controller 843. In some embodiments, display engine 840 includes dedicated logic capable of operating independently from the 3D pipeline. In some embodiments, display controller 843 couples to a display device (not shown), which may be a system-integrated display device, such as in a laptop computer, or an external display device attached via a display device connector.

[0146] In some embodiments, the geometry pipeline 820 and the media pipeline 830 are configurable to perform operations based on multiple graphics and media programming interfaces and are not specific to any one application programming interface (API). In some embodiments, driver software for the graphics processor translates API calls specific to a particular graphics or media library into commands that can be processed by the graphics processor. In some embodiments, support for the Open Graphics Library (OpenGL), Open Computing Language (OpenCL), and / or Vulkan graphics and compute APIs, all from the Khronos Group, is provided. In some embodiments, support for the Direct3D library from Microsoft Corporation may be provided. In some embodiments, combinations of these libraries may be supported. Support for the Open Source Computer Vision Library (OpenCV) may be provided. Future APIs with compatible 3D pipelines may also be supported, provided that a mapping from the future API's pipeline to the graphics processor's pipeline is possible.

[0147] Graphics Pipeline Programming Figure 9A is a block diagram illustrating a graphics processor command format 900 according to some embodiments. Figure 9B is a block diagram illustrating a graphics processor command sequence 910 according to an embodiment. Solid boxes in Figure 9A indicate components typically included in a graphics command, while dashed lines indicate components that are optional or that are only included in a subset of graphics commands. The example graphics processor command format 900 of Figure 9A includes data fields for identifying a client 902, a command operation code (opcode) 904, and data for the command 906. A sub-opcode 905 and a command size 908 are also included in some commands.

[0148] In some embodiments, client 902 specifies a client unit of the graphics device that will process the command data. In some embodiments, a graphics processor command parser examines the client field of each command to condition further processing of the command and route the command data to the appropriate client unit. In some embodiments, the graphics processor client units include a memory interface unit, a rendering unit, a 2D unit, a 3D unit, and a media unit. Each client unit has a corresponding processing pipeline that processes the command. When a command is received by a client unit, the client unit reads the opcode 904 and, if any, the sub-opcode 905 to determine the operation to perform. The client unit executes the command using the information in data field 906. For some commands, an explicit command size 908 is expected to specify the size of the command. In some embodiments, the command parser automatically determines the size of at least some commands based on the command opcode. In some embodiments, commands are aligned in multiples of double words. Other command formats are also possible.

[0149] The flow diagram of Figure 9B illustrates an exemplary graphics processor command sequence 910. In some embodiments, software or firmware in a data processing system featuring an embodiment of a graphics processor uses versions of the command sequence shown to set up, execute, and terminate a set of graphics operations. The sample command sequence is shown and described for illustrative purposes only, as embodiments are not limited to these particular commands or this command sequence. Furthermore, commands may be issued as a batch of commands within a command sequence such that the graphics processor processes the sequence of commands at least partially concurrently.

[0150] In some embodiments, the graphics processor command sequence 910 begins with a pipeline flush command 912, which causes any active graphics pipelines to complete any commands currently outstanding for that pipeline. In some embodiments, the 3D pipeline 922 and the media pipeline 924 do not operate simultaneously. The pipeline flush is executed to cause the active graphics pipelines to complete any outstanding commands. In response to the pipeline flush, the graphics processor's command parser pauses command processing until the active drawing engines complete any outstanding operations and the associated read caches are disabled. Optionally, any data in the rendering cache that is marked "dirty" can be flushed to memory. In some embodiments, the pipeline flush command 912 can be used for pipeline synchronization or before placing the graphics processor in a low-power state.

[0151] In some embodiments, a pipeline select command 913 is used when a command sequence requires the graphics processor to explicitly switch between pipelines. In some embodiments, the pipeline select command 913 is only needed once within an execution context before issuing a pipeline command, unless the context issues commands to both pipelines. In some embodiments, a pipeline flush command 912 is needed immediately before pipeline switching via the pipeline select command 913.

[0152] In some embodiments, pipeline control commands 914 are used to configure the graphics pipeline for operation and to program the 3D pipeline 922 and the media pipeline 924. In some embodiments, pipeline control commands 914 set the pipeline state for the active pipeline. In one embodiment, pipeline control commands 914 are used to clear data from one or more cache memories in the active pipeline before processing a batch of commands and for pipeline synchronization.

[0153] In some embodiments, the return buffer state command 916 is used to configure the set of return buffers to which each pipeline writes data. Some pipeline operations require the allocation, selection, or configuration of one or more return buffers to which the operation writes intermediate data during processing. In some embodiments, the graphics processor also uses one or more return buffers to store output data and for cross-thread communication. In some embodiments, the return buffer state 916 includes selecting the size and number of return buffers to use for a set of pipeline operations.

[0154] The remaining commands in the command sequence differ based on the active pipeline for operation. Based on the pipeline decision 920, the command sequence is tailored to either the 3D pipeline 922, starting with the 3D pipeline state 930, or the media pipeline 924, starting with the media pipeline state 940.

[0155] Commands for setting the 3D pipeline state 930 include 3D state setting commands for vertex buffer state, vertex element state, constant color state, depth buffer state, and other state variables that must be set before 3D primitive commands are processed. The values ​​of these commands are determined at least in part based on the particular 3D API in use. In some embodiments, the 3D pipeline state 930 commands can also selectively disable or bypass certain pipeline elements if those elements are not used.

[0156] In some embodiments, the 3D primitive 932 command is used to submit a 3D primitive to be processed by the 3D pipeline. The command and associated parameters passed to the graphics processor via the 3D primitive 932 command are forwarded to a vertex fetch function in the graphics pipeline. The vertex fetch function uses the 3D primitive 932 command data to generate a vertex data structure. The vertex data structure is stored in one or more return buffers. In some embodiments, the 3D primitive 932 command is used to perform vertex operations on the 3D primitive by a vertex shader. To process the vertex shader, the 3D pipeline 922 dispatches shader execution threads to the graphics processor execution units.

[0157] In some embodiments, the 3D pipeline 922 is triggered by an execute 934 command or event. In some embodiments, a register write triggers command execution. In some embodiments, execution is triggered by a "go" or "kick" command in a command sequence. In one embodiment, command execution is triggered using a pipeline synchronization command to flush the command sequence through the graphics pipeline. The 3D pipeline performs geometric operations on 3D primitives. Once the operations are complete, the resulting geometric objects are rasterized, and the pixel engine colors the resulting pixels. Additional commands to control pixel shading and pixel backend operations may be included for these operations.

[0158] In some embodiments, the graphics processor command sequence 910 follows the path of the media pipeline 924 when executing media video. In general, the specific use and manner of programming the media pipeline 924 depends on the media and the computational operations being performed. Certain media decode operations may be offloaded to the media pipeline during media decoding. In some embodiments, the media pipeline may be bypassed, and media decode may be performed in whole or in part using resources provided by one or more general-purpose processing cores. In one embodiment, the media pipeline also includes elements for general-purpose graphics processor unit (GPGPU) operation, where the graphics processor is used to perform SIMD vector operations using computational shader programs not explicitly related to rendering graphics primitives.

[0159] In some embodiments, the media pipeline 924 is configured similarly to the 3D pipeline 922. A set of commands for setting the media pipeline state 940 is dispatched or placed in the command queue before the media object commands 942. In some embodiments, the commands for the media pipeline state 940 include data for configuring the media pipeline elements used to process the media objects. This includes data for configuring the video decode and video encode logic in the media pipeline, such as the encode or decode format. In some embodiments, the commands for the media pipeline state 940 also support the use of one or more pointers to "indirect" state elements that contain a batch of state settings.

[0160] In some embodiments, a media object command 942 provides a pointer to a media object for processing by the media pipeline. A media object contains a memory buffer containing the video data to be processed. In some embodiments, all media pipeline state must be "valid" before issuing a media object command 942. Once the pipeline state is set and the media object command 942 is queued, the media pipeline 924 is triggered by an execute command 944 or equivalent execution event (e.g., a register write). The output from the media pipeline 924 can then be post-processed by operations provided by the 3D pipeline 922 or the media pipeline 924. In some embodiments, GPGPU operations are structured and executed in a similar manner to media operations.

[0161] Graphics Software Architecture 10 illustrates an exemplary graphics software architecture for a data processing system 1000 according to some embodiments. In some embodiments, the software architecture includes a 3D graphics application 1010, an operating system 1020, and at least one processor 1030. In some embodiments, the processor 1030 includes a graphics processor 1032 and one or more general-purpose processor cores 1034. The graphics application 1010 and the operating system 1020 each execute within the system memory 1050 of the data processing system.

[0162] In some embodiments, the 3D graphics application 1010 includes one or more shader programs that include shader instructions 1012. The shader language instructions may be in a high-level shader language such as Direct3D's High-level Shader Language (HLSL), OpenGL Shader Language (GLSL), etc. The application also includes executable instructions 1014 in a machine language suitable for execution by a general-purpose processor core 1034. The application also includes graphics objects 1016 that are defined by vertex data.

[0163] In some embodiments, the operating system 1020 is a Microsoft® Windows® operating system from Microsoft Corporation, a proprietary UNIX®-like operating system, or an open-source UNIX®-like operating system using a variant of the Linux® kernel. The operating system 1020 may support a graphics API 1022, such as the Direct3D API, the OpenGL API, or the Vulkan API. If the Direct3D API is used, the operating system 1020 uses a front-end shader compiler 1024 to compile the HLSL shader instructions 1012 into a lower-level shader language. The compilation may be just-in-time (JIT) compilation, or the application may perform shader pre-compilation. In some embodiments, high-level shaders are compiled into low-level shaders during compilation of the 3D graphics application 1010. In some embodiments, the shader instructions 1012 are provided in an intermediate format, such as a version of the Standard Portable Intermediate Representation (SPIR) used by the Vulkan API.

[0164] In some embodiments, user-mode graphics driver 1026 includes a back-end shader compiler 1027 for converting shader instructions 1012 into a hardware-specific representation. When the OpenGL API is used, shader instructions 1012 in the GLSL high-level language are passed to user-mode graphics driver 1026 for compilation. In some embodiments, user-mode graphics driver 1026 communicates with kernel-mode graphics driver 1029 using operating system kernel-mode functions 1028. In some embodiments, kernel-mode graphics driver 1029 communicates with graphics processor 1032 to dispatch commands and instructions.

[0165] IP Core Implementation One or more aspects of at least one embodiment may be implemented by representative code stored on a machine-readable medium that represents and / or defines logic within an integrated circuit, such as a processor. For example, the machine-readable medium may include instructions that represent various logic within a processor. When read by a machine, the instructions cause the machine to form logic to perform the techniques described herein. Such representations, known as "IP cores," are reusable logic units for integrated circuits that can be stored on tangible machine-readable media as hardware models that describe the structure of the integrated circuit. The hardware models can be provided to various customers or manufacturing facilities, which load the hardware models into manufacturing machines that produce the integrated circuits. The integrated circuits can be manufactured such that the circuits perform the operations described in connection with any of the embodiments described herein.

[0166] FIG. 11A is a block diagram illustrating an IP core development system 1100 that can be used to fabricate integrated circuits to perform operations according to an embodiment. The IP core development system 1100 may be used to generate modular, reusable designs that can be incorporated into larger designs or used to build entire integrated circuits (e.g., SOC integrated circuits). A design facility 1130 can generate a software simulation 1110 of the IP core design in a high-level programming language (e.g., C / C++). The software simulation 1110 can be used to design, test, and verify the behavior of the IP core using a simulation model 1112. The simulation model 1112 may include functional, behavioral, and / or timing simulations. A register transfer level (RTL) design 1115 can then be created or synthesized from the simulation model 1112. The RTL design 1115 is an abstraction of the behavior of the integrated circuit that models the flow of digital signals between hardware registers, including the associated logic that is implemented using the modeled digital signals. In addition to the RTL design 1115, lower level designs at the logic or transistor level may be generated, designed, or synthesized. Therefore, the specific details of the initial design and simulation may vary.

[0167] The RTL design 1115 or equivalent may be further synthesized by a design facility into a hardware model 1120, which may be in a hardware description language (HDL) or some other representation of the physical design data. The HDL may be further simulated or tested to verify the IP core design. The IP core design may be stored using non-volatile memory 1140 (e.g., a hard disk, flash memory, or any non-volatile storage medium) for delivery to a third-party manufacturing facility 1165. Alternatively, the IP core design may be transmitted via a wired connection 1150 or a wireless connection 1160 (e.g., via the Internet). The manufacturing facility 1165 may then manufacture an integrated circuit based at least in part on the IP core design. The manufactured integrated circuit may be configured to perform operations in accordance with at least one embodiment described herein.

[0168] FIG. 11B illustrates a cross-sectional side view of an integrated circuit package assembly 1170 according to some embodiments described herein. The integrated circuit package assembly 1170 illustrates an implementation of one or more processor or accelerator devices described herein. The package assembly 1170 includes multiple units of hardware logic 1172, 1174 coupled to a substrate 1180. The logic 1172, 1174 may be implemented, at least in part, in hardware, such as configurable logic or fixed-function logic, and may include any one or more portions of a processor core, graphics processor, or other accelerator device described herein. Each unit of logic 1172, 1174 may be implemented within a semiconductor die and coupled to the substrate 1180 via an interconnect structure 1173. The interconnect structure 1173 may be configured to route electrical signals between the logic 1172, 1174 and the substrate 1180 and may include, but is not limited to, interconnects such as bumps or pillars. In some embodiments, interconnect structure 1173 may be configured to route electrical signals, such as input / output (I / O) signals and / or power or ground signals associated with the operation of logic 1172, 1174. In some embodiments, substrate 1180 is an epoxy-based laminate substrate. Substrate 1180 may include other suitable types of substrates in other embodiments. Package assembly 1170 can be connected to other electrical devices via package interconnect 1183. Package interconnect 1183 may be coupled to a surface of substrate 1180 to route electrical signals to other electrical devices, such as a motherboard, another chipset, or a multi-chip module.

[0169] In some embodiments, the units of logic 1172, 1174 are electrically coupled to a bridge 1182 configured to route electrical signals between the logic 1172, 1174. The bridge 1182 may be a high-density interconnect structure that provides paths for the electrical signals. The bridge 1182 may include a bridge substrate composed of glass or a suitable semiconductor material. Electrical routing features can be formed in the bridge substrate to provide chip-to-chip connections between the logic 1172, 1174.

[0170] Although two units of logic 1172, 1174 and bridge 1182 are shown, embodiments described herein may include more or fewer logic units on one or more dies. If logic is included on a single die, bridge 1182 may be omitted, such that one or more dies can be connected by zero or more bridges. Alternatively, multiple dies or logic units can be connected by one or more bridges. Furthermore, multiple logic units, dies, and bridges can be connected to each other in other possible configurations, including three-dimensional configurations.

[0171] FIG. 11C illustrates a package assembly 1190 including multiple units of hardware logic chiplets connected to a substrate 1180 (e.g., a base die). Graphics processing units, parallel processors, and / or computational accelerators, as described herein, can be composed of multiple silicon chiplets fabricated separately. In this context, a chiplet is an at least partially packaged integrated circuit containing individual logic units that can be assembled with other chiplets into a larger package. A diverse set of chiplets with different IP core logic can be assembled into a single device. Furthermore, chiplets can be integrated into a base die or base chiplet using active interposer technology. The concepts described herein enable interconnection and communication between various forms of IP within a GPU. IP cores can be fabricated using various process technologies and configured during manufacturing, thereby avoiding the complexity of concentrating multiple IP, especially on a large SoC with multiple flavors of IP, in the same manufacturing process. Enabling the use of multiple process technologies improves time to market and provides a cost-effective way to create multiple product SKUs. Additionally, the de-aggregated IP is independently power-gated, allowing components not used in a given workload to be powered off, reducing overall power consumption.

[0172] The hardware logic chiplets may include special-purpose hardware logic chiplets 1172, logic or I / O chiplets 1174, and / or memory chiplets 1175. The hardware logic chiplets 1172 and logic or I / O chiplets 1174 may be implemented, at least in part, in configurable logic or fixed-function logic hardware and may include any one or more portions of the processor cores, graphics processors, parallel processors, or other accelerator devices described herein. The memory chiplets 1175 may be DRAM (e.g., GDDR, HBM) memory or cache (SRAM) memory.

[0173] Each chiplet may be fabricated as a separate conductor die and coupled to substrate 1180 via interconnect structures 1173. Interconnect structures 1173 may be configured to route electrical signals between the various chiplets and logic within substrate 1180. Interconnect structures 1173 may include interconnects such as, but are not limited to, bumps or pillars. In some embodiments, interconnect structures 1173 may be configured to route electrical signals, such as, for example, input / output (I / O) signals and / or power or ground signals associated with the operation of the logic, I / O, and memory chiplets.

[0174] In some embodiments, substrate 1180 is an epoxy-based laminate substrate. Substrate 1180 may comprise other suitable types of substrate in other embodiments. Package assembly 1190 can be connected to other electrical devices via package interconnect 1183. Package interconnect 1183 may be coupled to a surface of substrate 1180 to route electrical signals to other electrical devices, such as a motherboard, another chipset, or a multi-chip module.

[0175] In some embodiments, logic or I / O chiplets 1174 and memory chiplets 1175 can be electrically coupled via bridges 1187 configured to route electrical signals between the logic or I / O chiplets 1174 and the memory chiplets 1175. Bridges 1187 can be high-density interconnect structures that provide routes for electrical signals. Bridges 1187 can include a bridge substrate made of glass or a suitable semiconductor material. Electrical routing features can be formed on the bridge substrate to provide chip-to-chip connections between the logic or I / O chiplets 1174 and the memory chiplets 1175. Bridges 1187 can also be referred to as silicon bridges or interconnect bridges. For example, bridges 1187 can be embedded multi-die interconnect bridges (EMIBs) in some embodiments. In some embodiments, bridges 1187 can simply be direct connections from one chiplet to another.

[0176] Substrate 1180 may include hardware components for I / O 1191, cache memory 1192, and other hardware logic 1193. Fabric 1185 may be embedded within substrate 1180 to enable communication between the various logic chiplets and the logic 1191, 1193 within substrate 1180. In one embodiment, I / O 1191, fabric 1185, cache, bridges, and other hardware logic 1193 may be integrated into a base die that is stacked on top of substrate 1180. Fabric 1185 may be a network of chip interconnects or another form of packet-switched fabric that switches data packets between components of a package assembly.

[0177] In various embodiments, package assembly 1190 can include fewer or more components and chiplets interconnected by fabric 1185 or one or more bridges 1187. Chilets within package assembly 1190 can be arranged in a 3D or 2.5D format. In general, bridge structure 1187 can be used to facilitate point-to-point interconnections between, for example, logic or I / O chiplets and memory chiplets. Fabric 1185 can be used to interconnect various logic and / or I / O chiplets (e.g., chiplets 1172, 1174, 1191, 1193) with other logic and / or I / O chiplets. In one embodiment, in-board cache memory 1192 can function as a global cache for package assembly 1190, part of a distributed global cache, or a dedicated cache for fabric 1185.

[0178] FIG. 11D illustrates a package assembly 1194 including a replaceable chiplet 1195, according to one embodiment. The replaceable chiplet 1195 can be assembled into a standardized slot on one or more base chiplets 1196, 1198. The base chiplets 1196, 1198 can be coupled via a bridge interconnect 1197, which can be similar to other bridge interconnects described herein, such as an EMIB. Memory chiplets can also be connected to logic or I / O chiplets via a bridge interconnect. I / O and logic chiplets can communicate via an interconnect fabric. Each base chiplet can support one or more slots in a standardized format for one of logic, I / O, or memory / cache.

[0179] In one embodiment, the SRAM and power distribution circuitry may be fabricated within one or more of the base chiplets 1196, 1198, which may be fabricated using a different process technology than the replaceable chiplets 1195 stacked on top of the base chiplets. For example, the base chiplets 1196, 1198 may be fabricated using a larger-scale process technology, and the replaceable chiplets may be fabricated using a smaller-scale process technology. One or more of the replaceable chiplets 1195 may be memory (e.g., DRAM) chiplets. Various memory densities may be selected for the package assembly 1194 based on the target power and / or performance for the product using the package assembly 1194. Furthermore, logic chiplets having a variety of functional units may be selected during assembly based on the target power and / or performance of the product. Furthermore, chiplets containing different types of IP logic cores may be inserted into the replaceable chiplet slots, enabling hybrid processor designs that can mix and match different technology IP blocks.

[0180] System example for chip integrated circuits 12-13B illustrate exemplary integrated circuits and associated graphics processors that may be fabricated using one or more IP cores according to various embodiments described herein. In addition to what is shown, other logic and circuitry may be included, including additional graphics processors / cores, peripheral interface controllers, or general-purpose processor cores.

[0181] 12 is a block diagram illustrating an exemplary system on a chip integrated circuit 1200 that can be manufactured using one or more IP cores according to one embodiment. The exemplary integrated circuit 1200 includes one or more application processors 1205 (e.g., CPUs) and at least one graphics processor 1210, and may further include an image processor 1215 and / or a video processor 1220, any of which may be modular IP cores from the same or multiple different design facilities. The integrated circuit 1200 also includes a USB controller 1225, a UART controller 1230, an SPI / SDIO controller 1235, and an I / O controller 1240. 2 S / I 2 The integrated circuit may include peripheral or bus logic including a C controller 1240. Additionally, the integrated circuit may include a display device 1245 coupled to one or more of a High-Definition Multimedia Interface (HDMI®) controller 1250 and a Mobile Industrial Processor Interface (MIPI) display interface 1255. Storage may be provided by a flash memory subsystem 1260 including a flash memory and a flash memory controller. Memory interface may be provided through a memory controller 1265 for access to SDRAM or SRAM memory devices. Some integrated circuits further include an embedded security engine 1270.

[0182] 13A-13B are block diagrams illustrating exemplary graphics processors for use within an SoC in accordance with embodiments described herein. FIG. 13A illustrates an exemplary graphics processor 1310 of a system on a chip integrated circuit that can be fabricated using one or more IP cores, according to an embodiment. FIG. 13B illustrates an additional exemplary graphics processor 1340 of a system on a chip integrated circuit that can be fabricated using one or more IP cores, according to an embodiment. The graphics processor 1310 of FIG. 13A is an example of a low-power graphics processor core. The graphics processor 1340 of FIG. 13B is an example of a high-performance graphics processor core. Each of the graphics processors 1310, 1340 may be a variation of the graphics processor 1210 of FIG. 12.

[0183] As shown in FIG. 13A , the graphics processor 1310 includes a vertex processor 1305 and one or more fragment processors (e.g., 1315A, 1315B, 1315C, 1315D, 1315N-1, and 1315N). The graphics processor 1310 is capable of executing different shader programs with separate logic, such that the vertex processor 1305 is optimized to perform operations on vertex shader programs, while one or more fragment processor(s) 1315A-1315N perform fragment (e.g., pixel) shader operations on fragment or pixel shader programs. The vertex processor 1305 executes the vertex processing stage of the 3D graphics pipeline and generates primitive and vertex data. The fragment processor(s) 1315A-1315N use the primitive and vertex data generated by the vertex processor 1305 to generate a frame buffer that is displayed on a display device. In one embodiment, the fragment processor(s) 1315A-1315N are optimized to execute fragment shader programs, such as those provided in the OpenGL API, which can be used to perform operations similar to pixel shader programs, such as those provided in the Direct3D API.

[0184] The graphics processor 1310 further includes one or more memory management units (MMUs) 1320A-1320B, caches 1325A-1325B, and circuit interconnects 1330A-1330B. The one or more MMU(s) 1320A-1320B provide virtual memory for the graphics processor 1310, including the vertex processor 1305 and / or fragment processor(s) 1315A-1315N. 対The graphics processor 1310 provides a physical address mapping that can reference vertices or image / texture data stored in memory in addition to vertices or image / texture data stored in one or more caches 1325A-1325B. In one embodiment, one or more MMU(s) 1320A-1320B can synchronize with other MMUs in the system, including one or more MMUs associated with one or more application processors 1205, image processor 1215, and / or video processor 1220 of FIG. 12, so that each processor 1205-1220 can participate in a shared or unified virtual memory system. One or more circuit interconnect(s) 1330A-1330B enable an embodiment of the graphics processor 1310 to interface with other IP cores in the SoC via the SoC's internal bus or via a direct connection.

[0185] As shown in Figure 13B, graphics processor 1340 includes one or more MMUs 1320A-1320B, caches 1325A-1325B, and circuit interconnects 1330A-1330B of graphics processor 1310 of Figure 13A. Graphics processor 1340 also includes one or more shader cores 1355A-1355N (e.g., 1355A, 1355B, 1355C, 1355D, 1355E, 1355F, through 1355N-1, and 1355N), providing a unified shader core architecture in which a single core or type or cores can execute all types of programmable shader code, including shader program code implementing vertex shaders, fragment shaders, and / or compute shaders. The exact number of shader cores shown may vary depending on the embodiment and implementation. Additionally, the graphics processor 1340 includes an inter-core task manager 1345 that acts as a thread dispatcher for dispatching execution threads to one or more shader cores 1355A-1355N, and a tile unit 1358 that accelerates tile processing for tile-based rendering, where scene rendering operations are subdivided in image space, for example to exploit local spatial coherence within the scene or to optimize internal cache usage.

[0186] Tensor Acceleration Logic for Machine Learning Workloads FIG. 14 is a block diagram of a data processing system 1400 according to an embodiment. The data processing system 1400 is a heterogeneous processing system having a processor 1402, a unified memory 1410, and a GPGPU 1420 including machine learning acceleration logic. The processor 1402 and the GPGPU 1420 may be any processor and GPGPU / parallel processor as described herein. The processor 1402 may execute instructions for a compiler 1415 stored in system memory 1412. The compiler 1415 executes on the processor 1402 to compile source code 1414A into compiled code 1414B. The compiled code 1414B may include instructions that may be executed by the processor 1402 and / or instructions that may be executed by the GPGPU 1420. During compilation, compiler 1415 may perform operations to insert metadata, including hints regarding the level of data parallelism present in compiled code 14B and / or hints regarding data locality associated with threads to be dispatched based on compiled code 14B. Compiler 1415 may include the information necessary to perform such operations, or the operations may be performed with the assistance of runtime library 1416. Runtime library 1416 may also assist compiler 1415 in compiling source code 1414A and may include instructions that are linked at runtime with compiled code 14B to facilitate execution of the compiled instructions on GPGPU 1420.

[0187] Unified memory 1410 represents a unified address space that can be accessed by processor 1402 and GPGPU 1420. The unified memory can include system memory 1412 and GPGPU memory 1418. GPGPU memory 1418 is memory within the address space of GPGPU 1420 and can include some or all of system memory 1412. In one embodiment, GPGPU memory 1418 can also include at least a portion of any memory that is exclusively dedicated by GPGPU 1420. In one embodiment, compiled code 1414B stored in system memory 1412 can be mapped to GPGPU memory 1418 for access by GPGPU 1420.

[0188] The GPGPU 1420 includes multiple computation blocks 1424A-1424N, which may include one or more of the various processing resources described herein. The processing resources may be or include a variety of computational resources, such as execution units, compute units, streaming multiprocessors, graphics multiprocessors, or multi-core groups. In one embodiment, the GPGPU 1420 additionally includes a tensor (e.g., matrix) accelerator 1423, which may include one or more special function computation units designed to accelerate a subset of matrix operations (e.g., dot products). The tensor accelerator 1423 may also be referred to as a tensor accelerator or tensor core. In one embodiment, the logical components within the tensor accelerator 1423 may be distributed across the processing resources of the multiple computation blocks 1424A-1424N.

[0189] The GPGPU 1420 may also include a set of resources that may be shared by the compute blocks 1424A-1424N and the tensor accelerator 1423, including, but not limited to, a set of registers 1425, a power and performance module 1426, and a cache 1427. In one embodiment, the registers 1425 include directly and indirectly accessible registers, where the indirectly accessible registers are optimized for use by the tensor accelerator 1423. The power and performance module 1426 may be configured to adjust power allocation and clock frequency for the compute blocks 1424A-1424N and to control power gating of idle components within the compute blocks 1424A-1424N. In various embodiments, the cache 1427 may include an instruction cache and / or a lower-level data cache.

[0190] The GPGPU 1420 may additionally include an L3 data cache 1430, which may be used to cache data accessed from the unified memory 1410 by the compute elements in the compute blocks 1424A-1424N and / or the tensor accelerator 1423. In one embodiment, the L3 data cache 1430 includes a shared local memory 1432 that may be shared by the tensor accelerator 1423 and the compute elements in the compute blocks 1424A-1424N.

[0191] In one embodiment, the GPGPU 1420 includes instruction handling logic such as a fetch & decode unit 1421 and a scheduler controller 1422. The fetch & decode unit 1421 includes a fetch unit and a decode unit that fetch and decode instructions for execution by one or more of the compute blocks 1424A-1424N or the tensor accelerator 1423. Instructions can be scheduled by the scheduler controller 1422 to the appropriate functional units within the tensor accelerator or compute blocks 1424A-1424N. In one embodiment, the scheduler controller 1422 is an ASIC that can be configured to perform advanced scheduling operations. In one embodiment, the scheduler controller 1422 is a low-energy-per-instruction processing core or microcontroller that can execute scheduler instructions loaded from a firmware module.

[0192] In one embodiment, some functions performed by the compute blocks 1424A-1424N can be scheduled directly or offloaded to the tensor accelerator 1423. In various embodiments, the tensor accelerator 1423 includes processing element logic configured to efficiently perform matrix computation operations, such as multiplication and addition operations and dot product operations, used by 3D graphics or compute shader programs. In one embodiment, the tensor accelerator 1423 can be configured to accelerate operations used by machine learning frameworks. In one embodiment, the tensor accelerator 1423 is an application-specific integrated circuit explicitly configured to perform a specific set of parallel matrix multiplication and / or addition operations. In one embodiment, the tensor accelerator 1423 is a field-programmable gate array (FPGA) that provides fixed-function logic that can be updated between workloads. The set of matrix operations that can be performed by the tensor accelerator 1423 can be limited in relation to the operations that can be performed by the compute blocks 1424A-1424N. However, the tensor accelerator 1423 is capable of performing these operations at a significantly higher throughput compared to the computation blocks 1424A-1424N.

[0193] 15 illustrates a matrix operation 1505 performed by an instruction pipeline 1500 according to an embodiment. The instruction pipeline 1500 may be configured to perform matrix operations 1505 such as, but not limited to, a dot product operation. The dot product of two vectors is a scalar value equal to the sum of the products of corresponding components of the vectors. The dot product may be calculated as shown in equation (1) below:

number

[0194] The dot product can be used in the convolution operation of a convolutional neural network (CNN). FIG. 15 illustrates two-dimensional (2D) convolution using a matrix operation 1505 that includes a dot product operation. While a 2D convolution is shown, an N-dimensional convolution can be performed on an N-dimensional volume using an N-dimensional filter. A receptive field tile 1502 highlights a portion of an input volume in an input volume buffer 1504. The input volume buffer can be stored in memory 1530. A dot product matrix operation 1505 is performed between the data in the receptive field tile 1502 and the convolution filter to generate data points in an output buffer 1506, which can also be stored in memory 1530. Memory 1530 can be any memory described herein, including system memory 1412, GPGPU memory 1418, or one or more cache memories 1427, 1430, as shown in FIG. 14.

[0195] The combination of data points in the output buffer 1506 represents an activation map generated by the convolution operation. Each point in the activation map is generated by sliding a receptive field tile across the input volume buffer 1504. The activation map data can be input to an activation function to determine an output activation value. In one embodiment, the convolution of the input volume buffer 1504 can be defined within the framework as a high-level matrix operation 1505. The high-level matrix operation can be performed by primitive operations such as Basic Linear Algebra Subprogram (BLAS) operations. The primitive operations can be accelerated by hardware instructions executed by the instruction pipeline 1500.

[0196] The instruction pipeline 1500 used to accelerate hardware instructions may include an instruction fetch & decode unit 1421 that may fetch and decode hardware instructions and a scheduler controller 1422 that may schedule the decoded instructions to one or more processing resources within the tensor accelerator 1423 and / or the compute blocks 1424A-1424N. In one embodiment, the hardware instructions may be scheduled to the compute blocks 1424A-1424N and offloaded to the tensor accelerator 1423. One or more hardware instructions and associated data for performing the matrix operations 1505 may be stored in memory 1530. Output of the hardware instructions may also be stored in memory 1530.

[0197] In one embodiment, the tensor accelerator 1423 can execute one or more hardware instructions to perform the matrix operations 1505 using an integrated systolic array 1508 (DP logic). The systolic array 1508 can include a combination of programmable and fixed-function hardware that can be configured to perform dot-product operations. While the functional units within the computation blocks 1424A-1424N can also be configured to perform dot-product operations, the systolic array 1508 can be configured to perform a limited subset of dot-product operations at a significantly higher throughput compared to the computation blocks 1424A-1424N.

[0198] Scalable Acceleration of Sparse Matrix Multiplication Using Systolic Arrays with Feedback Inputs Described herein is an architecture that enables scalable sparse matrix multiplication acceleration using a systolic array with feedback inputs. The architecture accelerates systolic matrix multiplication in workloads where the workload data exhibits a large number of zeros. This architecture is easily scalable and allows for increased instruction throughput while maintaining the gains provided by optimizations introduced by exploiting the sparsity present in the workload data. Benefits include improved performance, reduced power consumption, and reduced area compared to other systolic arrays. This architecture improves the performance of systolic dot-product-accumulate operations by reducing the number of calculations in highly sparse data loads. The reduction in calculations results in reduced power consumption compared to conventional architectures. This architecture also scales better than existing systolic arrays by simplifying the design of the systolic array. The simplified design allows the architecture to be widely incorporated into accelerator designs, increasing the matrix processing throughput of those accelerators.

[0199] Data used in the calculation of machine learning algorithms often exhibits a large number of zeros as input elements. In neural network applications, this is naturally caused by the topology of the implemented network and the characteristics of the modeled neurons. For example, in a fully interconnected neural network, the output of a column or layer of neurons is connected to the input of each neuron in the next layer. The network consists of many layers, each of which may be characterized by a large number of neurons. The final stage in the calculation of a neuron's output is the implementation of an activation function. This function typically outputs a positive function evaluation if the neuron's input is calculated as positive, and zero if it is negative. Due to this, many of the neuron's output results may be zero, which is fed to the next network layer in the next step of the neural network calculation.

[0200] In neural network implementations featuring multiple layers and neurons, each layer is represented by a matrix of values ​​for the neuron's input weights and a matrix of values ​​for the inputs to the neuron. Every input to a neuron is multiplied by its weight and added to the result of the neuron's other inputs. An activation function is then applied to obtain the neuron's output, which feeds the next network layer. Several techniques have been used in hardware to quickly compute the multiplication and addition of neuron inputs and weights. Of particular relevance to this invention is the use of systolic arrays for the multiplication and accumulation operations.

[0201] FIG. 16 illustrates a systolic array 1600 including multiplier and adder circuits organized in a pipelined fashion. Inputs 1612A-1612H for a first input matrix are represented by data elements contained in inputs labeled Src1 and Src1+1 through Src1+7. Inputs 1610A-1610B correspond to a second input matrix and are labeled Src2. Input 1602, which may contain an initial accumulator value, may be provided as Src0. The processing elements 1611A-1611H of the systolic array 1600 operate as a pipelined structure, with each stage executing in a clock cycle. In every cycle, every stage may receive a new Src2 input, which may be calculated with either the new Src1 input or the old input. The Src2 input operates in conjunction with eight Src1 inputs (e.g., one Src1 input per stage). The data elements of the channel of the Src2 input are broadcast across all channels of the processing elements 1611A-1611H. The processing elements then operate on the Src2 channel with all channels of the Src1 input. In the first clock cycle, the Src1 input operates on the data elements of the first channel of Src2. In the next cycle, the second Src1 (labeled as Src1+1) operates on the data elements of the second channel of Src2. This sequence is repeated through the eight stages of the pipeline. Each stage adds its operation to the output of the previous stage. Multiple Src2 inputs are processed in a pipelined manner through the pipeline stages. As successive channels of the first Src2 input are pushed through the pipeline stages, a new Src2 input can be provided at the first stage.

[0202] The output 1622 from the final stage is labeled as Dst, where d=systolic depth and e=number of data elements per channel, and the output of the channel is described by equation (2) below:

number

[0203] As shown in equation (2), each channel can contain multiple data elements on which operations are performed in parallel. In one embodiment, each channel represents a four-element data vector, although a different number of elements can be configured for each channel. In one embodiment, the number of data elements in a channel can vary based on the size of each data element. For example, a dot product can be performed using a four-element vector with an 8-bit data type per element, a two-element vector with a 16-bit data type, an eight-element vector with a 4-bit data type (e.g., INT4), or a 16-element vector with a 2-bit data type (e.g., INT2). The number of channels can be automatically adjusted depending on the data types of Src1 and Src2. The instruction can also specify the required systolic depth to be used for the instruction.

[0204] In one embodiment, processing elements 1611A-1611H may read inputs 1610A-1610H, 1612A-1612H directly from a general-purpose register file. In one embodiment, systolic array 1600 includes logic to read inputs 1610A-1610H, 1612A-1612H from a general-purpose register file and store the input data in registers, buffers, or memory internal to the systolic array. The internal logic can then provide the input data elements to processing elements 1611A-1611H for processing. Output 1622 can be written to internal registers or memory of systolic array 1600 and / or written directly to the general-purpose register file.

[0205] In one embodiment, if an element input to a multiplier / adder in a stage is determined to be zero, the multiply / add can be bypassed and the previous input is simply propagated. If the input matrix is ​​sparse (e.g., contains many zeros), the number of operations to be performed is reduced. Some implementations can bypass multiply / accumulate stages to avoid the power consumption of performing multiply / add operations. However, simply bypassing an operation results in idle stages in the systolic chain. For example, if the elements of the Src2 input are all zero, bypassing the stage associated with that element would save power. However, no operation is performed that cycle. Therefore, the throughput of the operation remains unchanged. Other implementations can rearrange the inputs in such a way that entire sections of the array can be bypassed. Rearranging the inputs increases the scale-up cost due to the addition of extra hardware to rearrange the inputs. The architecture must consider the case of inputs without sparsity, which requires using the full depth of the pipeline. Therefore, the architecture should be designed so that all stages are always available.

[0206] Described herein, in various embodiments, is an architecture with optimizations for handling sparse inputs while avoiding the above problems. The embodiments provide a modular systolic array that can be easily scalable to meet the needs of various products and allows computation of only non-zero elements without additional hardware or idle clock cycles. These concepts are gradually explained in the following sections of this document.

[0207] Matrix multiplication accelerator with feedback input Systolic array 1600 is an 8-deep multiply / add array with accumulation and may be referred to as a Dot Product Accumulate Systolic (DPAS) array. Such architectural depth may be beneficial for some products, projects, and / or use cases. Other products, projects, and / or use cases may not require such a deep array and may benefit from a smaller area and / or lower power consumption, albeit with lower throughput. Alternatively, other products, projects, and / or use cases may benefit from the higher DPAS throughput achievable by using a larger pipeline depth. To reduce the amount of hardware used in the systolic array while maintaining the same programming model used for deeper arrays, a feedback input is added to the reduced-depth version of the systolic array shown in Figure 16.

[0208] Figures 17A-17B show an example of using a four-deep systolic array 1700 to compute an equivalent array of eight systolic stages. Figure 17A shows an array that receives the Src0 input from an external source and processes the first four stages with the Src1 and Src2 inputs. The output of this array is fed back to the second stage shown in Figure 17B. Figure 17B shows the next four stages being computed using feedback data that includes already processed values ​​and the Src1 and Src2 inputs.

[0209] As shown in FIG. 17A, systolic array 1700 can accept input 1602 as Src0 input, which is read by data selector 1704 (1702). Data selector 1704 selects between input 1602 and feedback input 1706. Processing elements 1611A-1611D can process inputs 1610A-1610D and 1612A-1612D in a manner similar to systolic array 1600. If four stages are sufficient to complete the operation, processing element 1611D can write output 1622 to a Dst register or memory specified by data selector 1724 (1722). If more stages are required, data selector 1724 can write feedback output 1726, which is provided as feedback input 1706 to processing element 1611A.

[0210] As shown in FIG. 17B, in one embodiment, feedback input 1706 can be further processed by processing elements 1611A-1611D. Feedback input 1706 includes values ​​that have already been processed. In one embodiment, feedback input 1706 can also include inputs 1610E-1610H and inputs 1612E-1612H, which can be pre-fetched during processing of the first four stages. Data selector 1704 selects feedback input 1706 for input by processing element 1611A. Processing elements 1611A-1611D can then process inputs 1610E-1610H and 1612E-1612H. Data selector 1724 can then write the result of the eighth stage as output 1622 to the designated Dst register.

[0211] The time diagrams of the arrays of Figures 16 and 17A-17B are shown in Figures 18A-18B.

[0212] Figures 18A-18B show two timing diagrams 1800, 1850. Figure 18A shows timing diagram 1800 corresponding to the 8-deep systolic array 1600 shown in Figure 16. Figure 18B shows timing diagram 1850 corresponding to the systolic array with feedback inputs shown in Figures 17A-17B. Multiple clock cycles are shown.

[0213] As shown in FIG. 18A, timing diagram 1800 illustrates a read cycle 1802, which corresponds to the read logic of systolic array 1600 reading inputs provided to the array. Systolic cycle 1804 corresponds to how these inputs are processed at each stage of the array. Write cycle 1806 corresponds to the output logic writing the output to a specified destination location. The input read in cycle 0 of read cycle 1802 is processed by the systolic array in cycles 0-7 of systolic cycle 1804. The input read in cycle 1 of read cycle 1802 is processed in cycles 1-8 of systolic cycle 1804. The processing beginning in cycle 0 of systolic cycle 1804 is output in cycle "N" of write cycle 1806. In one embodiment, the value of N may be related to the depth of the systolic array. An operation beginning in cycle 1 of the systolic array 1804 is processed in cycles 1-8 and output in cycle N+1 of the write array 1806. Due to the pipelined nature of the array, the calculations that result in the output in cycles N and N+1 are performed in parallel by the various pipelined stages (stage 1-stage 8) of the systolic array.

[0214] The read cycle 1802 precedes the systolic cycle 1804 by one or more cycles. For example, cycle 0 of the read cycle 1802 can occur before cycle 0 of the systolic cycle 1804. In one embodiment, cycle 1 of the read cycle 1802 can occur simultaneously with cycle 0 of the systolic cycle 1804. During cycle 0 of the systolic cycle 1804, stage 1 of the systolic array calculates Src0 + Src1 × Src2.0 based on the input read in cycle 0 of the read cycle. Each of elements 0-7 of Src1 is multiplied in parallel with element 0 of Src2 and added to the corresponding elements 0-7 of Src0. The result from stage 1 is passed to stage 2. In cycle 1 of systolic cycle 1804, stage 2 accumulates the result calculated by stage 1 in cycle 0 with the result of [Src1+1] * Src2.1, where element 1 of Src2 is multiplied by each element of "Src1+1." Src1 remains in stage 1, and in cycle 1, stage 1, along with its next Src0 and Src2 inputs, calculates [Src0+1] + Src1 * [Src2+1.0]. In the first stage, each of elements 0-7 of Src1 is multiplied in parallel with element 0 of [Src2+1] and added to the corresponding element of [Src0+1]. Processing continues in this pattern for each stage and each cycle, with results output from stage 8 beginning with cycle N of write cycle 1806.

[0215] As shown in FIG. 18B, timing diagram 1850 illustrates that systolic array 1700 of FIGS. 17A-17B processes the first group of inputs read in cycle 0 of read cycle 1852, similar to the first four stages of the eight-deep pipeline of systolic array 1600 of FIG. 16, starting with cycle 0 of systolic cycle 1854. It will be appreciated that in FIG. 18B, read cycle 1852 does not necessarily align with the corresponding cycle of systolic cycle 1854. The read logic of systolic array 1700 may read the first group of inputs in the same manner as systolic array 1600. The first four inputs read in cycles 0 through 3 of read cycle 1852 may be processed in a pipelined manner by systolic array 1700. The output is generated in the cycle labeled "N" in write cycle 1856. Therefore, the latency to the first output in cycle N of systolic array 1700 is the same as the latency through systolic array 1700. However, the throughput of systolic array 1700 is reduced because there is a delay between the outputs of Dst3 and Dst4, which are written in cycles N+8 to N+11 of the write cycle due to feedback.

[0216] For systolic array 1700, feedback begins in cycle 4 of systolic cycle 1854. Feedback occurs until cycle 7. Once feedback begins in cycle 4 of systolic cycle 1854, only the Src1 input is read by the processing element in cycles 4, 5, 6, and 7 of systolic cycle 1854, as represented by the dashed input. The next group of Src0 and Src2 inputs is read by the processing element beginning in cycle 8 of systolic cycle 1854. The read logic can delay reading the Src0 and Src2 inputs until these inputs are needed, or they can be read in conjunction with the Src1 input, as shown. Once the inputs are read, they are buffered by the systolic array and can be reused.

[0217] The advantages of the matrix multiplication accelerator with feedback (systolic array 1700) over systolic array 1600 are summarized as follows: Systolic array 1700 can compute a similar pipeline depth as systolic array 1600 using less hardware. Systolic array 1700 allows the use of the same instructions as systolic array 1600, allowing workloads developed for systolic array 1600 to be reused with systolic array 1700. Systolic array 1700 consumes less power by utilizing fewer pipeline stages. Systolic array 1700 can operate with reduced bandwidth for reads and writes to systolic array 1600. While systolic array 1700 can be implemented with any number of stages, it may be best to use a multiple of two to enable the features of the embodiments described below. Although the architecture of systolic array 1700 has reduced throughput, the same throughput as systolic array 1600 can be achieved by implementing multiple instances of systolic array 1700 in parallel.

[0218] A scalable matrix multiplication accelerator with feedback inputs. The second embodiment allows for increased throughput through the use of concurrent instruction execution using parallel units. Several instances or paths of the multiplication accelerator are executed in parallel. These instances can share Src1 or can have independent Src1 inputs. Each path has its own Src2 and Src0 inputs. These instances have their own Src2 and Src0 inputs. A version showing two paths with a depth of four stages is shown in Figure 19. Alternatively, a version using four paths with a depth of two stages is shown in Figure 20.

[0219] 19 shows a two-path matrix multiplication accelerator 1900, with each path having a depth of four stages. The two-path matrix multiplication accelerator 1900 includes input logic 1902A-1902B for the Src0 input, input buffers 1911A-1911B for storing data elements received from input logic 1910A-1910B, and input buffers 1913A-1913B for storing data elements received from shared input logic 1912 for Src1. Each stage includes a pair of processing elements capable of operating in parallel. Stage 1 includes processing elements 1931A-1931B, stage 2 includes processing elements 1932A-1932B, stage 3 includes processing elements 1933A-1933B, and stage 4 includes processing elements 1934A-1934B. The hardware logic of each of processing elements 1931A-1931B, 1932A-1932B, 1933A-1933B, and 1934A-1934B may be the same as or similar to the hardware logic of the processing elements (e.g., processing elements 1611A-1611D) of systolic array 1600 or systolic array 1700, and may be fabricated using the same or more advanced process technology. Additionally, the processing elements of two-path matrix multiplication accelerator 1900 may operate at a higher frequency compared to implementations of systolic array 1600. The processing elements may be fabricated using more advanced process technology.

[0220] Feedback may be implemented using data selectors the same as or similar to data selectors 1704, 1724. Depending on the configuration of the read logic, input data may be prefetched into an input buffer in advance or may be read from a register or cache within two-way matrix multiplication accelerator 1900 one or more cycles before entering processing elements 1931A-1931B. Stage 4 processing elements 1934A-1934B may feed back to corresponding processing elements 1931A-1931B in stage 1. Dynamic logic depth may be enabled in multiples of four. After a set number of logic stages, the results may be written to a specified destination by output logic 1922A-1922B.

[0221] FIG. 20 shows a four-path matrix multiplication accelerator 2000, with each path two stages deep. The four-path matrix multiplication accelerator 2000 contains the same number of processing elements as the two-path matrix multiplication accelerator 1900, but with twice as many paths, each path being half as deep. The four-path matrix multiplication accelerator 2000 includes input logic 2002A-2002D for Src0, input buffers 2011A-2011D for storing input elements read by input logic 2010A-2010D for Src2, and input buffers 2013A-2013D for storing input elements read by shared input logic 2012 for Src1. Processing elements 2031A-2031B enable parallel processing of stage 1. Processing elements 2032A-2032B enable parallel processing of stage 2. Stage 2 of each path may feed back to stage 1 or write the result to a destination specified by output logic 2022A-2022D. Processing elements 2031A-2031B, 2032A-2032B may include hardware logic similar to that of processing elements 1931A-1931B, 1932A-1932B, 1933A-1933B, 1934A-1934B, and similar hardware logic may be used to implement loopback functions.

[0222] The advantages of the two-path matrix multiplication accelerator 1900 or the four-path matrix multiplication accelerator 2000 include scalability, software compatibility, and throughput. The modular architecture of these accelerators allows for more efficient scaling compared to eight-deep systolic arrays. Different configurations of the matrix multiplication accelerator can be tailored to different product needs or use cases without redesign. Furthermore, the same software model is used, independent of the hardware implementation. An algorithm designed for instructions intended to be executed by an eight-stage systolic pipeline can be used in an implementation using a four-stage matrix multiplication accelerator. The hardware will use feedback to simulate an eight-stage pipeline in a manner transparent to software. Designs requiring high DPAS instruction throughput can use multiple paths. Implementations with a larger number of paths can be combined with higher-bandwidth input and output logic. In one embodiment, the two-way matrix multiplication accelerator 1900 and the four-way matrix multiplication accelerator 2000 are configured to bypass inputs with block sparsity with greater efficiency and / or finer granularity than is possible with an eight-deep systolic array.

[0223] Sparse Multiplication in a Scalable Matrix Multiplication Accelerator The third embodiment facilitates improved instruction throughput when processing data with irregular sparsity: elements of the Src1 and Src2 inputs can be individually selected by input multiplexer logic, and processing can be performed using only non-zero values.

[0224] FIG. 21 illustrates a scalable sparse matrix multiplication accelerator 2100 using a systolic array with feedback inputs. The scalable sparse matrix multiplication accelerator 2100 may include processing elements 2031A-2031D similar to those in the four-path matrix multiplication accelerator 2000 or any other processing elements described herein. Processing elements 2031A-2031B at the beginning of each path include input logic for Src0. Each stage of each path in the scalable sparse matrix multiplication accelerator 2100 can receive any element of Src1, either independently or jointly, via input selectors 2112A-2112D. Each stage of each path can receive any element of Src2. The independent Src2 input is provided by a separate input element selector (e.g., Src2A via input selector 2110A and input selector 2111A, and Src2B via input selector 2110B and input selector 2111B). Separate Src2 inputs allow each path to compute a different instruction. There is separate output logic 2122A-2122B for each path to allow output for different instructions.

[0225] 22 shows an Src2 input 2200 containing sparse data. In the illustrated example, sparse Src2 inputs 2200A-2200B (Src2A input 2200A [A0, A1, 0, A3, A4, A5, A6, 0], Src2B input 2200B [B0, 0, B2, B3, 0, 0, 0, 0]) can be processed by each path of the variant scalable sparse matrix multiplication accelerator 2100 using a common Src1 input. Each path of the scalable sparse matrix multiplication accelerator 2100 can receive a separate, unique Src0 input.

[0226] The first step in the computation process is to read the first Src2 element and rearrange the elements into groups of N elements, where N is the depth of the path through which the element is processed. Other implementations with a different number of paths may have different group sizes. For example, an accelerator based on the four-path matrix multiplication accelerator 2000 would use groups of four data elements. When possible, only non-zero data elements are selected. For example, non-zero values ​​in Src2A 2200A are rearranged into three groups: [A0,A1], [A3,A4], and [A5,A6]. Non-zero values ​​in Src2B 2200B are rearranged into two groups: [B0,B2] and [B3,0], with zero padding used to complete the second group. This rearrangement is used to allow the first element of each group to be fed to the first stage of the path and the second element of each group to be fed to the second stage of the path. The scalable sparse matrix multiplication accelerator 2100 does not require grouping of elements, but grouping of elements reduces the number of elements that may have to be fed to a stage.

[0227] In the second stage of the computation process, the groups are given paths. Instead of performing four passes to compute an instruction with a depth of 8 (e.g., eight feedback passes, each with two stages), only three feedback passes are needed because two elements are zero and do not require processing. The nature of the feedback allows that pass to be bypassed, and the accumulator value is sent directly to the output without consuming a computation stage. To maintain correct function computation, the correct Src1 element is input to the stage for a given Src2 element being computed. Thus, when processing the second group ([A3,A4]), the first stage reads Src1-3, and the second stage reads Src1-4. When processing Src2B2200B, only two groups ([B0,B2], [B3,0]) are computed. The two groups can be computed using two feedback passes instead of four. In the first pass, Src1-0 and Src1-2 are input to the first and second stages. In the second pass, Src1-3 and any Src1 element are input to the first and second stages, respectively.

[0228] In the third embodiment, the depth of the path constrains the number of zeros that can be reduced. A matrix multiplication accelerator with two stages in the path (scalable sparse matrix multiplication accelerator 2100) for an eight-element input can only perform reductions of 8, 6, 4, and 2 zeros. A matrix multiplication accelerator with four stages in the path can only perform reductions of 8 and 4 zeros. To allow for higher resolution of the sparse reduction, the fourth embodiment adds an output at each stage in the path, allowing each stage to receive a Src0 input, as shown in FIG. 23.

[0229] 23 shows a scalable sparse matrix multiplication accelerator 2300 that uses a systolic array with feedback inputs and outputs at each stage. The scalable sparse matrix multiplication accelerator 2300 includes similar hardware logic as the scalable sparse matrix multiplication accelerator 2100, with additional input and output logic that allows a Src0 element to be provided to each stage of each path and to provide separate outputs for each stage of each path. input selectors 2110A and 2111A for selecting the Src2A element for the first path; In addition to input selectors 2110B and 2111B that select the Src2B input for the second path, an input splitter 2303A-2303B is added for each path of the Src0 input. Each input splitter 2302A-2302B includes a demultiplexer or similar hardware logic to allow the Src0 input element read by input logic 2302A-2302B to be sent to each stage. Input selectors 2112A-2112D are also included to allow the Src1 input to be selected by each stage of each path. In addition to output logic 2122A-2122B from the second stage (processing elements 2331C-2331D) of each path, additional output logic 2322A-2322B is provided to allow output from the first stage (2331A-2331B) of each path. Processing elements 2331A-2331C may be similar to other processing elements described herein.

[0230] In operation, the scalable sparse matrix multiplication accelerator 2300 can be configured to accept groups of only one element. Two groups ([B0, B2], [B3, O]) are created for the non-zero elements of Src2 in the third embodiment (e.g., the scalable sparse matrix multiplication accelerator 2100), with the second group including zero padding. The optimization shown in FIG. 23 allows the groups to be formed as [B0, B2], [B3]. B0 and B2 would be assigned to the first and second stages of a path (e.g., either the first set including processing element 2331A and processing element 2331C, or the second set including processing element 2331B and processing element 2331D). After feedback, B3 is assigned to the first stage of that path. The first stage of the path can provide an output (e.g., by either output logic 2322A or 2322B) without consuming the second stage of the path (either processing element 2331C or processing element 2331D). Furthermore, the next Src2 input accepted for that path can start from the second stage, so two-element groups are assigned to the second and first stages, respectively. Src0 can be assigned to the second stage of the path (e.g., by either output logic 2322A or 2322B) for processing the new Src2 input.

[0231] In addition to the hardware logic of the scalable sparse matrix multiplication accelerator 2100 shown in FIG. 21 and the scalable sparse matrix multiplication accelerator 2300 shown in FIG. 23, some embodiments additionally include input and output hardware memory buffers. The input memory buffers can be used to store and prepare groups of Src0 and Src2 inputs, reducing the need for high-bandwidth input logic. The output buffers allow Dst outputs generated in the same cycle to be reliably written to memory at a slower rate, reducing the need for high-bandwidth output logic.

[0232] Additionally, some embodiments include a bypass for inputs with all elements equal to zero. The bypass allows the output logic to write Src0 directly without passing through the systolic array. This bypass is used in concert with data dependency measures to prevent the risk of read-after-write (RAW) during an instruction that could compromise data integrity.

[0233] FIG. 24 illustrates a method 2400 according to an embodiment in which hardware logic in a functional unit can execute an instruction to perform a systolic dot product with accumulation. Method 2400 can be performed by hardware and / or firmware logic of a scalable sparse matrix multiplication accelerator, as described herein. The hardware and / or firmware logic can receive non-zero source values ​​and a computation depth for an instruction to be executed by a matrix operation accelerator of a GPGPU (2402). The non-zero source values ​​can be non-zero values ​​grouped according to a pipeline depth for a path in the scalable sparse matrix multiplication accelerator. The computation depth can specify the number of systolic layers to use to compute the dot product for the instruction. The logic also receives an accumulator value and stores an initial value in the accumulator (2404). The accumulator value can be a zero value, an initial accumulator value, or a result from a previous pipeline stage. For a specified layer of computation, the logic may evaluate a write enable mask to determine a set of enabled parallel processing channels (2406). The write enable mask may be used to disable computation of specific channels. The write enable mask may be configured based on a decision mask provided with the instruction to be executed.

[0234] For each enabled parallel processing channel, the logic can generate a set of products based on element-wise multiplication of the source input elements (2408). For example, for a four-element dot product, four elements of two sources are multiplied to generate a set of products. At each layer of the systolic pipeline, the same Src2 element value is multiplied by multiple different Src1 values. The logic can then calculate the sum of the set of products and add the sum to the value in the accumulator (2410).

[0235] If method 2400 is executed by a processing element in the last computational layer (2411), the processing element may output the calculated sum to a designated output register (2414). Otherwise, the processing element may output its accumulator value to the next layer (2412). The next layer may be the next physical layer or the next virtual layer. Outputting to the next virtual layer includes providing a feedback value to a processing element in the first stage of the processing pipeline.

[0236] In one embodiment, the method 2400 of FIG. 24 may be performed by hardware logic constructed according to the pseudocode shown below. 4-element systolic dot product with accumulation

number

[0237] In the pseudocode above, Src0, Src1, and Src2 are registers that store the operand data. The systolic depth is specified by sdepth. The execution size corresponds to exec_size and specifies the number of parallel processing channels. The destination is specified by the Dst register. In the pseudocode, the registers identified refer to the regnum and subregnum fields. The regnum field provides the register number of the operand. The subregnum field provides the subregister number of the operand. The subregnum field, together with the corresponding RegNum field, provides a byte-aligned address relative to the origin of the register space. For some instructions, this field provides bits [4:0] of the byte address, while the RegNum field provides bits [12:5].

[0238] FIG. 25 illustrates a method 2500 for performing a matrix multiplication operation using a sparse Src2 input matrix. Method 2500 may be performed by the hardware and / or firmware logic of a scalable sparse matrix multiplication accelerator, as described herein. Method 2500 demonstrates operations using sparse data, such as the sparse Src2 inputs 2200A-2200B of FIG. 22. Method 2500 may be implemented using the scalable sparse matrix multiplication accelerator 2100 of FIG. 21 and / or the scalable sparse matrix multiplication accelerator 2300 of FIG. 23.

[0239] The method 2500 includes hardware and / or firmware logic for loading a plurality of data elements of a first matrix and a second matrix into a memory of a matrix multiplication accelerator (2502). The logic can then detect non-zero values ​​within the plurality of data elements of the second matrix (2504). The detection can be performed using vector comparison logic within the matrix multiplication accelerator. The logic can then group the detected non-zero values ​​within the plurality of data elements of the second matrix into groups containing one or more data elements (2506). The logic can then provide the data elements of the groups to corresponding stages of a processing pipeline of the matrix multiplication accelerator (2508). For a path having a two-stage pipeline, the group will include two Src2 elements. The first element of the group will be provided to the first stage, and the second element of the group will be provided to the second stage. For the scalable sparse matrix multiplication accelerator 2100, zero padding is used to pad the groups, if necessary. In the Scalable Sparse Matrix Multiplication Accelerator 2300, zero padding is not required.

[0240] The logic then provides (2510) the data elements of the first matrix to corresponding stages of the processing pipeline. The provided data elements are Src1 elements corresponding to the active channels calculated for the pipeline stage. If the elements are operated on using the Src2 zero-padding value, any Src1 elements may be provided. The processing elements in each active stage of the processing pipeline perform multiplication and accumulation operations (2512). In some situations, for example, when the scalable sparse matrix multiplication accelerator 2300 is processing a single element group, not all stages of the pipeline are active for the instruction. If a stage is not active for the instruction, it can be used to perform further operations for a different instruction. The logic then outputs or feeds back accumulated values ​​from each active stage of the processing pipeline (2514). The output can be written to an output register or memory location when the final stage of processing for the instruction is completed. Alternatively, the accumulated values ​​can be sent to the next pipeline stage. Outputting to the next pipeline stage may include writing the feedback output to the first physical stage of the processing pipeline.

[0241] Additional Computing Device Examples FIG. 26 is a block diagram of a computing device 2600 including a graphics processor 2604 according to one embodiment. Variants of the computing device 2600 may be, or may be included within, a communications device, such as a set-top box (e.g., an Internet-based cable television set-top box), a global positioning system (GPS)-based device, etc. The computing device 2600 may be, or may be included within, a mobile computing device, such as a cellular phone, a smartphone, a personal digital assistant (PDA), a tablet computer, a laptop computer, an e-reader, a smart television, a television platform, a wearable device (e.g., eyeglasses, a watch, a bracelet, a smart card, jewelry, clothing, etc.), a media player, etc. For example, in one embodiment, the computing device 2600 includes an integrated circuit ("IC"), such as a system-on-chip ("SoC" or "SOC"), integrating various hardware and / or software components of the computing device 2600 onto a single chip. Computing device 2600 may be a computing device such as data processing system 100 like that in FIG.

[0242] Computing device 2600 includes a graphics processor 2604. Graphics processor 2604 represents any graphics processor described herein. In one embodiment, graphics processor 2604 includes a cache 2614, which may be a single cache or may be divided into multiple segments of cache memory, including, but not limited to, any number of L1, L2, L3, or L4 caches, rendering caches, depth caches, sampler caches, and / or shader unit caches. In one embodiment, cache 2614 may be a last-level cache shared with application processor 2606.

[0243] In one embodiment, the graphics processor 2604 includes a graphics microcontroller that implements control and scheduling logic for the graphics processor. The control and scheduling logic may be firmware executed by the graphics microcontroller 2615. The firmware may be loaded at power-on by the graphics driver logic 2622. The firmware may also be programmed into electronically erasable programmable read-only memory or loaded from a flash memory device within the graphics microcontroller 2615. The firmware may enable the GPU OS 2616, which includes device management / driver logic 2617, 2618, and a scheduler 2619. The GPU OS 2616 may also include a graphics memory manager 2620, which may supplement or replace the graphics memory manager 2621 in the graphics driver logic 2622.

[0244] The graphics processor 2604 also includes a GPGPU engine 2644, which includes one or more graphics engines, graphics processor cores, and other graphics execution resources described herein. Such graphics execution resources may be embodied in forms including, but not limited to, execution units, shader engines, fragment processors, vertex processors, streaming multiprocessors, graphics processor clusters, or any collection of computational resources suitable for processing graphics or image resources or for performing general-purpose computation within a heterogeneous processor. The processing resources of the GPGPU engine 2644 may be contained within multiple tiles of hardware logic coupled to the board, as shown in Figures 11B-11D. The GPGPU engine 2644 may include GPU tiles 2645, which include graphics processing and execution resources, caches, samplers, etc. The GPU tiles 2645 may also include local volatile memory or may be coupled to one or more memory tiles, as shown, for example, in Figures 3B-3C.

[0245] The GPGPU engine 2644 may also include one or more specialized tiles 2646, including, for example, non-volatile memory tiles 2656, network processor tiles 2657, and / or general-purpose computation tiles 2658. The GPGPU engine 2644 also includes a matrix multiplication accelerator 2660. The general-purpose computation tiles 2658 may also include logic for accelerating matrix multiplication operations. The non-volatile memory tiles 2656 may include non-volatile memory cells and controller logic. The controller logic of the non-volatile memory tiles 2656 may be managed by one of the device management / driver logic 2617, 2618. The network processor tiles 2657 may include network processing resources coupled to physical interfaces in the input / output (I / O) sources 2610 of the computing device 2600. The network processor tiles 2657 may be managed by one or more of the device management / driver logic 2617, 2618.

[0246] The matrix multiplication accelerator 2660 is a modular, scalable, sparse matrix multiplication accelerator as described herein. The matrix multiplication accelerator 2660 may include multiple processing paths, each including multiple pipeline stages. Each processing path may execute individual instructions. In various embodiments, the matrix multiplication accelerator 2660 may have the architectural features of any one or more of the matrix multiplication accelerators described herein. For example, in one embodiment, the matrix multiplication accelerator 2660 is a four-deep systolic array 1700 with a feedback loop that may be configured to operate with a multiple of four logical stages (e.g., 4, 8, 12, 16, etc.). In one embodiment, the matrix multiplication accelerator 2660 includes one or more instances of the two-path matrix multiplication accelerator 1900 with a four-stage pipeline or the four-path matrix multiplication accelerator 2000 with a two-stage pipeline. In one embodiment, matrix multiplication accelerator 2660 includes processing elements configured as scalable sparse matrix multiplication accelerator 2100 or scalable sparse matrix multiplication accelerator 2300. Matrix multiplication accelerator 2660 can be configured to operate only on non-zero values ​​of at least the Src2 input, and can also bypass operations when zero values ​​are present in the Src1 input. When block sparsity exists, operations on entire sub-matrices can be bypassed. Matrix multiplication accelerator 2660 can also include any logic based on any combination of these embodiments.

[0247] As shown, in one embodiment, in addition to the graphics processor 2604, the computing device 2600 may further include any number and type of hardware and / or software components, including, but not limited to, an application processor 2606, memory 2608, and input / output (I / O) sources 2610. The application processor 2606 may interact with a hardware graphics pipeline, as shown in connection with FIG. 3A, to share graphics pipeline functions. Processed data is stored in buffers within the hardware graphics pipeline, and state information is stored in memory 2608. The resulting data may be transferred to a display controller for output by a display device, such as the display device 318 of FIG. 3A. The display device may be of various types, such as a cathode ray tube (CRT), thin film transistor (TFT), liquid crystal display (LCD), organic light emitting diode (OLED) array, etc., and may be configured to display information to a user via a graphical user interface.

[0248] The application processor 2606 may include one or more processors, such as processor 102 of FIG. 1, and may be a central processing unit (CPU) used at least in part to execute the operating system 2602 of the computing device 2600. The OS 2602 may serve as an interface between the hardware and / or physical resources of the computing device 2600 and one or more users. The OS 2602 may include driver logic for various hardware devices within the computing device 2600. The driver logic may include graphics driver logic 2622, which may include the user-mode graphics driver 1026 and / or the kernel-mode graphics driver 1029 of FIG. 10. The graphics driver logic may include a graphics memory manager 2621 to manage the virtual memory address space of the graphics processor 2604.

[0249] In some embodiments, the graphics processor 2604 may be part of the application processor 2606 (e.g., part of a physical CPU package), in which case at least a portion of the memory 2608 may be shared by the application processor 2606 and the graphics processor 2604, although at least a portion of the memory 2608 may be exclusive to the graphics processor 2604, or the graphics processor 2604 may have separate storage memory. The memory 2608 may include a pre-allocated area of ​​a buffer (e.g., a frame buffer); however, it should be understood by those skilled in the art that embodiments are not so limited and that any memory accessible to the underlying graphics pipeline may be used. The memory 2608 may include various forms of random access memory (RAM) (e.g., SDRAM, SRAM, etc.), including applications that utilize the graphics processor 2604 to render desktop or 3D graphics scenes. A memory controller hub, such as memory controller 116 in FIG. 1, may access data in memory 2608 and transfer it to graphics processor 2604 for graphics pipeline processing. Memory 2608 may be made available to other components within computing device 2600. For example, any data (e.g., input graphics data) received from various I / O sources 2610 of computing device 2600 may be temporarily queued in memory 2608 before being operated on by one or more processors (e.g., application processor 2606) in implementing a software program or application.Similarly, data that a software program determines should be transmitted from computing device 2600 to an external entity via one of the computing system interfaces or stored in an internal storage element is often temporarily queued in memory 2608 before it is transmitted or stored.

[0250] I / O sources may include devices such as a touchscreen, touch panel, touchpad, virtual or regular keyboard, virtual or regular mouse, ports, connectors, network devices, etc., and may be attached via platform controller hub 130 as referenced in FIG. 1 . Additionally, I / O sources 2610 may include one or more I / O devices implemented for transferring data to and / or from computing device 2600 (e.g., network adapters); or for large-scale non-volatile storage (e.g., SSD / HDD) within computing device 2600. User input devices, including alphanumeric and other keys, may be used to communicate information and command selections to graphics processor 2604. Another type of user input device is a mouse, trackball, touchscreen, touchpad, or cursor control device, such as cursor direction keys, which communicates directional information and command selections to the GPU and controls cursor movement on a display device. The camera and microphone array of the computing device 2600 may be used to observe gestures, record audio and video, and send and receive visual and audio commands.

[0251] The I / O sources 2610 may include one or more network interfaces. The network interfaces may include associated network processing logic and / or may be coupled to the network processor tiles 2657. The one or more network interfaces may provide access to a LAN, a wide area network (WAN), a metropolitan area network (MAN), a personal area network (PAN), Bluetooth, a cloud network, a cellular or mobile network (e.g., third generation (3G), fourth generation (4G), fifth generation (5G), etc.), an intranet, the Internet, etc. The network interfaces may include, for example, a wireless network interface having one or more antennas. The network interfaces may also include wired network interfaces, for example, for communicating with remote devices via a network cable, which may be, for example, an Ethernet cable, a coaxial cable, an optical fiber cable, a serial cable, or a parallel cable.

[0252] The network interface may provide access to a LAN, for example, by conforming to the IEEE 802.11 standard, and / or the wireless network interface may provide access to a personal area network, for example, by conforming to the Bluetooth® standard. Other wireless network interfaces and / or protocols, including earlier and later versions of the standard, may also be supported. In addition to or instead of communicating via a wireless LAN standard, the network interface may provide wireless communication using, for example, a time division multiple access (TDM) protocol, a Global System for Mobile Communications (GSM) protocol, a code division multiple access (CDMA) protocol, and / or any other type of wireless communication protocol.

[0253] It should be understood that a lesser or more equipped system than the above example may be preferred in a particular implementation, and thus the configuration of computing device 1900 may vary from implementation to implementation depending on many factors, such as price constraints, performance requirements, technological improvements, or other circumstances. Examples include (but are not limited to) a mobile device, a personal digital assistant, a mobile computing device, a smart phone, a cellular phone, a handset, a one-way pager, a two-way pager, a messaging device, a computer, a personal computer (PC), a desktop computer, a laptop computer, a notebook computer, a handheld computer, a tablet computer, a server, a server array or server farm, a web server, a network server, an Internet server, a workstation, a minicomputer, a mainframe computer, a supercomputer, a network appliance, a web appliance, a distributed computing system, a multiprocessor system, a processor-based system, a consumer electronics appliance, a programmable consumer electronics device, a television, a digital television, a set-top box, a wireless access point, a base station, a subscriber station, a mobile subscriber center, a radio network controller, a router, a hub, a gateway, a bridge, a switch, a machine, or combinations thereof.

[0254] Described herein is an accelerator device that includes a host interface, a fabric interconnect coupled to the host interface, and one or more hardware tiles coupled to the fabric interconnect, the one or more hardware tiles including sparse matrix multiplication acceleration hardware that includes a systolic array with a feedback input.

[0255] One embodiment provides a parallel processor comprising: a decode unit for decoding instructions into decoded instructions, the decoded instructions being instructions for performing parallel dot-product operations; and a pipelined systolic dot-product unit. The pipelined systolic dot-product unit is configured to execute the decoded instructions through multiple pipeline stages of a systolic processing pipeline. During execution of the decoded instructions, the dot products calculated in the first pipeline stage are configured to be selectively written via output hardware to a location selected from one of an output memory and a second pipeline stage, and the dot products calculated in the third pipeline stage are configured to be selectively written via output hardware to a location selected from one of the output memory and the first pipeline stage. In a further embodiment, the decoded instructions are associated with a first source operand and a second source operand, the first source operand being a reference to a memory storing a plurality of data elements of a first matrix, and the second source operand being a reference to a memory storing a plurality of data elements of a second matrix.

[0256] One embodiment provides an accelerator device comprising a host interface, a fabric interconnect coupled to the host interface, and one or more hardware tiles coupled to the fabric interconnect. The one or more hardware tiles include sparse matrix multiplication acceleration hardware including a modular systolic processing array having a feedback input. The modular systolic processing array includes one or more processing array modules having a first number of pipeline paths, the first number of pipeline paths having a second number of pipeline stages. The first pipeline stage can be configured to receive a feedback output from a final pipeline stage.

[0257] One embodiment provides a method for performing a dot product operation on a set of input matrices by a hardware matrix multiplication accelerator having a multi-stage processing pipeline. The method includes the steps of: reading a plurality of data elements of a first matrix into a memory of a hardware matrix multiplication accelerator using a first source operand; reading a plurality of data elements of a second matrix into a memory of the hardware matrix multiplication accelerator using a second source operand; detecting non-zero values ​​among the plurality of data elements of the second matrix; grouping the non-zero values ​​among the plurality of data elements of the second matrix into groups containing one or more data elements, the number of data elements in the group corresponding to the number of stages in a multi-stage processing pipeline of the hardware matrix multiplication accelerator; providing the data elements of the group to a corresponding stage of the processing pipeline by broadcasting the data elements to a plurality of channels of processing elements of the corresponding stage; multiplying the data elements of the provided group by the plurality of data elements of the first matrix to generate a set of products; summing the set of products and accumulating the sum of the set of products into an accumulator value; and writing the accumulator value to a next stage of the processing pipeline. In a further embodiment, writing the accumulator value to a next stage of the processing pipeline includes writing a pipeline feedback value to a first stage of the processing pipeline.Furthermore, detecting a non-zero value within the plurality of data elements of the second matrix may include detecting a non-zero value within a memory of a hardware matrix multiplication accelerator.

[0258] Those skilled in the art will recognize from the foregoing description that the broad technology of the embodiments can be implemented in a variety of forms. Thus, although the embodiments have been described with reference to specific examples thereof, the true scope of the embodiments should not be limited thereto, as other modifications will become apparent to those skilled in the art upon study of the drawings, the specification, and the following claims.

Claims

1. 1. A parallel processor system comprising: a decode unit for decoding an instruction into a decoded instruction, the decoded instruction being an instruction for performing a parallel dot product operation; and a pipelined systolic dot-product unit for executing the decoded instructions through a plurality of pipeline stages of a systolic processing pipeline, wherein during execution of the decoded instructions, dot products calculated in a first pipeline stage are configured to be selectively written by output hardware to locations selected from one of an output memory and a second pipeline stage, and dot products calculated in a third pipeline stage are configured to be selectively written by output hardware to locations selected from one of the output memory and the first pipeline stage; A parallel processor system comprising:

2. 2. The parallel processor system of claim 1, wherein the decoded instruction is associated with a first source operand and a second source operand, the first source operand being a reference to a memory storing a plurality of data elements of a first matrix, and the second source operand being a reference to a memory storing a plurality of data elements of a second matrix.

3. reading the data elements of the first matrix into a memory of the pipelined systolic dot product unit based on the first source operand; and reading the plurality of data elements of the second matrix into a memory of the pipelined systolic dot product unit based on the second source operand; 3. The parallel processor system of claim 2, further comprising a hardware circuit for:

4. detecting non-zero values ​​within the plurality of data elements of the second matrix, and grouping the detected non-zero values ​​within the plurality of data elements of the second matrix into groups containing one or more data elements; 4. The parallel processor system of claim 3, further comprising a hardware circuit for:

5. 5. The parallel processor system of claim 4, wherein said groups include a number of data elements corresponding to the number of pipeline stages in said systolic processing pipeline.

6. providing a first data element of the group to the first pipeline stage; and providing a second data element of the group to the second pipeline stage; 6. The parallel processor system of claim 5, further comprising a hardware circuit for:

7. 7. The parallel processor system of claim 6, wherein said third pipeline stage is said second pipeline stage, and said systolic processing pipeline comprises two pipeline stages.

8. a fourth pipeline stage between the second pipeline stage and the third pipeline stage, the systolic processing pipeline including four pipeline stages; 7. The parallel processor system of claim 6, further comprising:

9. providing a first set of data elements of the first matrix to the first pipeline stage; and providing a second set of data elements of the second matrix to the second pipeline stage; 7. The parallel processor system of claim 6, further comprising a hardware circuit for:

10. 10. A parallel processor system according to any preceding claim, wherein the systolic processing pipeline is a first systolic processing pipeline and the pipelined systolic dot-product unit comprises a second systolic processing pipeline.

11. 11. The parallel processor system of claim 10, wherein each pipeline stage of the first systolic processing pipeline and the second systolic processing pipeline includes one or more processing elements, each processing element including a plurality of parallel processor channels, each parallel processor channel configurable to perform operations on vectors of a plurality of data elements.

12. Host interface; an interconnect structure coupled to the host interface; and one or more hardware tiles coupled to the interconnect structure; wherein the one or more hardware tiles include sparse matrix multiplication acceleration hardware including a modular systolic processing array having a feedback input, the modular systolic processing array including one or more processing array modules having a first number of pipeline paths, the first number of pipeline paths having a second number of pipeline stages, the first pipeline stage being configurable to receive a feedback output from a last pipeline stage.

13. The accelerator device of claim 12 , wherein the modular systolic processing array comprises a plurality of array modules.

14. 13. The accelerator device of claim 12, wherein the one or more processing array modules include a pipeline path configured to include shared hardware circuitry for reading data elements associated with a first source input and separate hardware circuitry for reading data elements associated with a second source input.

15. 13. The accelerator device of claim 12, wherein the one or more processing array modules include a pipeline path configured to include a separate hardware circuit for reading a data element associated with a first source input and a separate hardware circuit for reading a data element associated with a second source input.

16. 16. The accelerator device of claim 15, wherein the one or more processing array modules include hardware circuitry configured to detect non-zero data elements in the second source input and selectively perform a dot product operation based on the non-zero data elements in the second source input and data elements in the first source input that correspond to the non-zero data elements in the second source input.

17. 17. The accelerator device of claim 16, wherein the one or more processing array modules include a pipeline path including separate output hardware for each pipeline stage.

18. 20. The accelerator device of claim 17, wherein the one or more processing array modules include a first pipeline path configurable to execute a first dot-product instruction having a first set of inputs and a second pipeline path configurable to execute a second dot-product instruction having a second set of inputs.

19. 1. A method for performing a dot product operation on a set of input matrices by a hardware matrix multiplication accelerator having a multi-stage processing pipeline, the method comprising: reading a plurality of data elements of a first matrix into a memory of the hardware matrix multiplication accelerator based on a first source operand; reading a plurality of data elements of a second matrix into the memory of the hardware matrix multiplication accelerator based on a second source operand; detecting non-zero values ​​within the plurality of data elements of the second matrix; grouping the non-zero values ​​within the plurality of data elements of the second matrix into groups containing one or more data elements, the number of data elements in the groups corresponding to the number of stages in the multi-stage processing pipeline of the hardware matrix multiplication accelerator; providing data elements of said group to corresponding stages of said multi-stage processing pipeline; multiplying the data elements of the provided group by a plurality of data elements of the first matrix to generate a set of products; summing the set of products and accumulating the sum of the set of products into an accumulator value; and writing the accumulator value to a next stage of the multi-stage processing pipeline; A method comprising:

20. Writing the accumulator value to the next stage of the multi-stage processing pipeline comprises writing a pipeline feedback value to a first stage of the multi-stage processing pipeline; 20. The method of claim 19, comprising:

21. Providing the data elements of the group to a corresponding stage of the multi-stage processing pipeline includes broadcasting the data elements to a plurality of channels of processing elements of the corresponding stage; 20. The method of claim 19, comprising:

22. detecting the non-zero values ​​within the plurality of data elements of the second matrix comprises detecting the non-zero values ​​within the memory of the hardware matrix multiplication accelerator; 20. The method of claim 19, comprising:

23. A system comprising means for carrying out the method according to any one of claims 19-22.

Images