Optimization mechanism for matrix operations
By defining a two-dimensional memory surface for matrix data retrieval, the method addresses the inefficiencies in deep learning algorithms' matrix layout operations, improving computational utilization and graphics computation performance.
Patent Information
- Application Number
- JP2020187255
- Authority / Receiving Office
- JP · JP
- Patent Type
- Patents
- Current Assignee / Owner
- Priority Date
- 2020-06-02
- Filing Date
- 2020-11-10
- Publication Date
- 2024-11-13
- Estimated Expiration
- 2040-11-10
AI Technical Summary
Deep learning algorithms face low computational utilization due to the resource-intensive nature of layout operations required for matrix multiplication workloads, particularly in sparse matrices, which are common in machine learning applications, leading to reduced performance in graphics computations.
Implementing instructions that define a two-dimensional memory surface for retrieving matrix data before performing matrix layout operations, optimizing the retrieval process to enhance computational efficiency.
This approach improves computational efficiency by reducing the clock cycles required for transpose operations, thereby enhancing the performance of graphics computations in deep learning applications.
Smart Images

Figure 0007582603000003 
Figure 0007582603000004 
Figure 0007582603000005
Abstract
Description
[Technical field]
[0001] Deep learning algorithms are currently implemented in various machine learning applications, such as audio / video recognition, video summarization, etc. Various forms of neural networks (e.g., Convolutional Neural Networks (CNNs), Recurrent Neural Networks (RNNs), Long Short-Term Memory (LSTMs), etc.) are applied to perform such workloads due to their high parallelism. Machine learning applications typically implement matrix multiplication workloads with a significant fraction of zeros (e.g., sparse matrices).
[0002] Deep learning (or machine learning) kernels often suffer from low computational utilization due to the resources required to perform layout operations on input data before executing arithmetic instructions. Most frameworks (e.g., Tensorflow) standardize data layout formats that cannot be directly consumed by machine learning operations. For example, a weight matrix may need to be transposed before performing multiplications. However, performing a transpose operation (or other operations such as transform operations) in software would consume an excessive amount of clock cycles and degrade the performance of graphics computations. [Brief description of the drawings]
[0003] So that the above-mentioned features of the present embodiments can be understood in detail, a more particular description of the above briefly summarized embodiments may be had by reference to embodiments, some of which are illustrated in the accompanying drawings. It should be noted, however, that the accompanying drawings illustrate only typical embodiments and therefore should not be considered as limiting the scope thereof.
[0004] [Figure 1] FIG. 1 is a block diagram of a processing system according to one embodiment.
[0005] [Figure 2A]1 illustrates a computing system and graphics processor provided by embodiments described herein. [Figure 2B] 1 illustrates a computing system and graphics processor provided by embodiments described herein. [Figure 2C] 1 illustrates a computing system and graphics processor provided by embodiments described herein. [Figure 2D] 1 illustrates a computing system and graphics processor provided by embodiments described herein.
[0006] [Figure 3A] 1 illustrates a block diagram of an additional graphics processor and computational accelerator architecture provided by an embodiment. [Figure 3B] 1 illustrates a block diagram of an additional graphics processor and computational accelerator architecture provided by an embodiment. [Figure 3C] 1 illustrates a block diagram of an additional graphics processor and computational accelerator architecture provided by an embodiment.
[0007] [Figure 4] FIG. 2 is a block diagram of a graphics processing engine of a graphics processor according to some embodiments.
[0008] [Figure 5A] 1 illustrates thread execution logic including an array of processing elements employed in a graphics processor core according to an embodiment. [Figure 5B] 1 illustrates thread execution logic including an array of processing elements employed in a graphics processor core according to an embodiment.
[0009] [Figure 6] 1 illustrates an additional execution unit according to an embodiment.
[0010] [Figure 7] FIG. 2 is a block diagram illustrating an instruction format of a graphics processor according to some embodiments.
[0011] [Figure 8] FIG. 2 is a block diagram of a graphics processor according to another embodiment.
[0012] [Figure 9A] 4 illustrates a command format and a command sequence for a graphics processor according to some embodiments. [Figure 9B] 4 illustrates a command format and a command sequence for a graphics processor according to some embodiments.
[0013] [Figure 10] 1 illustrates an exemplary graphics software architecture for a data processing system according to some embodiments.
[0014] [Figure 11A] 1 illustrates an integrated circuit package assembly according to one embodiment. [Figure 11B] 1 illustrates an integrated circuit package assembly according to one embodiment. [Figure 11C] 1 illustrates an integrated circuit package assembly according to one embodiment. [Figure 11D] 1 illustrates an integrated circuit package assembly according to one embodiment.
[0015] [Figure 12] 1 is a block diagram illustrating an exemplary system-on-chip integrated circuit according to one embodiment.
[0016] [Figure 13A] FIG. 13 is a block diagram illustrating an additional example graphics processor. [Figure 13B]FIG. 13 is a block diagram illustrating an additional example graphics processor.
[0017] [Figure 14] 1 illustrates a machine learning software stack according to an embodiment.
[0018] [Figure 15A] 1 illustrates layers of an exemplary deep neural network. [Figure 15B] 1 illustrates layers of an exemplary deep neural network.
[0019] [Figure 16] 1 illustrates an exemplary recurrent neural network.
[0020] [Figure 17] Demonstrates training and deployment of deep neural networks.
[0021] [Figure 18] FIG. 2 is a block diagram showing distributed learning.
[0022] [Figure 19] 1 is a block diagram of a data processing system according to an embodiment.
[0023] [Figure 20] 1 illustrates an embodiment of a matrix operation.
[0024] [Figure 21] 1 illustrates one embodiment of a data block associated with a message.
[0025] [Figure 22] 1 illustrates an embodiment of a block message.
[0026] [Figure 23A] FIG. 13 is a flow diagram illustrating an embodiment of a 2D block read operation. [Figure 23B]FIG. 13 is a flow diagram illustrating an embodiment of a 2D block read operation.
[0027] [Figure 24] 1 illustrates another embodiment of a data block associated with a message.
[0028] [Diagram 25] 1 illustrates one embodiment of a data block stored in a register file.
[0029] [Figure 26A] FIG. 13 is a flow diagram illustrating an embodiment of reading a 2D block with a transpose operation. [Figure 26B] FIG. 13 is a flow diagram illustrating an embodiment of reading a 2D block with a transpose operation.
[0030] [Figure 27A] 1 illustrates another embodiment of a data block stored in a register file. [Figure 27B] 1 illustrates another embodiment of a data block stored in a register file.
[0031] [Figure 28A] FIG. 13 is a flow diagram illustrating an embodiment of reading a 2D block with a transform operation. [Figure 28B] FIG. 13 is a flow diagram illustrating an embodiment of reading a 2D block with a transform operation.
[0032] [Figure 29] 1 illustrates yet another embodiment of a data block stored in a register file.
[0033] [Diagram 30] FIG. 11 is a flow diagram of one embodiment of an instruction execution process. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS
[0034] In an embodiment, an instruction is disclosed for performing a matrix layout operation prior to performing a machine learning matrix calculation. In such an embodiment, the instruction defines a two-dimensional (2D) memory surface that is used to retrieve matrix data from memory prior to performing the matrix layout operation. In a further embodiment, the instruction includes a plurality of parameters that define the 2D memory surface for retrieving blocks of matrix data from memory. System Overview
[0035] 1 is a block diagram of a processing system 100 according to one embodiment. System 100 may be used in a single processor desktop system, a multiprocessor workstation system, or a server system having multiple processors 102 or processor cores 107. In one embodiment, system 100 is a processing platform integrated in a system-on-chip (SoC) integrated circuit for use in mobile, handheld, or embedded devices, such as in Internet of Things (IoT) devices having wired or wireless connections to local or wide area networks.
[0036] In one embodiment, the system 100 can include, be coupled to, or be integrated within a server-based gaming platform, a gaming console, including a game and media console, a mobile gaming console, a handheld gaming console, or an online gaming console. In some embodiments, the system 100 is part of a mobile Internet-connected device, such as a mobile phone, a smart phone, a tablet computing device, or a laptop with low internal storage capacity. The processing system 100 can also include, be coupled to, or be integrated within a wearable device, such as a smart watch wearable device; smart eyewear or clothing that provides visual, audio, or haptic output to supplement a real-world visual, audio, or haptic experience, or is otherwise enhanced with augmented reality (AR) or virtual reality (VR) features to provide text, audio, graphics, video, holographic images, or video or haptic feedback; other augmented reality (AR) devices; or other virtual reality (VR) devices. In some embodiments, the processing system 100 comprises or is part of a television or set-top box device. In one embodiment, system 100 may include, be coupled to, or be integrated within a self-driving vehicle, such as a bus, a tractor trailer, an automobile, a motor or electric cycle, an airplane or glider (or any combination thereof), etc. The self-driving vehicle may use system 100 to process the sensed environment around the vehicle.
[0037] In some embodiments, the one or more processors 102 each include one or more processor cores 107 that process instructions that, when executed, perform operations for system or user software. In some embodiments, at least one of the one or more processor cores 107 is configured to process a particular instruction set 109. In some embodiments, the instruction set 109 may facilitate complex instruction set computing (CISC), reduced instruction set computing (RISC), or computing via very long instruction words (VLIW). The one or more processor cores 107 may process another instruction set 109, which may include instructions to facilitate emulation of other instruction sets. The processor cores 107 may include other processing devices, such as, for example, digital signal processors (DSPs).
[0038] In some embodiments, the processor 102 includes a cache memory 104. Depending on the architecture, the processor 102 may have a single internal cache or multiple levels of internal cache. In some embodiments, the cache memory is shared among various components of the processor 102. In some embodiments, the processor 102 also uses an external cache (e.g., a level 3 (L3) cache or a last level cache (LLC)) (not shown), which may be shared among the processor cores 107 using known cache coherency techniques. A register file 106 may additionally be included in the processor 102 and may include different types of registers (e.g., integer registers, floating point registers, status registers, and an instruction pointer register) for storing different types of data. Some registers may be general purpose registers, while other registers may be specific to the design of the processor 102.
[0039] In some embodiments, one or more processor(s) 102 are coupled to one or more interface bus(es) 110 to transmit communication signals, such as address, data, or control signals, between the processor 102 and other components in the system 100. In one embodiment, the interface bus 110 may be a processor bus, such as a version of a Direct Media Interface (DMI) bus. Note that the processor bus is not limited to a DMI bus, but may include one or more peripheral component interconnect buses (e.g., PCI, PCI Express), memory buses, or other types of interface buses. In one embodiment, the processor(s) 102 include an integrated memory controller 116 and a platform controller hub 130. The integrated memory controller 116 facilitates communication between memory devices and other components of the system 100, while the platform controller hub (PCH) 130 provides connectivity to I / O devices via a local I / O bus.
[0040] Memory device 120 may be a dynamic random access memory (DRAM) device, a static random access memory (SRAM) device, a flash memory device, a phase change memory device, or some other memory device with suitable performance to function as a process memory. In one embodiment, memory device 120 operates as a system memory for system 100 and may store data 122 and instructions 121 for use by one or more processors 102 in executing applications or processes. Memory controller 116 also couples to an optional external graphics processor 118, which may communicate with one or more graphics processors 108 within processor 102 to perform graphics and media operations. In some embodiments, graphics, media, and / or computation operations may be assisted by accelerator 112, which is a co-processor that may be configured to perform a specialized set of graphics, media, or computation operations. For example, in one embodiment, accelerator 112 is a matrix multiplication accelerator used to optimize machine learning or computation operations. In one embodiment, accelerator 112 is a ray tracing accelerator that can be used to perform ray tracing operations in cooperation with graphics processor 108. In one embodiment, an external accelerator 119 may be used instead of or in cooperation with accelerator 112.
[0041] In some embodiments, a display device 111 may be connected to the processor(s) 102. The display device 111 may be one or more of an internal display device, such as a mobile electronic device or laptop device, or an external display device connected via a display interface (e.g., DisplayPort, etc.). In one embodiment, the display device 111 may be a head mounted display (HMD), such as a stereoscopic display device for use in virtual reality (VR) or augmented reality (AR) applications.
[0042] In some embodiments, the platform controller hub 130 allows peripherals to be connected to the memory device 120 and the processor 102 via a high-speed I / O bus. The I / O peripherals include, but are not limited to, an audio controller 146, a network controller 134, a firmware interface 128, a wireless transceiver 126, a touch sensor 125, and a data storage device 124 (e.g., non-volatile memory, volatile memory, hard disk drive, flash memory, NAND, 3D NAND, 3D XPoint, etc.). The data storage device 124 can be connected via a storage interface (e.g., SATA) or via a peripheral bus such as a peripheral component interconnect bus (e.g., PCI, PCI Express). The touch sensor 125 can include a touch screen sensor, a pressure sensor, or a fingerprint sensor. The wireless transceiver 126 can be a Wi-Fi transceiver, a Bluetooth transceiver, or a mobile network transceiver such as a 3G, 4G, 5G, or LTE (Long Term Evolution) transceiver. Firmware interface 128 allows for communication with system firmware and may be, for example, a Unified Extensible Firmware Interface (UEFI). Network controller 134 may allow for network connectivity to a wired network. In some embodiments, a high performance network controller (not shown) couples to interface bus 110. In one embodiment, audio controller 146 is a high definition multi-channel audio controller. In one embodiment, system 100 includes an optional legacy I / O controller 140 for coupling legacy (e.g., Personal System 2 (PS / 2)) devices to the system. Platform controller hub 130 may also connect to one or more Universal Serial Bus (USB) controllers 142 for connecting input devices such as a keyboard and mouse 143 combination, a camera 144, or other USB input devices.
[0043] It will be understood that the illustrated system 100 is exemplary and not limiting, and other types of data processing systems with different configurations may be used. For example, an example of memory controller 116 and platform controller hub 130 may be integrated into a discreet external graphics processor, such as external graphics processor 118. In one embodiment, platform controller hub 130 and / or memory controller 116 may be external to one or more processor(s) 102. For example, system 100 may include external memory controller 116 and platform controller hub 130, which may be configured as a memory controller hub and a peripheral controller hub in a system chipset in communication with processor(s) 102.
[0044] For example, a circuit board ("sled") may be used to install components such as CPUs and memory and is designed to improve thermal performance. In some examples, processing components such as processors are located on the top side of the sled, while components closer to the memory, such as DIMMs, are located on the bottom side of the sled. As a result of the increased airflow provided by this design, the components may operate at higher frequencies and power levels than typical systems, thereby improving performance. Additionally, the sled is configured to blindly mate with power and data communication cables in the rack, enhancing the ability to be quickly removed, upgraded, reinstalled, and / or replaced. Similarly, the individual components located on the sled, such as the processors, accelerators, memory, and data storage drives, are configured to be easily upgraded since they are spaced apart from one another. In the illustrated embodiment, the components additionally include hardware authentication features to prove their authenticity.
[0045] Data centers can utilize a single network architecture ("fabric") that supports multiple other network architectures, including Ethernet and Omnipath. The threads can be coupled to switches via optical fiber, providing higher bandwidth and lower latency than typical twisted pair cabling (e.g., Cat 5, Cat 5e, Cat 6, etc.). The high bandwidth, low latency interconnect and network architecture allows data centers to pool resources such as physically discrete memory, accelerators (e.g., GPUs, graphics accelerators, FPGAs, ASICs, neural network and / or artificial intelligence accelerators, etc.), and data storage drives, and provide them to computational resources (e.g., processors) as needed, allowing the computational resources to access the pooled resources as if they were local.
[0046] A power source or power source can provide voltage and / or current to system 100 or any component or system described herein. In one example, the power source includes an AC-DC (alternating current to direct current) adapter for plugging into a wall outlet. Such an AC power source can be a renewable energy (e.g., solar power) source. In one example, the power source includes a DC power source, such as an external AC-DC converter. In one example, the power source or power source includes wireless charging hardware that charges by proximity to a charging field. In one example, the power source can include an internal battery, an AC power source, a motion-based power source, a solar power source, or a fuel cell source.
[0047] 2A-2D illustrate a computing system and a graphics processor provided by embodiments described herein. Elements in FIG. 2A-2D having the same reference numbers (or names) as elements in any other figures herein may operate or function in a manner similar to, but not limited to, that described elsewhere herein.
[0048] FIG. 2A is a block diagram of one embodiment of a processor 200 having one or more processor cores 202A-202N, an integrated memory controller 214, and an integrated graphics processor 208. Processor 200 may include additional cores up to and including additional core 202N, represented by a dashed box. Each of processor cores 202A-202N includes one or more internal cache units 204A-204N. In some embodiments, each processor core also has access to one or more shared cache units 206. The internal cache units 204A-204N and shared cache unit 206 represent a cache memory hierarchy within processor 200. The cache memory hierarchy may include at least one level of instruction and data cache within each processor core and one or more levels of shared mid-level cache, such as a level 2 (L2), level 3 (L3), level 4 (L4), or other level cache, where the highest level cache before external memory is classified as an LLC. In some embodiments, cache coherency logic maintains coherency between the various shared cache units 206 and 204A-204N.
[0049] In some embodiments, processor 200 may also include a set of one or more bus controller units 216 and a system agent core 210. The one or more bus controller units 216 manage a set of peripheral buses, such as one or more PCI or PCI Express buses. System agent core 210 provides management functions for various processor components. In some embodiments, system agent core 210 includes one or more integrated memory controllers 214 for managing access to various external memory devices (not shown).
[0050] In some embodiments, one or more of the processor cores 202A-202N include support for simultaneous multi-threaded processing. In such embodiments, the system agent core 210 includes components for coordinating and operating the processor cores 202A-202N during multi-threaded processing. The system agent core 210 may additionally include a power control unit (PCU) that includes logic and components for regulating the power state of the processor cores 202A-202N and the graphics processor 208.
[0051] In some embodiments, processor 200 additionally includes a graphics processor 208 for performing graphics processing operations. In some embodiments, graphics processor 208 couples to a system agent core 210 that includes a set of shared cache units 206, and one or more integrated memory controllers 214. In some embodiments, system agent core 210 also includes a display controller 211 for driving output of the graphics processor to one or more coupled displays. In some embodiments, display controller 211 may also be a separate module coupled with the graphics processor via at least one interconnect, or may be integrated within graphics processor 208.
[0052] In some embodiments, a ring-based interconnect unit 212 is used to couple the internal components of processor 200. However, alternative interconnect units may be used, such as a point-to-point interconnect, a switched interconnect, or other techniques, including those known in the art. In some embodiments, graphics processor 208 couples to ring interconnect 212 via I / O link 213.
[0053] The exemplary I / O link 213 represents at least one of several types of I / O interconnects, including an on-package I / O interconnect that facilitates communication between various processor components and a high-performance embedded memory module 218, such as an eDRAM module. In some embodiments, each of the processor cores 202A-202N and the graphics processor 208 may use the embedded memory module 218 as a shared last-level cache.
[0054] In some embodiments, the processor cores 202A-202N are homogenous cores executing the same instruction set architecture. In another embodiment, the processor cores 202A-202N differ in terms of instruction set architecture (ISA), where one or more of the processor cores 202A-202N execute a first instruction set and at least one of the other cores executes a subset of the first instruction set or executes a different instruction set. In one embodiment, the processor cores 202A-202N differ in terms of microarchitecture, where one or more cores having relatively high power consumption are combined with one or more power cores having lower power consumption. In one embodiment, the processor cores 202A-202N differ in terms of computational capabilities. Additionally, the processor 200 may be implemented on one or more chips or as a SoC integrated circuit having the illustrated components, in addition to other components.
[0055] FIG. 2B is a block diagram of hardware logic of a graphics processor core 219 according to some embodiments described herein. Elements in FIG. 2B having the same reference number (or name) as elements of any other figure herein may operate or function in a similar manner as described elsewhere herein, including, but not limited to, the same manner as described elsewhere herein. The graphics processor core 219 may be referred to as a core slice and may be one or more graphics cores in a modular graphics processor. The graphics processor core 219 is illustrative of one graphics core slice, and a graphics processor as described herein may include multiple graphics core slices based on target power and performance envelopes. Each graphics processor core 219 may include a fixed function block 230 coupled to multiple sub-cores 221A-221F, also referred to as sub-slices, that include modular blocks of general purpose and fixed function logic.
[0056] In some embodiments, fixed function block 230 includes a geometry / fixed function pipeline 231 that may be shared by all sub-cores in graphics processor core 219, e.g., in lower performance and / or lower power graphics processor implementations. In various embodiments, geometry / fixed function pipeline 231 includes a 3D fixed function pipeline (e.g., 3D pipeline 312 of FIGS. 3A-3C and 4, described below), a video front end unit, a thread spawner and thread dispatcher, and a unified return buffer manager that manages a unified return buffer (e.g., unified return buffer 418 of FIG. 4, described below).
[0057] In one embodiment, the fixed function block 230 also includes a graphics SoC interface 232, a graphics microcontroller 233, and a media pipeline 234. The graphics SoC interface 232 provides an interface between the graphics processor core 219 and other processor cores in the system-on-chip integrated circuit. The graphics microcontroller 233 is a programmable sub-processor that can be configured to manage various functions of the graphics processor core 219, including thread dispatch, scheduling, and preemption. The media pipeline 234 (e.g., media pipeline 316 of FIGS. 3A-3C and 4) includes logic to facilitate decoding, encoding, pre-processing, and / or post-processing of multimedia data, including image data and video data. The media pipeline 234 implements media operations via requests to computational or sampling logic in the sub-cores 221-221F.
[0058] In one embodiment, the graphics SoC interface 232 enables the graphics processor core 219 to communicate with general-purpose application processor cores (e.g., CPUs) and / or other components in the SoC, including memory hierarchy elements such as shared last-level cache memory, system RAM, and / or embedded on-chip or on-package DRAM. The graphics SoC interface 232 may also enable communication with fixed function devices in the SoC, such as a camera imaging pipeline, and may enable the use and / or implementation of global memory atomics that may be shared between the graphics processor core 219 and CPUs in the SoC. The graphics SoC interface 232 may also implement power management controls for the graphics processor core 219, and may enable an interface between the clock domain of the graphics processor core 219 and other clock domains in the SoC. In one embodiment, the graphics SoC interface 232 enables the receipt of command buffers from a command streamer and a global thread dispatcher configured to provide commands and instructions to each of one or more graphics cores in the graphics processor. Commands and instructions may be dispatched to the media pipeline 234 if media operations are to be performed, or to the geometry and fixed function pipelines (e.g., geometry and fixed function pipeline 231, geometry and fixed function pipeline 237) if graphics processing operations are to be performed.
[0059] Graphics microcontroller 233 can be configured to perform various scheduling and management tasks for graphics processor core 219. In one embodiment, graphics microcontroller 233 can perform scheduling of graphics and / or computational workloads on various graphics parallel engines within the arrays of execution units (EUs) 222A-222F, 224A-224F in sub-cores 221A-221F. In this scheduling model, host software executing on a CPU core of an SoC including graphics processor core 219 can submit a workload to one of multiple graphics processor doorbells, which invokes a scheduling operation on the appropriate graphics engine. Scheduling operations include determining which workload to run next, submitting the workload to a command streamer, preempting existing workloads running on the engines, monitoring the progress of the workload, and notifying the host software when the workload is completed. In one embodiment, graphics microcontroller 233 may also facilitate low power or idle states for graphics processor core 219 and may provide graphics processor core 219 with the ability to save and restore registers within graphics processor core 219 across low power state transitions independent of the operating system and / or graphics driver software on the system.
[0060] Graphics processor core 219 may have more or less than the illustrated sub-cores 221A-221F and may have up to N modular sub-cores. For each set of N sub-cores, graphics processor core 219 may also include shared function logic 235, shared and / or cache memory 236, geometry / fixed function pipeline 237, and additional fixed function logic 238 for accelerating various graphics and computational processing operations. Shared function logic 235 may include logic units associated with shared function logic 420 of FIG. 4 (e.g., sampler, arithmetic, and / or inter-thread communication logic) that may be shared by each of the N sub-cores in graphics processor core 219. Shared and / or cache memory 236 may be a last-level cache for the set of N sub-cores 221A-221F in graphics processor core 219 and may also function as a shared memory accessible by multiple sub-cores. The geometry / fixed-function pipeline 237 may be included in place of the geometry / fixed-function pipeline 231 in the fixed-function block 230 and may include the same or similar logic units.
[0061] In one embodiment, graphics processor core 219 includes additional fixed function logic 238, which may include various fixed function acceleration logic for use by graphics processor core 219. In one embodiment, additional fixed function logic 238 includes an additional geometry pipeline for use in position-only shading. In position-only shading, there are two geometry pipelines: a full geometry pipeline in geometry / fixed function pipelines 238 and 231, and a culling pipeline, which is an additional geometry pipeline that may be included in additional fixed function logic 238. In one embodiment, the culling pipeline is a scaled-down version of the full geometry pipeline. The full pipeline and the culling pipeline may run different instances of the same application, with each instance having a separate context. In position-only shading, long culling runs of discarded triangles may be hidden, allowing shading to complete sooner in some instances. For example, in one embodiment, the culling pipeline logic in the additional fixed function logic 238 can execute position shaders in parallel with the main application, and the culling pipeline generally produces significant results faster than the full pipeline because it only fetches and shades position attributes of vertices without rasterizing and rendering pixels to a frame buffer. The culling pipeline can use the generated significant results to calculate visibility information for all triangles, regardless of whether they have been culled or not. The full pipeline (which may be referred to in this instance as a replay pipeline) can consume the visibility information to skip culled triangles in order to shade only visible triangles that are eventually passed to the rasterization phase.
[0062] In one embodiment, the additional fixed function logic 238 may also include machine learning acceleration logic, such as fixed function matrix multiplication logic, for implementations that include optimizations for machine learning training or inference.
[0063] Contained within each graphics sub-core 221A-221F is a set of execution resources that may be used to execute graphics, media, and compute operations in response to requests by a graphics pipeline, a media pipeline, or a shader program. The graphics sub-cores 221A-221F include a number of EU arrays 222A-222F, 224A-224F, thread dispatch and inter-thread communication (TD / IC) logic 223A-223F, 3D (e.g., texture) samplers 225A-225F, media samplers 206A-206F, shader processors 227A-227F, and shared local memory (SLM) 228A-228F. The EU arrays 222A-222F and 224A-224F each include a number of execution units, which are general-purpose graphics processing units capable of performing floating-point and integer / fixed-point logic operations in service of graphics, media, or compute operations, including graphics, media, or compute shader programs. The TD / IC logic 223A-223F performs local thread dispatch and thread control operations for the execution units in the sub-cores and facilitates communication between threads executing on the execution units of the sub-cores. The 3D samplers 225A-225F can read textures or other 3D graphics related data into memory. The 3D samplers can read texture data differently based on the configured sample state and the texture format associated with a given texture. The media samplers 206A-206F can perform similar read operations based on the type and format associated with the media data. In one embodiment, each graphics sub-core 221A-221F can alternately include a unified 3D and media sampler. Threads executing on the execution units in each sub-core 221A-221F leverage a shared local memory 228A-228F within each sub-core, allowing threads executing within a thread group to execute using a common pool of on-chip memory.
[0064] 2C illustrates a graphics processing unit (GPU) 239 that includes a dedicated set of graphics processing resources arranged into multi-core groups 240A-240N. Although details of only a single multi-core group 240A are provided, it will be understood that the other multi-core groups 240B-240N may include the same or similar sets of graphics processing resources.
[0065] As shown, multi-core group 240A may include a set of graphics cores 243, a set of tensor cores 244, and a set of ray tracing cores 245. Scheduler / dispatcher 241 schedules and dispatches graphics threads for execution on the various cores 243, 244, and 245. A set of register files 242 store operand values used by cores 243, 244, and 245 in executing the graphics threads. These may include, for example, integer registers for storing integer values, floating point registers for storing floating point values, vector registers for storing packed data elements (integer and / or floating point data elements), and tile registers for storing tensor / matrix values. In one embodiment, the tile registers are implemented as a combined set of vector registers.
[0066] One or more combined level 1 (L1) cache and shared memory units 247 store graphics data, such as texture data, vertex data, pixel data, ray data, bounding volume data, etc., locally within each multicore group 240A. One or more texture units 247 may also be used to perform texturing operations, such as texture mapping and sampling. A level 2 (L2) cache 253, shared by all or a subset of the multicore groups 240A-240N, stores graphics data and / or instructions for multiple simultaneous graphics threads. As shown, the L2 cache 253 may be shared across multiple multicore groups 240A-240N. One or more memory controllers 248 couple the GPU 239 to memory 249, which may be system memory (e.g., DRAM) and / or dedicated graphics memory (e.g., GDDR6 memory).
[0067] Input / output (I / O) circuitry 250 couples GPU 239 to one or more I / O devices 252, such as a digital signal processor (DSP), a network controller, or a user input device. An on-chip interconnect may be used to couple I / O device 252 to GPU 239 and memory 249. One or more I / O memory management units (IOMMUs) 251 of I / O circuitry 250 directly couple I / O device 252 to system memory 249. In one embodiment, IOMMU 251 manages multiple sets of page tables to map virtual addresses to physical addresses in system memory 249. In this embodiment, I / O device 252, CPU(s) 246, and GPU(s) 239 may share the same virtual address space.
[0068] In one implementation, IOMMU 251 supports virtualization. In this case, it may manage a first set of page tables for mapping guest / graphics virtual addresses to guest / graphics physical addresses and a second set of page tables (e.g., in system memory 249) for mapping guest / graphics physical addresses to system / host physical addresses. The base addresses of each of the first and second sets of page tables may be stored in control registers and swapped out on a context switch (e.g., so that a new context is provided access to the associated set of page tables). Although not shown in FIG. 2C, each of cores 243, 244, and 245 and / or multicore groups 240A-240N may include a translation lookaside buffer (TLB) for caching guest virtual to guest physical, guest physical to host physical, and guest virtual to host physical translations.
[0069] In one embodiment, CPU 246, GPU 239, and I / O devices 252 are integrated onto a single semiconductor chip and / or chip package. The depicted memory 249 may be integrated onto the same chip or may be coupled to memory controller 248 via an off-chip interface. In one implementation, memory 249 includes GDDR6 memory that shares the same virtual address space as other physical system level memory, although the underlying principles are not limited to this particular implementation.
[0070] In one embodiment, tensor cores 244 include multiple execution units specifically designed to perform matrix operations, which are fundamental computational operations used to perform deep learning operations. For example, simultaneous matrix multiplication operations may be used for neural network training and inference. Tensor cores 244 may perform matrix processing using a variety of operand precisions, including single-precision floating point (e.g., 32-bit), half-precision floating point (e.g., 16-bit), integer word (16-bit), byte (8-bit), and half-byte (4-bit). In one embodiment, the neural network implementation extracts features of each rendered scene, potentially combining details from multiple frames to construct a high-quality final image.
[0071] In a deep learning implementation, parallel matrix multiplication work may be scheduled for execution on tensor cores 244. In particular, training of neural networks requires a significant number of matrix dot product operations. To process the dot product formulation of an N×N×N matrix multiplication, tensor cores 244 may include at least N dot product processing elements. Before the matrix multiplication begins, an entire matrix is loaded into a tile register, and at least one column of a second matrix is loaded every cycle for N cycles. In each cycle, there are N dot products to be processed.
[0072] Matrix elements may be stored in different precisions depending on the particular implementation, including 16-bit words, 8-bit bytes (e.g., INT8), 4-bit half-bytes (e.g., INT4), etc. Different precision modes may be specified for tensor cores 244 to ensure that the most efficient precision is used for different workloads (e.g., inference workloads that can tolerate quantization into bytes and half-bytes, etc.).
[0073] In one embodiment, ray tracing core 245 accelerates ray tracing operations for both real-time ray tracing and non-real-time ray tracing implementations. In particular, ray tracing core 245 includes ray traversal / intersection circuitry for performing ray traversal using a bounding volume hierarchy (BVH) and identifying intersections of rays with primitives enclosed within the BVH volume. Ray tracing core 245 may also include circuitry for performing depth checking and culling (e.g., using a Z-buffer or similar arrangement). In one implementation, ray tracing core 245 performs traversal and intersection operations in conjunction with the image denoising techniques described herein, at least some of which may be executed on tensor core 244. For example, in one embodiment, tensor core 244 implements a deep learning neural network to perform denoising of frames generated by ray tracing core 245. However, the CPU(s) 246, the graphics core 243, and / or the ray tracing core 245 may implement all or part of the denoising and / or deep learning algorithms.
[0074] Additionally, as discussed above, a distributed approach to denoising may be employed where the GPU 239 is in a computing device coupled to other computing devices via a network or high speed interconnect. In this embodiment, the interconnected computing devices share neural network learning / training data to improve the speed at which the entire system learns to perform denoising for different types of image frames and / or different graphics applications.
[0075] In one embodiment, ray tracing core 245 handles all BVH traversal and intersection of ray primitives, preventing graphics core 243 from being overloaded with thousands of instructions per ray. In one embodiment, each ray tracing core 245 includes a first set of specialized circuitry for performing bounding box tests (e.g., for traversal operations) and a second set of specialized circuitry for performing ray-triangle intersection tests (e.g., intersection of traversed rays). Thus, in one embodiment, multi-core group 240A can simply launch ray probes, and ray tracing core 245 independently performs ray traversal and intersection, returning hit data (e.g., hit, no hit, multiple hits, etc.) to the thread context. The other cores 243 and 244 are freed to perform other graphics or computational work while ray tracing core 245 performs traversal and intersection operations.
[0076] In one embodiment, each ray tracing core 245 includes a traversal unit for performing BVH test operations and an intersection unit for performing ray primitive intersection tests. The intersection unit generates a "hit", "no hit", or "multiple hit" response and provides it to the appropriate thread. During traversal and intersection operations, the execution resources of other cores (e.g., graphics core 243 and tensor core 244) are freed to perform other forms of graphics work.
[0077] In one particular embodiment described below, a hybrid rasterization / ray tracing approach is used in which work is distributed between the graphics core 243 and the ray tracing core 245.
[0078] In one embodiment, the ray tracing core 245 (and / or other cores 243 and 244) includes hardware support for a ray tracing instruction set, such as Microsoft's DirectX Ray Tracing (DXR), including the DispatchRays command, and includes ray generation shaders, nearest hit shaders, any hit shaders, and miss shaders, which allow for the allocation of a unique set of shaders and textures for each object. Another ray tracing platform that may be supported by the ray tracing core 245, graphics core 243, and tensor core 244 is Vulkan 1.1.85. It should be noted, however, that the principles underlying the present invention are not limited to any particular ray tracing ISA.
[0079] In general, the various cores 245, 244, and 243 may support a ray tracing instruction set that includes instructions / functions for ray generation, closest hit, any hit, ray-primitive intersection, building hierarchical bounding boxes per primitive, misses, visits, and exceptions. More specifically, in one embodiment, the ray tracing instructions include instructions for performing the following functions:
[0080] Ray generation - Ray generation instructions can be executed for each pixel, sample, or other user-defined work allocation.
[0081] Closest Hit - A closest hit instruction can be executed to locate the closest intersection point of a ray with a primitive in a scene.
[0082] Any Hit - The any hit command identifies multiple intersection points between rays and primitives in the scene and potentially identifies a new closest intersection point.
[0083] The intersection-intersection instructions perform ray-primitive intersection tests and output the results.
[0084] Build a per-primitive hierarchical bounding box - This instruction builds a bounding box around a given primitive or group of primitives (e.g., when building a new BVH or other acceleration data structure).
[0085] Miss - indicates that the ray misses all geometry in the scene, or a specified region of the scene.
[0086] Visit - indicates the child volume through which the ray passes.
[0087] Exceptions - Contains various types of exception handlers (e.g., invoked for various error conditions).
[0088] 2D is a block diagram of a general purpose graphics processing unit (GPGPU) 270 that may be configured as a graphics processor and / or computational accelerator, according to embodiments described herein. GPGPU 270 may be interconnected with a host processor (e.g., one or more CPU(s) 246) and memories 271 and 272 via one or more system and / or memory buses. In one embodiment, memory 271 is a system memory that may be shared with one or more CPU(s) 246, while memory 272 is a device memory dedicated to GPGPU 270. In one embodiment, components within GPGPU 270 and device memory 272 may be mapped to memory addresses accessible to said one or more CPU(s) 246. Access to memories 271 and 272 may be facilitated via memory controller 268. In one embodiment, memory controller 268 may include an internal direct memory access (DMA) controller 269 or logic for performing operations that would otherwise be performed by a DMA controller.
[0089] The GPGPU 270 includes a number of cache memories, including an L2 cache 253, an L1 cache 254, an instruction cache 255, and a shared memory 256, at least some of which may be partitioned as cache memories. The GPGPU 270 also includes a number of computation units 260A-260N. Each computation unit 260A-260N includes a set of vector registers 261, scalar registers 262, a vector logic unit 263, and a scalar logic unit 264. The computation units 260A-260N may also include a local shared memory 265 and a program counter 266. The computation units 260A-260N may be coupled with a constant cache 267, which may be used to store constant data, which is data that does not change during execution of a kernel or shader program executing on the GPGPU 270. In one embodiment, the constant cache 267 is a scalar data cache, and the cached data may be fetched directly into the scalar registers 262.
[0090] During operation, one or more CPU(s) 246 can write commands to registers or memory in GPGPU 270 that are mapped to an accessible address space. Command processor 257 can read commands from registers or memory and determine how those commands are processed in GPGPU 270. Thread dispatcher 258 can then be used to dispatch threads to computation units 260A-260N to execute those commands. Each computation unit 260A-260N can execute threads independently of the other computation units. In addition, each computation unit 260A-260N can be independently configured for conditional computation and can conditionally output computation results to memory. Command processor 257 can interrupt one or more CPU(s) 246 when submitted commands are completed.
[0091] 3A-3C show block diagrams of additional graphics processor and computation accelerator architectures provided by embodiments described herein. Elements in FIG. 3A-3C having the same reference numbers (or names) as elements in any other figures herein can operate or function in any manner similar to, but not limited to, those described elsewhere herein.
[0092] 3A is a block diagram of a graphics processor 300, which may be a discrete graphics processing unit or may be integrated with multiple processing cores or other semiconductor devices, such as, but not limited to, memory devices and network interfaces. In some embodiments, the graphics processor communicates with registers on the graphics processor and commands located in the processor memory via a memory-mapped I / O interface. In some embodiments, the graphics processor 300 includes a memory interface 314 for accessing memory. The memory interface 314 may be an interface to local memory, one or more internal caches, one or more shared external caches, and / or system memory.
[0093] In some embodiments, the graphics processor 300 also includes a display controller 302 for driving the display output of data to a display device 318. The display controller 302 includes hardware for one or more overlay planes for the display and composition of multiple layers of video or user interface elements. The display device 318 may be an internal or external display device. In one embodiment, the display device 318 is a head-mounted display device, such as a virtual reality (VR) display device or an augmented reality (AR) display device. In some embodiments, the graphics processor 300 includes a video codec engine 306 for encoding, decoding, or transcoding media to, from, or between one or more media encoding formats. These media encoding formats include, but are not limited to, Moving Picture Experts Group (MPEG) formats such as MPEG-2, Advanced Video Coding (AVC) formats such as H.264 / MPEG-4 AVC, H.265 / HEVC, Alliance for Open Media (AOMedia) VP8, VP9, and Society of Motion Picture and Television Engineers (SMPTE) 421M / VC-1, Joint Photographic Experts Group (JPEG) formats, and Motion JPEG (MJPEG) formats.
[0094] In some embodiments, graphics processor 300 includes a block image transfer (BLIT) engine 304 for performing two-dimensional (2D) rasterization operations, including, for example, bit-boundary block transfers. However, in one embodiment, 2D graphics operations are performed using one or more components of a graphics processing engine (GPE) 310. In some embodiments, GPE 310 is a computation engine for performing graphics operations, including three-dimensional (3D) graphics operations and media operations.
[0095] In some embodiments, GPE 310 includes a 3D pipeline 312 for performing 3D operations such as rendering 3D images and scenes using processing functions that operate on 3D primitive shapes (e.g., rectangles, triangles, etc.). 3D pipeline 312 includes programmable fixed-function elements that perform various tasks within the element and / or create execution threads in 3D / media subsystem 315. While 3D pipeline 312 can be used to perform media operations, embodiments of GPE 310 also include a media pipeline 316 that is specifically used to perform media operations such as video post-processing and image processing.
[0096] In some embodiments, the media pipeline 316 includes fixed function or programmable logic units for performing one or more specialized media operations, such as video decode acceleration, video deinterlacing, video encode acceleration, etc., instead of or for the video codec engine 306. In some embodiments, the media pipeline 316 additionally includes a thread creation unit for creating threads for execution on the 3D / media subsystem 315. The created threads perform computations for the media operations on one or more graphics execution units included in the 3D / media subsystem 315.
[0097] In some embodiments, the 3D / media subsystem 315 includes logic for executing threads created by the 3D pipeline 312 and the media pipeline 316. In one embodiment, the pipelines send thread execution requests to the 3D / media subsystem 315, which includes thread dispatch logic for arbitrating and dispatching the various requests to available thread execution resources. The execution resources include an array of graphics execution units for processing the 3D and media threads. In some embodiments, the 3D / media subsystem 315 includes one or more internal caches for thread instructions and data. In some embodiments, the subsystem also includes shared memory, including registers and addressable memory, for sharing data between threads and storing output data.
[0098] FIG. 3B illustrates a graphics processor 320 having a tile architecture in accordance with an embodiment described herein. In one embodiment, the graphics processor 320 includes a graphics processing engine cluster 322 having multiple instances of the graphics processing engine 310 of FIG. 3A in the graphics engine tiles 310A-310D. Each of the graphics engine tiles 310A-310D can be interconnected via a set of tile interconnects 323A-323F. Each of the graphics engine tiles 310A-310D can also be connected to memory modules or memory devices 326A-326D via memory interconnects 325A-325D. The memory devices 326A-326D can use any graphics memory technology. For example, the memory devices 326A-326D can be graphics double data rate (GDDR) memory. In one embodiment, the memory devices 326A-326D are high bandwidth memory (HBM) modules that can be on-die with the respective graphics engine tiles 310A-310D. In one embodiment, memory devices 326A-326D are stacked memory devices that can be stacked on top of their respective graphics engine tiles 310A-310D. In one embodiment, each graphics engine tile 310A-310D and associated memory device 326A-326D resides on a separate chiplet that is attached to a base die or base substrate, as shown in further detail in Figures 11B-11D.
[0099] The graphics processing engine cluster 322 may be connected to an on-chip or on-package fabric interconnect 324. The fabric interconnect 324 may enable communication between the graphics engine tiles 310A-310D and components such as the video codec engine 306 and one or more copy engines 304. The copy engines 304 may be used to move data externally, internally, or between the memory devices 326A-326D and memory external to the graphics processor 320 (e.g., system memory). The fabric interconnect 324 may also be used to interconnect the graphics engine tiles 310A-310D. The graphics processor 320 may optionally include a display controller 302 to enable connection to an external display device 318. The graphics processor may be configured as a graphics or computation accelerator. In an accelerator configuration, the display controller 302 and the display device 318 may be omitted.
[0100] Graphics processor 320 may be connected to a host system via host interface 328. Host interface 328 may enable communication between graphics processor 320, system memory, and / or other system components. Host interface 328 may be, for example, a PCI Express bus or another type of host system interface.
[0101] FIG. 3C illustrates a computation accelerator 330 according to an embodiment described herein. The computation accelerator 330 may include an architecture similar to the graphics processor 320 of FIG. 3B and is optimized for computation acceleration. The computation engine cluster 332 may include a set of computation engine tiles 340A-340D that include execution logic optimized for parallel or vector-based general-purpose computation operations. In some embodiments, the computation engine tiles 340A-340D do not include fixed-function graphics processing logic, but in one embodiment, one or more of the computation engine tiles 340A-340D may include logic to perform media acceleration. The computation engine tiles 340A-340D may be connected to memory devices 326A-326D via memory interconnects 325A-325D. The memory devices 326A-326D and memory interconnects 325A-325D may be of similar technology to the graphics processor 320 or may be different. The graphics compute engine tiles 340A-340D may be interconnected via a set of tile interconnects 323A-323F, may be connected to the fabric interconnect 324, and / or may be interconnected to one another. In one embodiment, the compute accelerator 330 includes a large L3 cache 336 that may be configured as a device-wide cache. The compute accelerator 330 may also be connected to a host processor and memory via a host interface 328 in a manner similar to the graphics processor 320 of FIG. 3B. Graphics Processing Engine
[0102] FIG. 4 is a block diagram of a graphics processing engine 410 of a graphics processor according to some embodiments. In one embodiment, the graphics processing engine (GPE) 410 is a version of the GPE 310 of FIG. 3A and may represent the graphics engine tiles 310A-310D of FIG. 3B. Elements in FIG. 4 having the same reference numbers (or names) as elements in any other figure herein may operate or function in any manner similar to that described elsewhere herein, including, but not limited to, in FIG. 3A. For example, FIG. 3A illustrates a 3D pipeline 312 and a media pipeline 316. The media pipeline 316 is optional in some embodiments of the GPE 410 and may not be explicitly included within the GPE 410. For example, and in at least one embodiment, a separate media and / or image processor is coupled to the GPE 410.
[0103] In some embodiments, the GPE 410 is coupled to or includes a command streamer 403 that provides a command stream to the 3D pipeline 312 and / or the media pipeline 316. In some embodiments, the command streamer 403 is coupled to a memory, which may be one or more of a system memory or an internal cache memory and a shared cache memory. In some embodiments, the command streamer 403 receives commands from the memory and sends commands to the 3D pipeline 312 and / or the media pipeline 316. The commands are instructions fetched from a ring buffer that stores commands for the 3D pipeline 312 and the media pipeline 316. In one embodiment, the ring buffer may additionally include a batch command buffer that stores a batch of multiple commands. The commands for the 3D pipeline 312 may also include references to data stored in memory, such as, but not limited to, vertex and geometry data for the 3D pipeline 312 and / or image data and memory objects for the media pipeline 316. The 3D pipeline 312 and the media pipeline 316 process commands and data by executing operations through logic within the respective pipelines or by dispatching one or more execution threads to the graphics core array 414. In one embodiment, the graphics core array 414 includes one or more blocks of graphics cores (e.g., graphics core(s) 415A, graphics core(s) 415B), each block including one or more graphics cores. Each graphics core includes a set of graphics execution resources including general-purpose and graphics-specific execution logic for performing graphics and computation operations, as well as fixed-function texture processing and / or machine learning and artificial intelligence acceleration logic.
[0104] In various embodiments, the 3D pipeline 312 may include fixed function and programmable logic for processing one or more shader programs, such as vertex shaders, geometry shaders, pixel shaders, fragment shaders, compute shaders, or other shader programs, by processing instructions and dispatching execution threads to the graphics core array 414. The graphics core array 414 provides a unified block of execution resources for use in processing these shader programs. The general-purpose execution logic (e.g., execution units) within the graphics core(s) 415A-414B of the graphics core array 414 includes support for a variety of 3D API shader languages and can execute multiple concurrent threads of execution associated with multiple shaders.
[0105] In some embodiments, the graphics core array 414 includes execution logic for performing media functions such as video and / or image processing. In one embodiment, the execution unit includes general purpose logic that is programmable to perform parallel general purpose computing operations in addition to graphics processing operations. The general purpose logic may perform processing operations in parallel or in conjunction with general purpose logic in the processor core(s) 107 or processor cores 202A-202N of FIG. 1, as in FIG. 2A.
[0106] Output data generated by threads executing on graphics core array 414 may output data to memory in unified return buffer (URB) 418. URB 418 may store data for multiple threads. In some embodiments, URB 418 may be used to send data between different threads executing on graphics core array 414. In some embodiments, URB 418 may additionally be used for synchronization between threads on the graphics core array and fixed function logic in shared function logic 420.
[0107] In some embodiments, graphics core array 414 is scalable such that the array includes a variable number of graphics cores, each having a variable number of execution units based on the target power and performance levels of GPE 410. In one embodiment, execution resources are dynamically scalable such that execution resources may be enabled or disabled as needed.
[0108] The graphics core array 414 couples to shared function logic 420, which includes multiple resources shared among the graphics cores in the graphics core array. The shared functions in the shared function logic 420 are hardware logic units that provide specialized auxiliary functions to the graphics core array 414. In various embodiments, the shared function logic 420 includes, but is not limited to, a sampler 421, math logic 422, and inter-thread communication (ITC) 423 logic. Additionally, some embodiments implement one or more cache(s) 425 in the shared function logic 420.
[0109] Shared functionality is implemented at least when the demand for a given specialized functionality is insufficient for inclusion within the graphics core array 414. Instead, a single instantiation of that specialized functionality is implemented as a standalone entity within the shared functionality logic 420 and shared among the execution resources within the graphics core array 414. The exact set of functionality shared among and included within the graphics core array 414 varies by embodiment. In some embodiments, certain shared functionality within the shared functionality logic 420 that is used extensively by the graphics core array 414 may be included within the shared functionality logic 416 within the graphics core array 414. In various embodiments, the shared functionality logic 416 within the graphics core array 414 may include some or all of the logic within the shared functionality logic 420. In one embodiment, all logic elements within the shared functionality logic 420 may be duplicated within the shared functionality logic 416 of the graphics core array 414. In one embodiment, the shared functionality logic 420 is favorably omitted from the shared functionality logic 416 within the graphics core array 414. Execution Unit
[0110] 5A and 5B illustrate thread execution logic 500 including an array of processing elements employed in a graphics processor core according to embodiments described herein. Elements in FIG. 5A and 5B having the same reference numbers (or names) as elements in any other figures herein may operate or function in any manner similar to that described elsewhere herein, including, but not limited to, those described elsewhere herein. FIG. 5A and 5B show an overview of thread execution logic 500 that may be representative of the hardware logic illustrated in each of sub-cores 221A-221F of FIG. 2B. FIG. 5A is representative of an execution unit in a general-purpose graphics processor, and FIG. 5B is representative of an execution unit that may be used in a computational accelerator.
[0111] As shown in FIG. 5A, in some embodiments, the thread execution logic 500 includes a shader processor 502, a thread dispatcher 504, an instruction cache 506, a scalable execution unit array including multiple execution units 508A-508N, a sampler 510, a shared local memory 511, a data cache 512, and a data port 514. In one embodiment, the scalable execution unit array can be dynamically scaled by enabling or disabling one or more execution units (e.g., any of execution units 508A, 508B, 508C, and 508D through 508N-1 and 508N) based on the computational demands of the workload. In one embodiment, the included components are interconnected via an interconnect fabric associated with each component. In some embodiments, the thread execution logic 500 includes one or more connections to a memory, such as a system memory or a cache memory, via one or more of the instruction cache 506, the data port 514, the sampler 510, and the execution units 508A-508N. In some embodiments, each execution unit (e.g., 508A) is a stand-alone, programmable, general-purpose computational unit capable of executing multiple simultaneous hardware threads, processing multiple data elements in parallel for each thread. In various embodiments, the array of execution units 508A-508N is scalable to include any number of individual execution units.
[0112] In some embodiments, the execution units 508A-508N are primarily used to execute shader programs. The shader processor 502 can process various shader programs and dispatch execution threads associated with the shader programs via the thread dispatcher 504. In one embodiment, the thread dispatcher includes logic for arbitrating thread initiation requests from the graphics and media pipelines and instantiating the requested threads on one or more of the execution units 508A-508N. For example, the geometry pipeline can dispatch vertex, tessellation, or geometry shaders to thread execution logic for processing. In some embodiments, the thread dispatcher 504 can also handle run-time thread creation requests from executing shader programs.
[0113] In some embodiments, the execution units 508A-508N support an instruction set that includes native support for many standard 3D graphics shader instructions so that shader programs from graphics libraries (e.g., Direct 3D and OpenGL) execute with minimal translation. The execution units support vertex and geometry processing (e.g., vertex programs, geometry programs, vertex shaders), pixel processing (e.g., pixel shaders, fragment shaders), and general-purpose processing (e.g., compute and media shaders). Each of the execution units 508A-508N is capable of multi-issue single instruction multiple data (SIMD) execution, allowing for multi-threaded operation to enable an efficient execution environment despite higher latency memory accesses. Each hardware thread within each execution unit has a dedicated high-bandwidth register file and associated independent thread state. Execution is multi-issue per clock with a pipeline that allows integer, single and double precision floating point operations, SIMD branching capabilities, logic operations, transcendental operations, and other miscellaneous operations. While waiting for data from memory or one of the shared functions, dependency logic within the execution units 508A-508N puts the waiting thread to sleep until the requested data is returned. While the waiting thread is sleeping, hardware resources may be dedicated to processing other threads. For example, during the delay associated with a vertex shader operation, the execution unit may execute operations for another type of shader program, including a pixel shader, a fragment shader, or a different vertex shader. In various embodiments, execution through the use of Single Instruction Multiple Threads (SIMT) may be applied as an alternative to or in addition to the use of SIMD. References to SIMD cores or operations may also apply to SIMT or to SIMD in combination with SIMT.
[0114] Each execution unit of execution units 508A-508N operates on an array of data elements. The number of data elements is the "execution size," or number of channels, of the instruction. An execution channel is the logical execution unit for accessing, masking, and flow control of data elements within an instruction. The number of channels may be independent of the number of physical arithmetic logic units (ALUs) or floating point units (FPUs) for a particular graphics processor. In some embodiments, execution units 508A-508N support integer and floating point data types.
[0115] The instruction set of the execution unit includes SIMD instructions. The various data elements can be stored in registers as packed data types, and the execution unit processes the various elements based on the data size of the elements. For example, when operating on a 256-bit wide vector, the 256-bit vector is stored in a register, and the execution unit operates on the vector as four separate 54-bit packed data elements (quadword (QW) sized data elements), eight separate 32-bit packed data elements (doubleword (DW) sized data elements), sixteen separate 16-bit packed data elements (word (W) sized data elements), or thirty-two separate 8-bit data elements (byte (B) sized data elements). However, different vector widths and register sizes are possible.
[0116] In one embodiment, one or more execution units can be combined into fused execution units 509A-509N, which have thread control logic (507A-507N) common to the fused execution units EU. Multiple EUs can be fused into an EU group. Each EU in a fused EU group can be configured to execute a separate SIMD hardware thread. The number of EUs in a fused EU group can vary depending on the embodiment. Additionally, various widths of SIMD can be executed per EU, including but not limited to SIMD8, SIMD16, and SIMD32. Each fused graphics execution unit 509A-509N includes at least two execution units. For example, fused execution unit 509A includes a first EU 508A, a second EU 508B, and thread control logic 507A common to the first EU 508A and the second EU 508B. The thread control logic 507A controls the threads executed in the fused graphic execution unit 509A, and enables each EU in the fused execution units 509A-509N to execute using a common instruction pointer register.
[0117] One or more internal instruction caches (e.g., 506) are included in the thread execution logic 500 to cache thread instructions for the execution units. In some embodiments, one or more data caches (e.g., 512) are included to cache thread data during thread execution. Threads executing on the execution logic 500 may also store explicitly managed data in shared local memory 511. In some embodiments, a sampler 510 is included to provide texture sampling for 3D operations and media sampling for media operations. In some embodiments, the sampler 510 includes specialized texture or media sampling functions to process texture or media data during the sampling process before providing the sampled data to the execution units.
[0118] During execution, the graphics and media pipelines send thread start requests to the thread execution logic 500 via thread creation and dispatch logic. Once a group of geometry objects has been processed and rasterized into pixel data, pixel processor logic (e.g., pixel shader logic, fragment shader logic, etc.) in the shader processor 502 is invoked to further compute output information and cause the results to be written to an output surface (e.g., color buffer, depth buffer, stencil buffer, etc.). In some embodiments, the pixel shader or fragment shader computes values for various vertex attributes to be interpolated across the rasterized object. In some embodiments, the pixel processor logic in the shader processor 502 then executes pixel or fragment shader programs provided by an application programming interface (API). To execute the shader programs, the shader processor 502 dispatches threads to the execution units (e.g., 508A) via the thread dispatcher 504. In some embodiments, the shader processor 502 uses texture sampling logic in the sampler 510 to access texture data in texture maps stored in memory. Arithmetic operations on the texture data and the input geometry data compute pixel color data for each geometry fragment or discard one or more pixels from further processing.
[0119] In some embodiments, the data port 514 provides a memory access mechanism for the thread execution logic 500 to output processed data to memory for further processing on the output pipeline of the graphics processor. In some embodiments, the data port 514 includes or is coupled to one or more cache memories (e.g., data cache 512) for caching data for memory access via the data port.
[0120] In one embodiment, the execution logic 500 may also include a ray tracer 505 that may provide ray tracing acceleration functionality. The ray tracer 505 may support a ray tracing instruction set that includes instructions / functions for ray generation. The ray tracing instruction set may be similar to or different from the ray tracing instruction set supported by the ray tracing core 245 of FIG. 2C.
[0121] 5B illustrates exemplary internal details of an execution unit 508 according to an embodiment. The graphics execution unit 508 may include an instruction fetch unit 537, a general register file array (GRF) 524, an architectural register file array (ARF) 526, a thread arbiter 522, a send unit 530, a branch unit 532, a set of SIMD floating point units (FPUs) 534, and, in one embodiment, a set of dedicated integer SIMD ALUs 535. The GRF 524 and ARF 526 include a general register file and a set of architectural register files associated with each concurrent hardware thread that may be active in the graphics execution unit 508. In one embodiment, architectural state for each thread is maintained in the ARF 526, while data used during thread execution is stored in the GRF 524. The execution state of each thread, including each thread's instruction pointer, may be held in thread-specific registers within the ARF 526.
[0122] In one embodiment, the graphics execution unit 508 has an architecture that is a combination of simultaneous multi-threading (SMT) and fine-grained interleaved multi-threading (IMT) with a modularity that can be tuned at design time based on the target number of simultaneous threads and number of registers per execution unit, where the resources of the execution unit are divided across the logic used to execute multiple simultaneous threads. Also, the number of logical threads that can be executed by the graphics execution unit 508 is not limited to the number of hardware threads, and multiple logical threads can be assigned to each hardware thread.
[0123] In one embodiment, the graphics execution unit 508 can jointly issue multiple instructions, each of which can be a different instruction. The thread arbiter 522 of the graphics execution unit threads 508 can dispatch these instructions to either the transmit unit 530, the branch unit 532, or the SIMD FPU(s) 534 for execution. Each execution thread can access 128 general purpose registers in the GRF 524, where each register can store 32 bytes accessible as a SIMD 8-element vector of 32-bit data elements. In one embodiment, each execution unit thread accesses 4K bytes in the GRF 524, although the embodiment is not so limited and in other embodiments, greater or less register resources may be provided. In one embodiment, the graphics execution unit 508 is divided into seven hardware threads that can independently perform computational operations, although the number of threads per execution unit can also vary depending on the embodiment. For example, in one embodiment, up to 16 hardware threads are supported. In an embodiment in which the seven threads can access 4K bytes, the GRF 524 can store a total of 28K bytes. If 16 threads can access 4K bytes, then the GRF 524 can store a total of 64K bytes. In flexible addressing mode, registers can be addressed together to effectively build wide registers or to represent stride-rectangle block data structures.
[0124] In one embodiment, memory operations, sampler operations, and other longer latency system communications are dispatched via "send" instructions executed by a message passing send unit 530. In one embodiment, branch instructions are dispatched to a dedicated branch unit 532 to facilitate SIMD divergence and eventual convergence.
[0125] In one embodiment, the graphics execution unit 508 includes one or more SIMD floating point units (FPU(s)) 534 for performing floating point operations. In one embodiment, the FPU(s) 534 also support integer operations. In one embodiment, the FPU(s) 534 can SIMD execute up to M 32-bit floating point (or integer) operations, or SIMD execute up to 2M 16-bit integer or 16-bit floating point operations. In one embodiment, at least one of the FPU(s) provides extended math capabilities to support high throughput transcendental math functions and 54-bit double precision floating point. In some embodiments, a set of 8-bit integer SIMD ALUs 535 are also present and may be specifically optimized for performing operations associated with machine learning computations.
[0126] In one embodiment, an array of multiple instances of the graphics execution unit 508 can be instantiated in a graphics sub-core grouping (e.g., a sub-slice). For scalability, the product architect can select the exact number of execution units per sub-core grouping. In one embodiment, the execution unit 508 can execute instructions across multiple execution channels. In a further embodiment, each thread executing on the graphics execution unit 508 executes on a different channel.
[0127] FIG. 6 illustrates an additional execution unit 600 according to one embodiment. The execution unit 600 may be a compute-optimized execution unit for use in the compute engine tiles 340A-340D of FIG. 3C, for example, but not limited to, the graphics engine tiles 310A-310D of FIG. 3B. In one embodiment, the execution unit 600 includes a thread control unit 601, a thread state unit 602, an instruction fetch / prefetch unit 603, and an instruction decode unit 604. The execution unit 600 additionally includes a register file 606 that stores registers that can be allocated to hardware threads within the execution unit. The execution unit 600 additionally includes a send unit 607 and a branch unit 608. In one embodiment, the send unit 607 and the branch unit 608 may operate similarly to the send unit 530 and the branch unit 532 of the graphics execution unit 508 of FIG. 5B.
[0128] The execution unit 600 also includes a computation unit 610 that includes multiple different types of functional units. In one embodiment, the computation unit 610 includes an ALU unit 611 that includes an array of arithmetic logic units. The ALU unit 611 may be configured to perform 64-bit, 32-bit, and 16-bit integer and floating point operations. The integer and floating point operations may be performed simultaneously. The computation unit 610 may also include a systolic array 612, and a math unit 613. The systolic array 612 includes a width W and depth D network of data processing units that can be used to systolicly perform vector or other data parallel operations. In one embodiment, the systolic array 612 may be configured to perform matrix operations, such as matrix dot product operations. In one embodiment, the systolic array 612 supports 16-bit floating point operations, as well as 8-bit and 4-bit integer operations. In one embodiment, the systolic array 612 may be configured to accelerate machine learning operations. In such an embodiment, systolic array 612 may be configured to support the bfloat 16-bit floating point format. In one embodiment, math unit 613 may be included to perform a particular subset of arithmetic operations more efficiently and with less power consumption than subsequent ALU unit 611. Math unit 613 may include variations of math logic that may be found in shared function logic of graphics processing engines provided by other embodiments (e.g., math logic 422 of shared function logic 420 of FIG. 4). In one embodiment, math unit 613 may be configured to perform 32-bit and 64-bit floating point operations.
[0129] The thread control unit 601 includes logic for controlling the execution of threads in the execution units. The thread control unit 601 may include thread arbitration logic for starting, stopping, and pre-empting the execution of threads in the execution units 600. The thread state unit 602 may be used to store thread states for threads assigned to execute on the execution units 600. Storing thread states in the execution units 600 allows for rapid pre-emption of threads when they become blocked or idle. The instruction fetch / pre-fetch unit 603 may fetch instructions from an instruction cache of higher level execution logic (e.g., instruction cache 506 as shown in FIG. 5A). The instruction fetch / pre-fetch unit 603 may also issue pre-fetch requests for instructions to be loaded into the instruction cache based on an analysis of currently executing threads. The instruction decode unit 604 may be used to decode instructions to be executed by the computation units. In one embodiment, the instruction decode unit 604 may be used as a secondary decoder to decode complex instructions into micro-operation configurations.
[0130] Execution unit 600 additionally includes a register file 606 that can be used by hardware threads executing on execution unit 600. The registers in register file 606 can be divided across logic used to execute multiple simultaneous threads within compute units 610 of execution unit 600. The number of logical threads that can be executed by graphics execution unit 600 is not limited to the number of hardware threads, and multiple logical threads can be assigned to each hardware thread. The size of register file 606 can vary across embodiments based on the number of hardware threads supported. In one embodiment, register renaming may be used to dynamically allocate registers to hardware threads.
[0131] 7 is a block diagram illustrating a graphics processor instruction format 700 according to some embodiments. In one or more embodiments, a graphics processor execution unit supports an instruction set having instructions in multiple formats. The solid lined boxes indicate components that are typically included in the execution unit's instructions, while the dashed lines include components that are optional or are included in only a subset of the instructions. In some embodiments, the instruction format 700 described and illustrated is a macro instruction, in that it is the instructions provided to the execution unit as opposed to micro operations that result from instruction decode once the instruction is processed.
[0132] In some embodiments, the graphics processor execution units natively support instructions in a 128-bit instruction format 710. A 64-bit compressed instruction format 730 is available for some instructions based on the selected instruction, instruction options, and number of operands. The 128-bit native instruction format 710 provides access to all instruction options, while some options and operations are restricted in the 64-bit format 730. The native instructions available in the 64-bit format 730 vary depending on the embodiment. In some embodiments, the instructions are partially compressed using a set of index values in the index field 713. The execution unit hardware references a set of compression tables based on the index values and uses the output of the compression tables to reconstruct the native instruction in the 128-bit instruction format 710. Instructions of other sizes and formats may be used.
[0133] For each format, the instruction opcode 712 defines the operation that the execution unit performs. The execution unit executes each instruction in parallel across multiple data elements of each operand. For example, in response to an add instruction, the execution unit performs an add operation simultaneously across each color channel representing a texture or picture element. By default, the execution unit executes each instruction across all data channels of an operand. In some embodiments, the instruction control field 714 allows control over certain execution options such as channel selection (e.g., predicate) and data channel order (e.g., swizzle). For instructions in the 128-bit instruction format 710, the exec-size field 716 limits the number of data channels that are executed in parallel. In some embodiments, the exec-size field 716 is not available for use with the 64-bit compact instruction format 730.
[0134] Some execution unit instructions have up to three operands, including two source operands, src0 720 and src1 722, and one destination 718. In some embodiments, an execution unit supports two destination instructions, where one of the destinations is implied. Data manipulation instructions may have a third source operand (e.g., SRC2 724), where the instruction opcode 712 determines the number of source operands. The last source operand of an instruction may be an immediate (e.g., hard-coded) value passed with the instruction.
[0135] In some embodiments, the 128-bit instruction format 710 includes an access / address mode field 726 that specifies, for example, whether a direct register addressing mode or an indirect register addressing mode is used. If a direct register addressing mode is used, the register addresses of one or more operands are provided directly by bits in the instruction.
[0136] In some embodiments, the 128-bit instruction format 710 includes an access / address mode field 726 that specifies an address mode and / or access mode for the instruction. In one embodiment, the access mode is used to define the data access alignment for the instruction. Some embodiments support access modes including a 16-byte aligned access mode and a 1-byte aligned access mode, where the byte alignment of the access mode determines the access alignment of the instruction operands. For example, in a first mode, the instruction may use byte-aligned addressing for source and destination operands, and in a second mode, the instruction may use 16-byte aligned addressing for all source and destination operands.
[0137] In one embodiment, the address mode portion of the access / address mode field 726 determines whether the instruction uses direct or indirect addressing. If the direct register addressing mode is used, bits in the instruction directly provide the register address of one or more operands. If the indirect register addressing mode is used, the register address of one or more operands may be calculated based on an address register value and an address immediate field in the instruction.
[0138] In some embodiments, instructions are grouped based on bit fields of the instruction opcode 712 to simplify opcode decode 740. For an 8-bit opcode, bits 4, 5, and 6 allow the execution unit to determine the type of opcode. The opcode grouping with the precision shown is merely exemplary. In some embodiments, the move and logical opcode group 742 includes data movement and logical instructions (e.g., move, cmp). In some embodiments, the move and logical opcode group 742 shares five most significant bits (MSBs), where move instructions are of the form 0000xxxxb and logical instructions are of the form 0001xxxxb. The flow control instructions group 744 (e.g., call, jump) includes instructions of the form 0010xxxxb (e.g., 0x20). Miscellaneous instruction group 746 includes a combination of instructions including synchronization instructions (e.g., Wait, send) in the format of 0011xxxxb (e.g., 0x30). Parallel arithmetic instruction group 748 includes component-wise arithmetic instructions (e.g., add, multiply (mul)) in the format of 0100xxxxb (e.g., 0x40). Parallel arithmetic group 748 performs arithmetic operations in parallel across data channels. Vector arithmetic group 750 includes arithmetic instructions (e.g., dp4) in the format of 0101xxxxb (e.g., 0x50). Vector arithmetic group performs arithmetic such as dot product calculations on vector operands. In one embodiment, the illustrated opcode decode 740 can be used to determine which portion of the execution unit is used to execute the decoded instruction. For example, some instructions may be designated as systolic instructions that are executed by a systolic array. Other instructions, such as ray tracing instructions (not shown), may be routed to a ray tracing core or ray tracing logic within a slice or partition of the execution logic. Graphics Pipeline
[0139] Figure 8 is a block diagram of another embodiment of a graphics processor 800. Elements in Figure 8 having the same reference numbers (or names) as elements in any other figure herein may operate or function in any manner similar to, but not limited to, that described elsewhere herein.
[0140] In some embodiments, graphics processor 800 includes a geometry pipeline 820, a media pipeline 830, a display engine 840, thread execution logic 850, and a rendering output pipeline 870. In some embodiments, graphics processor 800 is a graphics processor in a multi-core processing system that includes one or more general-purpose processing cores. The graphics processor is controlled by register writes to one or more control registers (not shown) or via commands issued to graphics processor 800 via a ring interconnect 802. In some embodiments, ring interconnect 802 couples graphics processor 800 to other processing components, such as other graphics processors or general-purpose processors. Commands from ring interconnect 802 are interpreted by command streamer 803, which supplies instructions to individual components of geometry pipeline 820 or media pipeline 830.
[0141] In some embodiments, command streamer 803 directs the operation of vertex fetcher 805, which reads vertex data from memory and executes vertex processing commands provided by command streamer 803. In some embodiments, vertex fetcher 805 provides the vertex data to vertex shader 807, which performs coordinate space transformations and lighting operations on each vertex. In some embodiments, vertex fetcher 805 and vertex shader 807 execute the vertex processing instructions by dispatching execution threads to execution units 852a-852b via thread dispatcher 831.
[0142] In some embodiments, execution units 852A and 852B are arrays of vector processors with instruction sets for performing graphics and media operations. In some embodiments, execution units 852A and 852B have an associated L1 cache 851 that is unique to each array or shared between the arrays. The cache can be configured as a data cache, an instruction cache, or a single cache that is partitioned to contain data and instructions in different partitions.
[0143] In some embodiments, the geometry pipeline 820 includes a tessellation component for performing hardware accelerated tessellation of 3D objects. In some embodiments, the programmable hull shader 811 configures the tessellation operations. The programmable domain shader 817 provides back-end evaluation of the tessellation output. The tessellator 813 operates at the direction of the hull shader 811 and includes special purpose logic for generating a set of detailed geometric objects based on a coarse geometric model provided as input to the geometry pipeline 820. In some embodiments, the tessellation components (e.g., the hull shader 811, the tessellator 813, and the domain shader 817) can be bypassed if tessellation is not used.
[0144] In some embodiments, a complete geometry object can be processed by the geometry shader 819 via one or more threads dispatched to execution units 852A and 852B, or can proceed directly to the clipper 829. In some embodiments, the geometry shader operates on entire geometry objects, rather than vertices or patches of vertices like previous stages of the graphics pipeline. If tessellation is disabled, the geometry shader 819 receives input from the vertex shader 807. In some embodiments, the geometry shader 819 is programmable by the geometry shader program to perform geometry tessellation if the tessellation unit is disabled.
[0145] A clipper 829 processes the vertex data before rasterization. The clipper 829 may be a fixed function clipper or a programmable clipper with clipping and geometry shader functionality. In some embodiments, a rasterizer and depth test component 873 in the rendering output pipeline 870 dispatches pixel shaders to convert geometry objects into pixel-by-pixel representations. In some embodiments, pixel shader logic is included in the thread execution logic 850. In some embodiments, an application can bypass the rasterizer and depth test component 873 to access unrasterized vertex data via the stream output unit 823.
[0146] Graphics processor 800 has an interconnect bus, interconnect fabric, or some other interconnect mechanism that allows data and message passing between major components of the processor. In some embodiments, execution units 852A and 852B and associated logic units (e.g., L1 cache 851, sampler 854, texture cache 858, etc.) interconnect via data port 856 to perform memory accesses and communicate with components of the processor's rendering output pipeline. In some embodiments, sampler 854, caches 851 and 858, and execution units 852A and 852B each have a separate memory access path. In one embodiment, texture cache 858 can also be configured as a sampler cache.
[0147] In some embodiments, the render output pipeline 870 includes a rasterizer and depth test component 873 that converts vertex-based objects into an associated pixel-based representation. In some embodiments, the rasterizer logic includes a windower / masker unit for performing fixed function triangle and line rasterization. In some embodiments, an associated render cache 878 and depth cache 879 are also available. A pixel operations component 877 performs pixel-based operations on the data, although in some instances pixel operations associated with 2D operations (e.g., bit block image transfer with blending) are performed by the 2D engine 841 or substituted at display time by the display controller 843 using overlay display planes. In some embodiments, a shared L3 cache 875 is available to all graphics components, allowing sharing of data without using main system memory.
[0148] In some embodiments, graphics processor media pipeline 830 includes a media engine 837 and a video front end 834. In some embodiments, video front end 834 receives pipeline commands from command streamer 803. In some embodiments, media pipeline 830 includes a separate command streamer. In some embodiments, video front end 834 processes media commands before sending the commands to media engine 837. In some embodiments, media engine 837 includes a thread creation function to create threads for dispatching to thread execution logic 850 via thread dispatcher 831.
[0149] In some embodiments, graphics processor 800 includes a display engine 840. In some embodiments, display engine 840 is external to graphics processor 800 and couples to the graphics processor via ring interconnect 802, or some other interconnect bus or fabric. In some embodiments, display engine 840 includes a 2D engine 841 and a display controller 843. In some embodiments, display engine 840 includes special purpose logic operable independently of the 3D pipeline. In some embodiments, display controller 843 couples to a display device (not shown), which may be a system integrated display device, such as a laptop computer, or an external display device attached via a display device connector.
[0150] In some embodiments, the geometry pipeline 820 and the media pipeline 830 are configurable to perform operations based on multiple graphics and media programming interfaces and are not specific to any one application programming interface (API). In some embodiments, driver software for the graphics processor translates calls to APIs specific to a particular graphics or media library into commands that can be processed by the graphics processor. In some embodiments, support is provided for all of the Open Graphics Library (OpenGL), Open Computing Language (OpenCL), and / or Vulkan graphics and compute APIs from the Khronos Group. In some embodiments, support may also be provided for the Direct3D library from Microsoft Corporation. In some embodiments, a combination of these libraries may be supported. Also, support may be provided for the Open Source Computer Vision Library (OpenCV). Future APIs with compatible 3D pipelines will also be supported if a mapping from the pipeline of the future API to the pipeline of the graphics processor is possible. Graphics Pipeline Programming
[0151] Figure 9A is a block diagram illustrating a graphics processor command format 900 according to some embodiments. Figure 9B is a block diagram illustrating a graphics processor command sequence 910 according to one embodiment. The solid lined boxes in Figure 9A indicate components that are typically included in a graphics command, while the dashed lines include components that are optional or are included in only a subset of the graphics commands. The example graphics processor command format 900 of Figure 9A includes data fields to identify a client 902, a command operation code (opcode) 904, and data for the command 906. Sub-opcodes 905 and a command size 908 are also included in some commands.
[0152] In some embodiments, the client 902 specifies a client unit of the graphics device that will process the command data. In some embodiments, a command parser of the graphics processor examines the client field of each command to condition further processing of the command and route the command data to the appropriate client unit. In some embodiments, the client units of the graphics processor include a memory interface unit, a rendering unit, a 2D unit, a 3D unit, and a media unit. Each client unit has a corresponding processing pipeline that processes the command. Once a command is received by a client unit, the client unit reads the opcode 904 and, if present, the sub-opcode 905 to determine the operation to perform. The client unit executes the command using the information in the data field 906. For some commands, an explicit command size 908 is expected to specify the size of the command. In some embodiments, the command parser automatically determines the size of at least a portion of the command based on the command opcode. In some embodiments, the commands are aligned via multiples of double words. Other command formats can be used.
[0153] The flow diagram of Figure 9B illustrates an exemplary graphics processor command sequence 910. In some embodiments, software or firmware of a data processing system featuring an embodiment of a graphics processor uses versions of the illustrated command sequence to set up, execute, and terminate a set of graphics operations. The sample command sequence is shown and described for illustrative purposes only, as embodiments are not limited to these particular commands or this command sequence. Furthermore, these commands may be issued as a batch of commands in the command sequence such that the graphics processor processes the command sequence at least partially in unison.
[0154] In some embodiments, the graphics processor command sequence 910 may begin with a pipeline flush command 912 to cause any active graphics pipelines to complete any currently pending commands for the pipeline. In some embodiments, the 3D pipeline 922 and the media pipeline 924 do not operate simultaneously. The pipeline flush is executed to cause the active graphics pipelines to complete any currently pending commands. In response to the pipeline flush, the graphics processor's command parser pauses command processing until the active drawing engines complete pending operations and the associated read caches are invalidated. Optionally, any data in the render cache that is marked as "dirty" may be flushed to memory. In some embodiments, the pipeline flush command 912 may be used for pipeline synchronization or before placing the graphics processor into a low power state.
[0155] In some embodiments, the pipeline select command 913 is used when a command sequence requests the graphics processor to explicitly switch between pipelines. In some embodiments, the pipeline select command 913 is requested only once within an execution context before issuing commands for a pipeline, unless the context plans to issue commands for both pipelines. In some embodiments, a pipeline flush command 912 is requested immediately prior to a pipeline switch via the pipeline select command 913.
[0156] In some embodiments, pipeline control commands 914 are used to configure the graphics pipeline for operations and to program the 3D pipeline 922 and the media pipeline 924. In some embodiments, pipeline control commands 914 configure the pipeline state for active pipelines. In one embodiment, pipeline control commands 914 are used for pipeline synchronization and to clear data from one or more cache memories in an active pipeline before processing a batch of commands.
[0157] In some embodiments, the return buffer state command 916 is used to configure a set of return buffers for each pipeline to write data to. Some pipeline operations require the allocation, selection, or configuration of one or more return buffers to which the operations write intermediate data during processing. In some embodiments, the graphics processor also uses one or more return buffers to store output data and perform cross-thread communication. In some embodiments, the return buffer state command 916 includes selecting the size and number of return buffers to use for a set of pipeline operations.
[0158] The remaining commands in the command sequence differ based on the active pipeline for the operation. Based on the pipeline decision 920, the command sequence is tailored for the 3D pipeline 922, which begins at the 3D pipeline state 930, or the media pipeline 924, which begins at the media pipeline state 940.
[0159] Commands for configuring the 3D pipeline state 930 include 3D state setting commands for vertex buffer states, vertex element states, certain color states, depth buffer states, and other state variables that should be configured before 3D primitive commands are processed. The values of these commands are determined at least in part based on the particular 3D API in use. In some embodiments, the 3D pipeline state 930 commands can also selectively disable or bypass certain pipeline elements if those particular pipeline elements are not used.
[0160] In some embodiments, 3D primitive 932 commands are used to submit 3D primitives to be processed by the 3D pipeline. Commands and associated parameters passed to the graphics processor via 3D primitive 932 commands are forwarded to a vertex fetch function in the graphics pipeline. The vertex fetch function uses the 3D primitive 932 command data to generate a vertex data structure. The vertex data structure is stored in one or more return buffers. In some embodiments, the 3D primitive 932 commands are used to perform vertex operations on the 3D primitives via vertex shaders. To process the vertex shaders, the 3D pipeline 922 dispatches shader execution threads to the graphics processor execution units.
[0161] In some embodiments, the 3D pipeline 922 is triggered via an execution 934 command or event. In some embodiments, a register write triggers the execution of the command. In some embodiments, the execution is triggered via a "go" or "kick" command in the command sequence. In one embodiment, the execution of the command is triggered using a pipeline synchronization command to flush the command sequence through the graphics pipeline. The 3D pipeline performs geometry processing for the 3D primitives. Once the operations are complete, the resulting geometry objects are rasterized and the pixel engine colors the resulting pixels. Additional commands to control pixel shading and back-end operations of the pixels may also be included for these operations.
[0162] In some embodiments, the graphics processor command sequence 910 follows the path of the media pipeline 924 in performing media operations. In general, the specific use and programming aspects for the media pipeline 924 depend on the media or computational operations being performed. Certain media decode operations may be offloaded to the media pipeline during media decoding. In some embodiments, the media pipeline may also be bypassed and media decoding may be performed in whole or in part using resources provided by one or more general purpose processing cores. In one embodiment, the media pipeline also includes elements for general purpose graphics processor unit (GPGPU) operations, where the graphics processor is used to perform SIMD vector operations using computational shader programs that are not explicitly related to rendering graphics primitives.
[0163] In some embodiments, the media pipeline 924 is configured in a similar manner to the 3D pipeline 922. A set of commands to configure the media pipeline state 940 is dispatched or placed in a queued command before the media object commands 942. In some embodiments, the commands to configure the media pipeline state 940 include data to configure elements of the media pipeline used to process media objects. This includes data to configure video decode and video encode logic in the media pipeline, such as encode or decode formats. In some embodiments, the commands for the media pipeline state 940 support the use of one or more pointers to "indirect" state elements that contain a batch of state settings.
[0164] In some embodiments, the media object command 942 provides a pointer to a media object for processing by the media pipeline. A media object includes a memory buffer that contains video data to be processed. In some embodiments, all media pipeline state must be valid before issuing a media object command 942. Once the pipeline state is configured and the media object command 942 is queued, the media pipeline 924 is triggered via an execute command 944 or equivalent execute event (e.g., register write). The output from the media pipeline 924 may then be post-processed by operations provided by the 3D pipeline 922 or the media pipeline 924. In some embodiments, GPGPU operations are configured and executed in a similar manner to the media operations. Graphics Software Architecture
[0165] 10 illustrates an exemplary graphics software architecture for a data processing system 1000 according to some embodiments. In some embodiments, the software architecture includes a 3D graphics application 1010, an operating system 1020, and at least one processor 1030. In some embodiments, the processor 1030 includes a graphics processor 1032 and one or more general-purpose processor cores 1034. The graphics application 1010 and the operating system 1020 each execute within the system memory 1050 of the data processing system.
[0166] In some embodiments, the 3D graphics application 1010 includes one or more shader programs that include shader instructions 1012. The shader language instructions may be in a high-level shader language, such as Direct3D High Level Shader Language (HLSL) or OpenGL Shader Language (GLSL). The application also includes a number of executable instructions 1014 in a machine language suitable for execution by a general-purpose processor core 1034. The application also includes a number of graphics objects 1016 defined by vertex data.
[0167] In some embodiments, the operating system 1020 may be the Microsoft Windows operating system available from Microsoft Corporation, a proprietary UNIX-style operating system, or an open source UNIX-style operating system that uses a variant of the Linux kernel. The operating system 1020 may support a graphics API 1022, such as the Direct3D API, the OpenGL API, or the Vulkan API. If the Direct3D API is used, the operating system 1020 uses a front-end shader compiler 1024 to compile any shader instructions 1012 in HLSL into a low-level shader language. The compilation may be a just-in-time (JIT) compilation, or the application may perform shader pre-compilation. In some embodiments, the high-level shaders are compiled into low-level shaders during compilation of the 3D graphics application 1010. In some embodiments, the shader instructions 1012 are provided in an intermediate format, such as a version of the Standard Portable Intermediate Representation (SPIR) used by the Vulkan API.
[0168] In some embodiments, the user mode graphics driver 1026 includes a back-end shader compiler 1027 for converting the shader instructions 1012 into a hardware-specific representation. If the OpenGL API is used, the shader instructions 1012 in the GLSL high-level language are passed to the user mode graphics driver 1026 for compilation. In some embodiments, the user mode graphics driver 1026 communicates with a kernel mode graphics driver 1029 using operating system kernel mode functions 1028. In some embodiments, the kernel mode graphics driver 1029 communicates with a graphics processor 1032 to dispatch commands and instructions. IP Core Implementation
[0169] One or more aspects of at least one embodiment may be implemented by representative code stored on a machine-readable medium that represents and / or defines logic within an integrated circuit, such as a processor. For example, the machine-readable medium may include instructions that represent various logic within a processor. The instructions, when read by a machine, may cause the machine to manufacture logic that performs the techniques described herein. Such representations, known as "IP cores," are reusable units of logic for integrated circuits that may be stored on a tangible machine-readable medium as a hardware model that describes the structure of the integrated circuit. The hardware model may be provided to various customers or manufacturing facilities, which load the hardware model into manufacturing equipment that produces the integrated circuit. The integrated circuit may be manufactured such that the circuit performs the operations described in connection with any of the embodiments described herein.
[0170] FIG. 11A is a block diagram illustrating an IP core development system 1100 that may be used to fabricate an integrated circuit for performing operations according to one embodiment. The IP core development system 1100 may be used to generate modular, reusable designs that can be incorporated into a larger design or to build an entire integrated circuit (e.g., a SOC integrated circuit). The design facility 1130 may generate a software simulation 1110 of the IP core design in a high-level programming language (e.g., C / C++). The software simulation 1110 may be used to design, test, and verify the operation of the IP core using a simulation model 1112. The simulation model 1112 may include functional simulation, behavioral simulation, and / or timing simulation. A register transfer level (RTL) design 1115 may then be created or synthesized from the simulation model 1112. The RTL design 1115 is an abstraction of the operation of the integrated circuit, including modeling the flow of digital signals between hardware registers and associated logic that is executed using the modeled digital signals. In addition to the RTL design 1115, lower level designs at the logic or transistor level may also be created, designed, or synthesized. Thus, the specific details of the initial design and simulation may vary.
[0171] Further, the RTL design 1115 or equivalent may be synthesized by a design facility into a hardware model 1120, which may be a hardware description language (HDL) or some other representation of physical design data. The HDL may be further simulated or tested to verify the IP core design. The IP core design may be stored using a non-volatile memory 1140 (e.g., a hard disk, a flash memory, or any non-volatile storage medium) for delivery to a third-party manufacturing facility 1165. Alternatively, the IP core design may be transmitted via a wired connection 1150 or a wireless connection 1160 (e.g., via the Internet). The manufacturing facility 1165 may then manufacture an integrated circuit based at least in part on the IP core design. The manufactured integrated circuit may be configured to perform operations according to at least one embodiment described herein.
[0172] 11B shows a cross-sectional side view of an integrated circuit package assembly 1170 according to some embodiments described herein. The integrated circuit package assembly 1170 illustrates an implementation of one or more processors or accelerator devices as described herein. The package assembly 1170 includes multiple hardware units of logic 1172 and 1174 coupled to a substrate 1180. The logic 1172 and 1174 may be at least partially implemented in configurable logic hardware or fixed function logic hardware and may include one or more portions of any of the processor core(s), graphics processor(s), or other accelerator devices described herein. Each unit of logic 1172 and 1174 may be implemented in a semiconductor die and coupled to the substrate 1180 via an interconnect structure 1173. The interconnect structure 1173 may be configured to route electrical signals between the logic 1172 and 1174 and the substrate 1180 and may include interconnects such as, for example, but not limited to, steps or posts. In some embodiments, the interconnect structure 1173 may be configured to route electrical signals, such as, for example, input / output (I / O) signals and / or power or ground signals associated with the operation of the logic 1172 and 1174. In some embodiments, the substrate 1180 is an epoxy-based laminate substrate. In other embodiments, the substrate 1180 may include other suitable types of substrates. The package assembly 1170 may be connected to other electrical devices via the package interconnect 1183. The package interconnect 1183 may be coupled to a surface of the substrate 1180 for routing electrical signals to other electrical devices, such as a motherboard, other chipset, or multi-chip module.
[0173] In some embodiments, the units of logic 1172 and 1174 are electrically coupled to a bridge 1182 configured to route electrical signals between the logic 1172 and 1174. The bridge 1182 may be a dense interconnect structure that provides routes for the electrical signals. The bridge 1182 may include a bridge substrate constructed of glass or a suitable semiconductor material. Electrical routing features can be formed on the bridge substrate to provide chip-to-chip connections between the logic 1172 and 1174.
[0174] Although two units of logic 1172 and 1174 and bridge 1182 are illustrated, the embodiments described herein may include more or fewer logic units on one or more dies. If the logic is included in a single die, bridge 1182 may be omitted, and thus one or more dies may be connected by zero or more bridges. Alternatively, multiple dies or logic units can be connected by one or more bridges. Additionally, multiple logic units, dies, and bridges can be connected together in other possible configurations, including three-dimensional configurations.
[0175] FIG. 11C illustrates a package assembly 1190 that includes multiple units of hardware logic chiplets connected to a substrate 1180 (e.g., a base die). Graphics processing units, parallel processors, and / or computational accelerators as described herein can be composed of various silicon chiplets that are individually manufactured. In this context, a chiplet is an at least partially packaged integrated circuit that includes separate logic units that can be assembled with other chiplets into a larger package. A set of various chiplets with different IP core logic can be assembled into a single device. In addition, active interposer technology can be used to integrate chiplets into a base die or base chiplet. The concepts described herein enable interconnection and communication between different forms of IP in a GPU. IP cores can be manufactured using different process technologies and configured during manufacturing, which avoids the complexity of converging multiple IPs into the same manufacturing process, especially in large SoCs with several flavors of IP. Enabling the use of multiple process technologies improves time to market and provides a cost-effective way to create multiple product SKUs. Additionally, the decomposed IP is susceptible to being independently power-gated, allowing components not being used in a given workload to be powered off, reducing overall power consumption.
[0176] The hardware logic chiplets can include special purpose hardware logic chiplets 1172, logic or I / O chiplets 1174, and / or memory chiplets 1175. The hardware logic chiplets 1172 and logic or I / O chiplets 1174 can be implemented at least in part in configurable logic or fixed function logic hardware and can include one or more portions of any of the processor core(s), graphics processor(s), parallel processors, or other accelerator devices described herein. The memory chiplets 1175 can be DRAM (e.g., GDDR, HBM) memory or cache (SRAM) memory.
[0177] Each chiplet can be fabricated as a separate semiconductor die and coupled to the substrate 1180 via interconnect structures 1173. The interconnect structures 1173 may be configured to route electrical signals between the various chiplets and logic within the substrate 1180. The interconnect structures 1173 may include interconnects such as, but are not limited to, bumps or posts. In some embodiments, the interconnect structures 1173 may be configured to route electrical signals, such as input / output (I / O) signals and / or power or ground signals associated with operation of the logic, I / O and memory chiplets, for example.
[0178] In some embodiments, the substrate 1180 is an epoxy-based laminate substrate. In other embodiments, the substrate 1180 may include other suitable types of substrates. The package assembly 1190 can be connected to other electrical devices via the package interconnect 1183. The package interconnect 1183 may be coupled to a surface of the substrate 1180 for routing electrical signals to other electrical devices, such as a motherboard, other chipset, or multi-chip module.
[0179] In some embodiments, the logic or I / O chiplets 1174 and the memory chiplets 1175 may be electrically coupled through a bridge 1187 configured to route electrical signals between the logic or I / O chiplets 1174 and the memory chiplets 1175. The bridge 1187 may be a dense interconnect structure that provides a route for the electrical signals. The bridge 1187 may include a bridge substrate constructed of glass or a suitable semiconductor material. Electrical routing features may be formed on the bridge substrate to provide chip-to-chip connections between the logic or I / O chiplets 1174 and the memory chiplets 1175. The bridge 1187 may be referred to as a silicon bridge or an interconnect bridge. For example, in some embodiments, the bridge 1187 is an embedded multi-die interconnect bridge (EMIB). In some embodiments, the bridge 1187 may simply be a direct connection from one chiplet to another.
[0180] Substrate 1180 can include hardware components for I / O 1191, cache memory 1192, and other hardware logic 1193. Fabric 1185 is embedded within substrate 1180 to enable communication between various logic chiplets within substrate 1180 and logic 1191 and 1193. In one embodiment, I / O 1191, fabric 1185, cache, bridges, and other hardware logic 1193 can be integrated into a base die stacked on top of substrate 1180.
[0181] In various embodiments, the package assembly 1190 can include fewer or more components and chiplets interconnected by the fabric 1185 or one or more bridges 1187. The chiplets in the package assembly 1190 may be arranged in a 3D or 2.5D array. In general, the bridge structures 1187 may be used to facilitate point-to-point interconnections between, for example, logic or I / O chiplets and memory chiplets. The fabric 1185 may be used to interconnect various logic and / or I / O chiplets (e.g., chiplets 1172, 1174, 1191, and 1193) with other logic and / or I / O chiplets. In one embodiment, the cache memory 1192 in the substrate can function as a global cache for the package assembly 1190, part of a distributed global cache, or a dedicated cache for the fabric 1185.
[0182] FIG. 11D illustrates a package assembly 1194 including a replaceable chiplet 1195 according to one embodiment. The replaceable chiplet 1195 can be assembled into a standardized slot on one or more base chiplets 1196 and 1198. The base chiplets 1196 and 1198 can be coupled via a bridge interconnect 1197, which can be similar to other bridge interconnects described herein, for example, an EMIB. The memory chiplets can also be connected to logic or I / O chiplets via the bridge interconnect. The I / O chiplets and logic chiplets can communicate via an interconnect fabric. The base chiplets can each support one or more slots of a standardized format for either logic or I / O or memory / cache.
[0183] In one embodiment, the SRAM and power delivery circuitry can be fabricated in one or more of the base chiplets 1196 and 1198, which can be fabricated using a different process technology than the replaceable chiplets 1195 that are stacked on top of the base chiplets. For example, the base chiplets 1196 and 1198 can be fabricated using a larger process technology, while the replaceable chiplets can be fabricated using a smaller process technology. One or more of the replaceable chiplets 1195 can be memory (e.g., DRAM) chiplets. Different memory densities can be selected for the package assembly 1194 based on the target power and / or performance for the product that uses the package assembly 1194. In addition, logic chiplets with different types and numbers of functional units can be selected at assembly time based on the power and / or target performance of the product. In addition, chiplets containing different types of IP logic cores can be inserted into the replaceable chiplet slots, enabling hybrid processor designs where IP blocks of different technologies can be mixed and matched. Exemplary System-on-Chip Integrated Circuits
[0184] 12, 13A, and 13B illustrate example integrated circuits and associated graphics processors that may be manufactured using one or more IP cores, in accordance with various embodiments described herein. In addition to what is illustrated, other logic and circuitry may be included, including additional graphics processors / cores, peripheral interface controllers, or general-purpose processor cores.
[0185] 12 is a block diagram illustrating an exemplary system-on-chip integrated circuit 1200 that may be manufactured using one or more IP cores according to one embodiment. The exemplary integrated circuit 1200 includes one or more application processors 1205 (e.g., CPUs) at least one graphics processor 1210, and may additionally include an image processor 1215 and / or a video processor 1220, any of which may be modular IP cores from the same or multiple different design facilities. The integrated circuit 1200 includes a USB controller 1225, a UART controller 1230, an SPI / SDIO controller 1235, and an I / O controller 1236. 2 S / I 2 The integrated circuit includes peripheral or bus logic including a C controller 1240. Additionally, the integrated circuit may include a display device 1245 coupled to one or more of a High-Definition Multimedia Interface (HDMI®) controller 1250 and a Mobile Industry Processor Interface (MIPI) display interface 1255. Storage may be provided by a flash memory subsystem 1260 including a flash memory and a flash memory controller. Memory interfacing may be provided via a memory controller 1265 for access to an SDRAM or SRAM memory device. A portion of the integrated circuit further includes an embedded security engine 1270.
[0186] 13A and 13B are block diagrams illustrating an example graphics processor for use in an SoC according to an embodiment described herein. FIG. 13A illustrates an example graphics processor 1310 of a system-on-chip integrated circuit that may be fabricated using one or more IP cores according to an embodiment. FIG. 13B illustrates an additional example graphics processor 1340 of a system-on-chip integrated circuit that may be fabricated using one or more IP cores according to an embodiment. The graphics processor 1310 of FIG. 13A is an example of a low power graphics processor core. The graphics processor 1340 of FIG. 13B is an example of a higher performance graphics processor core. Each of the graphics processors 1310 and 1340 may be a variation of the graphics processor 1210 of FIG. 12.
[0187] As shown in FIG. 13A, the graphics processor 1310 includes a vertex processor 1305 and one or more fragment processor(s) 1315A-1315N (e.g., 1315A, 1315B, 1315C, and 1315D, via 1315N-1 and 1315N). The one or more fragment processor(s) 1315A-1315N perform fragment (e.g., pixel) shading operations of a fragment or pixel shader program, while the graphics processor 1310 can execute different shader programs through separate logic such that the vertex processor 1305 is optimized to perform operations for the vertex shader programs. The vertex processor 1305 executes the vertex processing stage of the 3D graphics pipeline and generates primitive and vertex data. The fragment processor(s) 1315A-1315N use the primitive and vertex data generated by the vertex processor 1305 to generate a frame buffer that is displayed on a display device. In one embodiment, the fragment processor(s) 1315A-1315N are optimized to execute fragment shader programs such as those provided in the OpenGL API, which may be used to perform operations similar to pixel shader programs such as those provided in the Direct 3D API.
[0188] Graphics processor 1310 additionally includes one or more memory management units (MMUs) 1320A and 1320B, cache(s) 1325A and 1325B, and circuit interconnect 1330A and 1330B that provide virtual to physical address mapping for graphics processor 1310, which includes vertex processor 1305 and / or fragment processor(s) 1315A-1315N that may reference vertex or image / texture data stored in memory in addition to vertex or image / texture data stored in one or more cache(s) 1325A and 1325B. In one embodiment, one or more MMU(s) 1320A-1320B may be synchronized with other MMUs in the system, including one or more MMUs associated with one or more application processor(s) 1205, image processor 1215, and / or video processor 1220 of Figure 12, such that each processor 1205-1220 can participate in a shared or unified virtual memory system. One or more circuit interconnect(s) 1330A and 1330B enable graphics processor 1310 to interface with other IP cores in the SoC via the SoC's internal bus or via a direct connection, according to an embodiment.
[0189] As shown in FIG. 13B, the graphics processor 1340 includes one or more MMU(s) 1320A and 1320B, cache(s) 1325A and 1325B, and circuit interconnect(s) 1330A and 1330B of the graphics processor 1310 of FIG. 13A. The graphics processor 1340 includes one or more shader core(s) 1355A-1355N (e.g., 1355N-1, and 1455A, 1355B, 1355C, 1355D, 1355E, and 1355F, through 1355N) that provide a unified shader core architecture in which a single core or type or cores can execute all types of programmable shader code, including shader program code for implementing vertex shaders, fragment shaders, and / or compute shaders. The exact number of existing shader cores can vary depending on the embodiment and implementation. In addition, the graphics processor 1340 includes an inter-core task manager 1345 that functions as a thread dispatcher to dispatch execution threads to one or more shader cores 1355A-1355N, and a tiling unit 1358 that accelerates tiling operations for tile-based rendering, where rendering operations for a scene are subdivided in image space, e.g., to exploit local spatial coherence within the scene or to optimize internal cache usage. Machine Learning Overview
[0190] Machine learning algorithms are algorithms that can learn based on a dataset. Embodiments of machine learning algorithms can be designed to model high-level abstractions within a dataset. For example, an image recognition algorithm can be used to determine which of several categories a given input belongs to, a regression algorithm can output a numerical value given an input, and a pattern recognition algorithm can be used to generate converted text or perform speech and / or voice recognition from text.
[0191] An exemplary type of machine learning algorithm is a neural network. There are many types of neural networks, and a simple type of neural network is a feedforward network. A feedforward network may be implemented as an acyclic graph with nodes arranged in layers. Typically, the topology of a feedforward network includes an input layer and an output layer separated by at least one hidden layer. The hidden layer converts the inputs received by the input layer into a representation useful for generating an output at the output layer. The nodes of the network are fully connected to the nodes of adjacent layers via edges, but there are no edges between the nodes within each layer. Data received at the nodes of the input layer of a feedforward network is propagated (i.e., "feeded forward") to the nodes of the output layer through an activation function that calculates the state of the nodes of each successive layer in the network based on the coefficients ("weights") associated with each of the edges connecting each layer. Depending on the particular model represented by the algorithm being executed, the output from a neural network algorithm can take a variety of forms.
[0192] Before a machine learning algorithm can be used to model a particular problem, the algorithm is trained using a training data set. Training a neural network involves selecting a topology for the network, using a set of training data that represents the problem modeled by the network, and adjusting the weights until the network model runs with minimal error for all instances of the training data set. For example, during a supervised learning training process for a neural network, outputs generated by the network in response to inputs that represent instances of the training data set are compared with outputs labeled as "correct" for that instance, an error signal representing the difference between the output and the labeled output is calculated, and weights associated with the connections are adjusted to minimize the error as the error signal propagates backwards through the layers of the network. The network is considered "trained" when the error for each output generated from an instance of the training data set is minimized.
[0193] The accuracy of machine learning algorithms can be significantly affected by the quality of the datasets used to train the algorithm. The training process is computationally intensive and can require a significant amount of time on traditional general-purpose processors. Thus, parallel processing hardware is used to train many types of machine learning algorithms. This is particularly useful for optimizing the training of neural networks, as the calculations performed in adjusting the coefficients of a neural network naturally lend themselves to parallel implementation. In particular, many machine learning algorithms and software applications are adapted to take advantage of the parallel processing hardware within general-purpose graphics processing devices.
[0194] FIG. 14 is a generalized diagram of a machine learning software stack 1400. A machine learning application 1402 can be configured to train a neural network using a training dataset or to implement machine intelligence using a trained deep neural network. The machine learning application 1402 can include training and inference functions for the neural network and / or specialized software that can be used to train the neural network prior to deployment. The machine learning application 1402 can implement any type of machine intelligence, including, but not limited to, image recognition, mapping and localization, autonomous navigation, speech synthesis, medical imaging, or language translation.
[0195] Hardware acceleration for machine learning applications 1402 can be enabled through machine learning framework 1404. Machine learning framework 1404 can provide a library of machine learning primitives. Machine learning primitives are basic operations typically performed by machine learning algorithms. Without machine learning framework 1404, developers of machine learning algorithms would be required to create and optimize the main computational logic associated with the machine learning algorithm, and then reoptimize the computational logic as new parallel processors are developed. Instead, machine learning applications can be configured to perform the necessary computations using primitives provided by machine learning framework 1404. Exemplary primitives include tensor convolutions, activation functions, and pooling, which are computational operations performed while training a convolutional neural network (CNN). Machine learning framework 1404 can also provide primitives for implementing basic linear algebra subprograms performed by many machine learning algorithms, such as matrix and vector operations.
[0196] The machine learning framework 1404 can process input data received from the machine learning application 1402 and generate appropriate inputs for the computation framework 1406. The computation framework 1406 can abstract the underlying instructions provided to the GPGPU driver 1408 to enable the machine learning framework 1404 to fully exploit hardware acceleration via the GPGPU hardware 1410 without requiring the machine learning framework 1404 to have in-depth knowledge of the architecture of the GPGPU hardware 1410. Additionally, the computation framework 1406 can enable hardware acceleration across different types and generations of GPGPU hardware 1410. Machine learning neural network implementation
[0197] The computing architecture provided by the embodiments described herein can be configured to perform a type of parallel processing that is particularly suitable for training and deploying neural networks for machine learning. A neural network can be generalized as a network of functions with graph relationships. As known in the art, there are various types of implementations of neural networks used in machine learning. One exemplary type of neural network is a feedforward network, as described above.
[0198] A second exemplary type of neural network is the convolutional neural network (CNN). CNNs are specialized feed-forward neural networks for processing data with a known grid-like topology, such as image data. Thus, CNNs are commonly used for computational vision and image recognition applications, but may also be used for other types of pattern recognition, such as speech and language processing. The nodes in the input layer of a CNN are organized into a set of "filters" (feature detectors inspired by the receptive fields found in the retina), and the output of each set of filters is propagated to the nodes in successive layers of the network. A CNN's computation involves applying the mathematical operation of convolution to each filter to produce an output for that filter. Convolution is a special species of mathematical operation performed by two functions to produce a third function that is a modified version of one of the two original functions. In convolutional network terminology, the first function for the convolution can be referred to as the input, and the second function can be referred to as the convolution kernel. The output may be referred to as a feature map. For example, the input to a convolutional layer can be a multidimensional array of data that defines the various color components of the input image. The convolution kernel can be a multidimensional array of parameters, where the parameters are adapted by a training process for the neural network.
[0199] Recurrent neural networks (RNNs) are a family of feedforward neural networks that contain feedback connections between layers. RNNs allow for modeling of data in a sequential manner by sharing parameter data across different parts of the neural network. The architecture of an RNN contains cycles. A cycle represents the influence of a current value of a variable on its own value at a future time, as at least a portion of the output data from the RNN is used as feedback to process subsequent inputs in the sequence. This feature makes RNNs particularly useful for language processing, due to the nature of the variables from which language data can be constructed.
[0200] The following figures present example feedforward, CNN, and RNN networks, as well as describing the general process for training and deploying each type of network. It will be understood that these descriptions are exemplary and non-limiting with respect to any particular embodiment described herein, and that the illustrated concepts can be applied to deep neural networks and machine learning techniques in general.
[0201] Deep learning can be performed using the exemplary neural network described above. Deep learning is machine learning using deep neural networks. The deep learning neural networks used in deep learning are artificial neural networks consisting of multiple hidden layers, as opposed to shallow neural networks that contain only a single hidden layer. Deep neural networks are generally more computationally intensive to train. However, adding hidden layers of the network allows for multi-stage pattern recognition, resulting in less output error compared to shallow machine learning techniques.
[0202] A deep neural network used in deep learning typically includes a front-end network for performing feature recognition coupled to a back-end network representing a mathematical model that can perform operations (e.g., object classification, speech recognition, etc.) based on feature representations provided to the model. Deep learning allows machine learning to be performed without the need for manual feature engineering on the model. Instead, the deep neural network can learn features based on statistical structures or correlations in the input data. The learned features can be provided to a mathematical model that can map the detected features to an output. The mathematical model used in the network is generally specialized for the particular task to be performed, with different models being used to perform different tasks.
[0203] Once a neural network is structured, a learning model can be applied to the network to train the network to perform a specific task. The learning model describes how to adjust the weights in the model to reduce the network's output error. Backpropagation of error is a common method used to train neural networks. An input vector is presented to the network for processing. The output of the network is compared to the desired output using a loss function, and an error value is calculated for each neuron in the output layer. The error values are then propagated backwards until each neuron has an associated error value that roughly represents its contribution to the original output. The network can then learn from these errors using algorithms such as stochastic gradient descent algorithms to update the neural network weights.
[0204] 15A and 15B illustrate an exemplary convolutional neural network. FIG. 15A shows various layers within a CNN. As shown in FIG. 15A, an exemplary CNN used to model image processing can receive an input 1502 describing the red, green, and blue (RGB) components of an input image. The input 1502 can be processed by multiple convolutional layers (e.g., a first convolutional layer 1504, a second convolutional layer 1506). The output from the multiple convolutional layers can optionally be processed by a set of fully connected layers 1508. The neurons in the fully connected layers are fully connected to all activations of the previous layer, as described above for feedforward networks. The output from the fully connected layers 1508 can be used to generate an output result from the network. The activations in the fully connected layers 1508 can be calculated using matrix multiplication instead of convolution. Not all CNN implementations utilize fully connected layers 1508. For example, in some implementations, the second convolutional layer 1506 can generate the output of the CNN.
[0205] Convolutional layers are loosely connected, unlike the configuration of traditional neural networks found in the fully connected layer 1508. Traditional neural network layers are fully connected, such that every output unit interacts with every input unit. However, convolutional layers are loosely connected because, as shown, the output of the convolution of the field is input to the nodes of the subsequent layer (instead of the respective state values of each node in the field). The kernel associated with the convolutional layer performs the convolution operation, but the output is sent to the next layer. The dimensionality reduction that occurs within the convolutional layer is one aspect that allows CNNs to scale to process large images.
[0206] FIG. 15B illustrates exemplary computational stages within a convolutional layer of a CNN. An input to the convolutional layer 1512 of a CNN can be processed in three stages in the convolutional layer 1514. The three stages can include a convolution stage 1516, a detector stage 1518, and a pooling stage 1520. The convolutional layer 1514 can then output data to the next convolutional layer. The final convolutional layer of the network can generate output feature map data or provide input to a fully connected layer to generate, for example, a classification value for the input to the CNN.
[0207] The convolution stage 1516 performs several convolutions in parallel to generate a set of linear activations. The convolution stage 1516 can include an affine transformation, which is any transformation that can be specified as a linear transformation plus a translation. Affine transformations include rotation, translation, scaling, and combinations of these transformations. The convolution stage calculates the output of a function (e.g., a neuron) coupled to a particular region of the input, which can be determined as a local region associated with the neuron. The neuron calculates a dot product between the neuron's weight and the region in the local input to which the neuron is connected. The output from the convolution stage 1516 defines a set of linear activations that are processed by successive stages of the convolution layer 1514.
[0208] The linear activations may be processed by a detector stage 1518, where each linear activation is processed by a nonlinear activation function. The nonlinear activation function increases the nonlinear characteristics of the entire network without affecting the receptive fields of the convolutional layers. Several types of nonlinear activation functions may be used. One particular type of nonlinear activation function is the rectified linear unit (ReLU), which uses an activation function defined as f(x)=max(0,x), such that the activation has a threshold at zero.
[0209] The pooling stage 1520 uses a pooling function that replaces the output of the second convolutional layer 1506 with a summary statistic of nearby outputs. The pooling function can be used to introduce translation invariance into the neural network so that small translations to the input do not change the pooled output. Invariance to local translations can be useful in scenarios where the presence of a feature in the input data is more important than the exact location of the feature. Various types of pooling functions can be used during the pooling stage 1520, including max pooling, average pooling, and l2-norm pooling. In addition, some CNN implementations do not include a pooling stage. Instead, such implementations substitute an additional convolution stage with an increased stride for the previous convolution stage.
[0210] The output from the convolutional layer 1514 can then be processed by the next layer 1522, which can be an additional convolutional layer or one of the fully connected layers 1508. For example, the first convolutional layer 1504 of FIG. 15A can output to the second convolutional layer 1506, which in turn can output to the first layer of the fully connected layers 1508.
[0211] FIG. 16 illustrates an exemplary recurrent neural network. In a recurrent neural network (RNN), the previous state of the network influences the output of the current state of the network. RNNs can be constructed in a variety of ways with a variety of functions. The use of RNNs generally revolves around using mathematical models to predict the future based on a sequence of preceding inputs. For example, an RNN may be used to perform statistical language modeling to predict the next word given a previous word sequence. The illustrated RNN 1600 can be described as having an input layer 1602 that receives input vectors, a hidden layer 1604 to implement a recurrent function, a feedback mechanism 1605 to allow "memory" of the previous state, and an output layer 1606 to output the results. The RNN 1600 operates based on time steps. The state of the RNN at a given time step is influenced based on the previous time step via the feedback mechanism 1605. For a given time step, the state of the hidden layer 1604 is defined by the previous state and the input at the current time step. An initial input (x1) at a first time step may be processed by the hidden layer 1604. A second input (x2) may be processed by the hidden layer 1604 using state information determined during the processing of the initial input (x1). A given state is expressed as s t =(Ux t +Ws t-1 ) where the function f is typically a nonlinearity such as a variant of the hyperbolic tangent function (Tanh) or the rectifier function f(x)=max(0,x). However, the particular mathematical function used in the hidden layer 1604 may vary depending on the particular implementation details of the RNN 1600.
[0212] In addition to the basic CNN and RNN networks mentioned above, variations of those networks may also be possible. One variation of RNN is the long short-term memory (LSTM) RNN. LSTM RNNs can learn long-term dependencies that may be needed to process longer language sequences. A variation of CNN is the convolutional deep belief network, which has a similar structure to CNNs and is trained in a similar manner to deep belief networks. Deep belief networks (DBNs) are generative neural networks that consist of multiple layers of stochastic (random) variables. DBNs can be trained layer-by-layer using greedy unsupervised learning. The learned DBN weights can then be used to provide a pre-trained neural network by determining an optimal initial set of weights for the neural network.
[0213] FIG. 17 illustrates the training and deployment of deep neural networks. Once a given network is structured for a task, the neural network is trained using a training dataset 1702. Various training frameworks have been developed to allow hardware acceleration of the training process. For example, the machine learning framework 1404 of FIG. 14 may be configured as a training framework 1704. The training framework 1704 may hook into an untrained neural network 1706, allowing the untrained neural net to be trained using parallel processing resources as described herein to generate a trained neural network 1708. To start the training process, initial weights may be selected randomly or pre-trained with a deep belief network. The training cycle is then performed in either a supervised or unsupervised manner.
[0214] Supervised learning is a learning method in which training is performed as a mediated operation, such as when a training data set 1702 includes inputs paired with desired outputs for the inputs, or when a training data set includes inputs with known outputs and the neural network's output is manually graded. The network processes the inputs and compares the resulting outputs to a set of expected or desired outputs. Errors are then propagated backwards through the system. The training framework 1704 can adjust the weights that control the untrained neural network 1706. The training framework 1704 can provide tools to monitor how well the untrained neural network 1706 is converging toward a model suitable for generating correct answers based on known input data. The training process is iterative such that the network weights are adjusted to refine the outputs generated by the neural network. The training process can continue until the neural network reaches a statistically desired accuracy associated with the trained neural network 1708. The trained neural network 1708 can then be deployed to implement any number of machine learning operations to generate inference results 814 based on the input of new data 812.
[0215] Unsupervised learning is a learning method in which a network attempts to train itself using unlabeled data. Thus, in unsupervised learning, the training dataset 1702 will include input data without associated output data. The untrained neural network 1706 can learn groupings within the unlabeled inputs and determine how individual inputs relate to the entire dataset. Unsupervised training can be used to generate self-organizing maps, which are a type of trained neural network 1708 that can perform operations useful for reducing the dimensionality of data. Unsupervised training can also be used to perform anomaly detection, allowing for the identification of data points within the input dataset that deviate from normal patterns of data.
[0216] Variations of supervised and unsupervised training may also be employed. Semi-supervised learning is a technique in which labeled and unlabeled data of the same fabric are mixed in the training dataset 1702. Incremental learning is a variant of supervised learning, where input data is continually used to further train the model. Incremental learning allows the trained neural network 1708 to adapt to new data 1712 without forgetting the knowledge implanted in the network during initial training.
[0217] The training process, especially for deep neural networks, whether supervised or unsupervised, can be computationally too intensive for a single computation node. Instead of using a single computation node, a distributed network of computation nodes can be used to speed up the training process.
[0218] 18 is a block diagram illustrating distributed learning. Distributed learning is a training model that uses multiple distributed computing nodes to perform supervised or unsupervised learning of neural networks. The distributed computing nodes may each include one or more host processors and one or more general-purpose processing nodes. As shown, distributed learning may be performed with model parallelism 1802, data parallelism 1804, or a combination of model parallelism and data parallelism 1804.
[0219] In model parallelization 1802, different computational nodes in a distributed system can perform training computations for different portions of a single network. For example, each layer of a neural network can be trained by a different processing node of the distributed system. Advantages of model parallelization include the ability to scale to particularly large models. Splitting up the computations associated with different layers of a neural network enables the training of very large neural networks, where the weights of all layers do not fit into the memory of a single computational node. In some instances, model parallelization is particularly useful in performing unsupervised training of large neural networks.
[0220] In data parallelism 1804, different nodes of the distributed network have a complete instance of the model, and each node receives a different portion of the data. The results from the different nodes are then combined. Different approaches to data parallelism are possible, but all of the data parallelism training approaches require techniques to combine the results and synchronize the model parameters between each node. Exemplary approaches to combining data include parameter averaging and update-based data parallelism. Parameter averaging trains each node on a subset of the training data and sets the global parameters (e.g., weights, biases, etc.) to the average value of the parameters from each node. Parameter averaging uses a central parameter server that maintains the parameter data. Update-based data parallelism is similar to parameter averaging, except that instead of transferring parameters from nodes to the parameter server, model updates are transferred. Additionally, update-based data parallelism can be performed in a distributed manner where updates are compressed and transferred between nodes.
[0221] The composite parallelization of model and data 1806 can be implemented, for example, in a distributed system where each compute node contains multiple GPUs. Each node can have a full instance of the model with individual GPUs in each node, which are used to train different parts of the model.
[0222] Distributed training incurs increased overhead compared to training on a single machine, however, the parallel processors and GPGPUs described herein can each implement various techniques to reduce the overhead of distributed training, including techniques that enable high bandwidth GPU-to-GPU data transfers and accelerated remote data synchronization. Example Machine Learning Applications
[0223] Machine learning can be applied to solving a variety of technical problems, including, but not limited to, computer vision, autonomous driving and navigation, speech recognition, and language processing. Computer vision has traditionally been one of the most active research areas for the application of machine learning. Computer vision applications range from replicating human visual capabilities, such as face recognition, to creating new categories of visual capabilities. For example, computer vision applications can be configured to recognize sound waves from vibrations triggered by objects seen in videos. Parallel processor-accelerated machine learning allows computer vision applications to be trained using much larger training datasets than previously possible, allowing inference systems to be deployed using low-power parallel processors.
[0224] Accelerated machine learning with parallel processors has applications in autonomous driving including lane and road sign recognition, obstacle avoidance, navigation, and driving control. Accelerated machine learning techniques can be used to train driving models based on data sets that define appropriate responses to specific training inputs. The parallel processors described herein enable rapid training of increasingly complex neural networks used in autonomous driving solutions and enable deployment of low power inference processors on mobile platforms suitable for integration into autonomous vehicles.
[0225] Deep neural networks accelerated by parallel processors are enabling a machine learning approach to automatic speech recognition (ASR), which involves generating a function that computes the most probable language sequence given an input acoustic sequence. Accelerated machine learning using deep neural networks makes it possible to replace the hidden Markov model (HMM) and Gaussian mixture model (GMM) traditionally used for ASR.
[0226] Accelerated machine learning with parallel processors can also be used to accelerate natural language processing. Automated learning procedures can leverage statistical inference algorithms to generate models that are robust to erroneous or unfamiliar input. Exemplary applications of natural language processors include automatic machine translation between human languages.
[0227] Parallel processing platforms used for machine learning can be divided into training platforms and deployment platforms. Training platforms are typically highly parallelized and include optimizations to accelerate multi-GPU single-node training or multi-node, multi-GPU training, while deployed machine learning (e.g., inference) platforms typically include low-power parallel processors suitable for use in products such as cameras, autonomous robots, and autonomous cars. GPGPU with matrix acceleration logic
[0228] 19 is a block diagram of a data processing system 1900 according to one embodiment. The data processing system 1900 is a heterogeneous processing system having a processor 1902, a unified memory 1910, and a GPGPU 1920 including machine learning acceleration logic. The processor 1902 and the GPGPU 1920 can be any of the processors and GPGPUs / parallel processors described herein. The processor 1902 can execute instructions for a compiler 1915 stored in a system memory 1912. The compiler 1915 executes on the processor 1902 to compile source code 1914A into compiled code 1914B. The compiled code 1914B can include instructions that can be executed by the processor 1902 and / or instructions that can be executed by the GPGPU 1920. During compilation, compiler 1915 may perform operations to insert metadata including hints regarding the level of data parallelism present in compiled code 1914B and / or hints regarding data locality associated with threads to be dispatched based on compiled code 1914B. Compiler 1915 may include information necessary to perform such operations, or the operations may be performed with the assistance of runtime library 1916. Runtime library 1916 may assist compiler 1915 in compiling source code 1914A, and may also include instructions that are linked at runtime with compiled code 1914B to facilitate execution of the compiled instructions on GPGPU 1920.
[0229] Unified memory 1910 represents a unified address space that may be accessed by processor 1902 and GPGPU 1920. The unified memory may include GPGPU memory 1918 and system memory 1912. GPGPU memory 1918 is memory within the address space of GPGPU 1920 and may include some or all of system memory 1912. In one embodiment, GPGPU memory 1918 may also include at least a portion of any memory dedicated for exclusive use by GPGPU 1920. In one embodiment, compiled code 1914B stored in system memory 1912 may be mapped to GPGPU memory 1918 for access by GPGPU 1920.
[0230] The GPGPU 1920 includes multiple computation blocks 1924A-1924N, which may include one or more of the various computation units or execution elements described herein. In one embodiment, the GPGPU 1920 further includes a matrix accelerator 1923, which may include one or more special function computation units designed to accelerate a subset of matrix operations (e.g., inner products, etc.). The GPGPU 1920 may also include a set of resources shareable by the computation blocks 1924A-1924N and the matrix accelerator 1923, including, but not limited to, a set of registers 1925, a power and performance module 1926, and a cache 1927. In one embodiment, the registers 1925 include directly accessible registers and indirectly accessible registers, where the indirectly accessible registers are optimized for use by the matrix accelerator 1923. The power and performance module 1926 can be configured to adjust the power supply and clock frequency for the compute blocks 1924A-1924N to power power-gated idle components within the compute blocks 1924A-1924N. In various embodiments, the cache 1927 can include an instruction cache and / or a lower level data cache.
[0231] The GPGPU 1920 may further include an L3 data cache 1930 that may be used to cache data accessed from the unified memory 1910 by the matrix accelerator 1923 and / or the computational elements in the computation blocks 1924A-1924N. In one embodiment, the L3 data cache 1930 includes a shared local memory 1932 that is shareable by the computational elements in the computation blocks 1924A-1924N and the matrix accelerator 1923.
[0232] In one embodiment, the GPGPU 1920 includes instruction handling logic such as a fetch and decode unit 1921 and a scheduler controller 1922. The fetch and decode unit 1921 includes a fetch unit and a decode unit for fetching and decoding instructions for execution by one or more computation blocks 1924A-1924N or a matrix accelerator 1923. Instructions can be scheduled to appropriate functional units within the computation blocks 1924A-1924N or a matrix accelerator via the scheduler controller 1922. In one embodiment, the scheduler controller 1922 is an ASIC configurable to perform advanced scheduling operations. In one embodiment, the scheduler controller 1922 is a microcontroller or a low energy per instruction processing core that can execute scheduler instructions loaded from a firmware module.
[0233] In one embodiment, some functions performed by the computation blocks 1924A-1924N can be scheduled or offloaded directly to the matrix accelerator 1923. In various embodiments, the matrix accelerator 1923 includes processing element logic configured to efficiently perform matrix computation operations such as multiplication and addition operations and dot product operations used by 3D graphics or computation shader programs. In one embodiment, the matrix accelerator 1923 can be configured to accelerate operations used by machine learning frameworks. In one embodiment, the matrix accelerator 1923 is an application-specific integrated circuit explicitly configured to perform a specific set of parallel matrix multiplication and / or addition operations. In one embodiment, the matrix accelerator 1923 is a field programmable gate array (FPGA) that provides fixed-function logic that is updatable between workloads. The set of matrix operations that may be performed by the matrix accelerator 1923 may be limited to the operations that may be performed by the computation blocks 1924A-1924N. However, the matrix accelerator 1923 can perform these operations with significantly higher throughput compared to the computation blocks 1924A-1924N.
[0234] 20 illustrates one embodiment of a matrix operation 2005 performed by an instruction pipeline 2000, according to one embodiment. The instruction pipeline 2000 may be configured to perform a matrix operation 2005, such as, but not limited to, a dot product operation. The dot product of two vectors is a scalar value equal to the sum of the products of corresponding components of the vectors. The dot product may be calculated as follows:
number
[0235] The dot product can be used in the convolution operation of a convolutional neural network (CNN). FIG. 19 illustrates a two-dimensional (2D) convolution using a matrix operation 2005 that includes a dot product operation. Although a 2D convolution is illustrated, an N-dimensional convolution can be performed on an N-dimensional volume using an N-dimensional filter. A receptive field tile 2002 highlights a portion of an input volume in an input volume buffer 2004. The input volume buffer can be stored in a memory 2030. A matrix dot product operation 2005 can be performed between the data in the receptive field tile 2002 and a convolution filter to generate a data point in an output buffer 2006, which can also be stored in the memory 2030. The memory 2030 can be any of the memories described herein, including the system memory 1912, the GPGPU memory 1918, or one or more cache memories 1927 and 1930 as in FIG. 19.
[0236] The combination of data points in the output buffer 2006 represents an activation map generated by the convolution operation. Each point in the activation map is generated by sliding a receptive field tile through the input volume buffer 2004. The activation map data can be input to an activation function to determine an output activation value. In one embodiment, the convolution of the input volume buffer 2004 can be defined within the framework as a high-level matrix operation 1705. The high-level matrix operation can be performed via primitive operations such as Basic Linear Algebra Subprogram (BLAS) operations. The primitive operations can be accelerated via hardware instructions executed by the instruction pipeline 2000.
[0237] The instruction pipeline 2000 used to accelerate hardware instructions may include an instruction fetch and decode unit 1921 that may fetch and decode hardware instructions, and a scheduler controller 1922 that may schedule the decoded instructions to one or more execution units in the computation blocks 1924A-1924N and / or the matrix accelerator 1923. In one embodiment, the hardware instructions may be scheduled to the computation blocks 1924A-1924N and offloaded to the matrix accelerator 1923. One or more hardware instructions and associated data for performing the matrix operation 2005 may be stored in the memory 2030. Output of the hardware instructions may also be stored in the memory 2030.
[0238] In one embodiment, the matrix accelerator 1923 can execute one or more hardware instructions to perform the matrix operations 2005 using integrated dot product logic 2008 (DP logic). The dot product logic 2008 can include a combination of programmable and fixed function hardware configurable to perform dot product operations. Although the functional units in the computation blocks 1924A-1924N can also be configured to perform dot product operations, the dot product logic 2008 can be configured to perform a limited subset of dot product operations at a significantly higher throughput compared to the computation blocks 1924A-1924N.
[0239] During the processing of matrix multiplication operations in CNNs such as Residual Neural Network (ResNet)-50, there exists a significant percentage of sparsity (e.g., zero values) in the activation matrices across multiple layers. In fact, in some cases, the sparsity can reach as high as 99.9%. Such a high percentage of sparse matrices provides opportunities for processing optimizations, including bypassing redundant memory accesses and computations, thus reducing energy consumption by saving unnecessary processing.
[0240] As discussed above, having to perform matrix layout operations (e.g., transform and transpose operations) before computation operations reduces GPU performance. According to one embodiment, a block instruction is provided for optimizing deep learning kernels. In such an embodiment, the instruction, when executed, reads a matrix tile from a GRF, such as GRF 537 described above with reference to Figures 5A and 5B, and loads the tile into the GRF, thereby eliminating a move instruction. In a further embodiment, the instruction includes a user-defined message parameter that defines a two-dimensional (2D) memory plane. In this manner, the instruction enables software to implement standardized input data layouts defined by multi-platform machine learning and deep learning frameworks and enables matrix operations (e.g., transform and transpose operations) to be performed on the graphics hardware.
[0241] According to one embodiment, the received command includes a message header payload that includes 2D surface parameters that define the block start position (x,y), width, height, and array length array_length. In a further embodiment, the 2D command has the following characteristics: a. 2D surface parameters supplied via the message header. The 2D surface parameters include the base address of the surface, its height, its width (in bytes), and its pitch (in bytes). The surface width, pitch, and height are 24-bit unsigned integers. b. Message access elements of size 4 bits, 8 bits, 16 bits, 32 bits, 64 bits, etc. c. The message block parameters specify the block by start (x, y), height, and width (e.g., width is represented by # elements), where x and y are of the form S32 (32-bit signed integers). d. If outbound, the read returns 0. The write is dropped. This way tiles outside the matrix can be read without causing a page fault.
[0242] In one embodiment, the received command also includes a message descriptor indicating a message type. In such an embodiment, the message descriptor identifies which of a plurality of operations is to be performed. In a further embodiment, the message descriptor includes a bit associated with each type of operation that may be performed. For example, a first bit may indicate that a 2D block read with a transpose operation is to be performed, and a second bit may indicate that a 2D block read with a transform operation is to be performed. In a further embodiment, the message is treated as a 2D block read message if it is determined that neither bit is set. In other embodiments, different types of bit encodings may be implemented to indicate the operation type. Reading 2D blocks
[0243] According to one embodiment, an instruction is implemented that includes a 2D block read message to specify one or more blocks to read from memory. In such an embodiment, an array length (or array_len) attribute included in the message parameters indicates the amount of blocks to be read from memory. In a further embodiment, the blocks are organized horizontally in memory and the return GRF is sequential (e.g., starting with the leftmost block and incrementing from left to right).
[0244] FIG. 21 illustrates one embodiment of a data block associated with a message having parameters including a block size, block_size, of 16×32B, and two arrays, ary_len (e.g., two blocks to be read from memory). As shown in FIG. 21, the starting GRF number is identified in the message as 0. Because the block size is 16×32B, each block message (or block) returns 8 GRFs. Thus, the GRF number for the second block starts at 8. Also, the layout of the second block is contiguous to the right of the first block (e.g., memory addresses increase from left to right).
[0245] In further embodiments, a single 2D block with an array_len of N is logically equivalent to a separate block with an array_len of N ones, where block[n](x,y), width, height, and grf_num are calculated as shown in FIG. 22. Multiple blocks are supported to ensure that full cache line requests are generated in linear memory. A single block matches the tile size used by a thread. Thus, in some embodiments, the tile width may not fill an entire cache line.
[0246] In a further embodiment, the 2D block read operation supports d8, d16, d32, and d64 data sizes, as specified in Table 1. However, other embodiments may feature different data and block sizes. In yet another embodiment, the product of element size element_size (bytes) * max block_size maximum block size typically matches the granularity of GPU memory requests (e.g., 64B in this example) to maximize memory efficiency. [Table 1]
[0247] FIG. 23A is a flow diagram illustrating one embodiment of a 2D block read operation. In processing block 2310, the message header is read to determine block parameters. Thus, the starting position of the tile (xy), the width and height of the block, and the length of the array are determined. In processing block 2320, a block of data is retrieved (or fetched) from memory based on the parameters. FIG. 24 illustrates one embodiment of a block of data associated with a message with parameters including a block size of 32x32B, block_size, and an array of array_len of 2. In processing block 2330, the block data is stored in a GRF buffer. FIG. 25 illustrates one embodiment of a block of data stored in a GRF buffer. As shown in FIG. 25, RnBm represents Byte-m of Row-n of a memory block stored in a GRF buffer (e.g., in grf0-grf31). FIG. 23B illustrates one embodiment of pseudocode for a 2D block read operation. Reading 2D blocks with transposition
[0248] According to one embodiment, an instruction including a 2D block read with a transpose message is implemented to specify multiple blocks to read from memory and perform a matrix transpose on the blocks. A matrix transpose is an operator that generates a transposed matrix that results from flipping a matrix diagonally (e.g., switching the row and column indices of the matrix). In one embodiment, the rows of the block data read from memory are transposed before being stored as columns in the GRF buffer. In a further embodiment, the transpose may only support an array_len of 1. This is because splitting one large block vertically to create multiple smaller blocks does not change the GRF layout of the transpose message. However, in other embodiments, different array lengths may be supported.
[0249] In one embodiment, the GRF data (e.g., per block column) is rounded to the next power of 2 bytes. That is, the block height is rounded appropriately so that the transposed GRF pitch per column is a power of 2 bytes. For example, if the block height is 7 and the element size is 2B, the block height is rounded to 8 and the transposed GRF pitch is 8*2B=16 bytes. In one embodiment, the padded GRF bytes due to rounding up are 0. In such an embodiment, the maximum number of GRFs that can be returned from this message is 32.
[0250] 26A is a flow diagram illustrating one embodiment of a 2D block read and transpose operation. At process block 2610, the message header is read to determine block parameters, as described above. At process block 2620, a block of data is fetched from memory based on the parameters. At process block 2630, each row of the block data read from memory is transposed. At process block 2640, the transposed data is stored in a GRF buffer.
[0251] Figure 27A illustrates one embodiment of a data block stored in a GRF buffer. As shown in Figure 27A, the data block has an element size of 1 byte, a block size of 32x64, and an array array_len=1. Figure 27B illustrates another embodiment of a data block stored in a GRF buffer with an element size of 2 bytes (e.g., d16), a block size of 32x16 blocks, and an array array_len=1. Figure 26B illustrates one embodiment of pseudocode for a 2D block read with a transpose operation. 2D Block Read with Transformation
[0252] According to one embodiment, messages including read 2D block with transform messages are implemented to specify multiple blocks to read from memory and perform a matrix transform on the blocks. In such an embodiment, a Vector Neural Network Instruction (VNNI) transform is performed such that n elements from a column are taken (e.g., where n=4B / element size element_size) and stored back-to-back in the same GRF buffer. Thus, for an 8b int, the number of elements is 4, and for an F16 / Bf16, the number of elements is 2.
[0253] 28A is a flow diagram illustrating one embodiment of a process performed during execution of a 2D block read and transform instruction. At process block 2810, the message header is read to determine block parameters as described above. At process block 2820, a block of data is fetched from memory based on the parameters. At process block 2830, each row of the block data read from memory is transformed. At process block 2840, the transformed data is stored in a GRF buffer.
[0254] Figure 29 illustrates one embodiment of a data block stored in a GRF buffer. As shown in Figure 29, the data block has an element size of 1 byte, a block size of 16, and an array array_len=4. Figure 28B illustrates one embodiment of pseudocode for reading a 2D block with a transform operation.
[0255] According to one embodiment, the instruction includes a read of a 2D block with a transpose and transform operation. In this embodiment, the read of a 2D block with a transpose and transform operation includes the same functionality as a 4B transpose (or D-word). Thus, whenever a transpose and transform operation is to be performed (e.g., by the matrix accelerator 1923), a read of a 2D block with a transpose message is included in the instruction.
[0256] As described above, a message descriptor included in an instruction is implemented to indicate the type of message. Thus, the message indication is examined prior to determining the type of operation to be performed by execution. FIG. 30 is a flow diagram illustrating one embodiment of an instruction execution process. At processing block 3010, an instruction is received. At processing block 3020, the instruction is decoded. At processing block 3030, the message descriptor included in the instruction is examined to determine the type of matrix manipulation operation to be performed. At processing block 3040, the layout operation indicated in the message descriptor (e.g., 2D block read, 2D block read with transpose, 2D block read with transform, etc.) is performed based on the parameters in the message header. Thereafter, a matrix multiplication operation may be performed.
[0257] The embodiments may be implemented as one or more microchips or integrated circuits interconnected using a mother board, hardwired logic, software stored by a memory device and executed by a microprocessor, firmware, application specific integrated circuits (ASICs), and / or field programmable gate arrays (FPGAs), or any combination thereof. The term "logic" may illustratively include software or hardware, and / or a combination of software and hardware.
[0258] The embodiments may be provided, for example, as a computer program product, which may include one or more machine-readable media having stored thereon machine-executable instructions that, when executed by one or more machines, such as a computer, a network of computers, or other electronic devices, may result in the one or more machines performing operations in accordance with the embodiments described herein. Machine-readable media may include, but are not limited to, floppy disks, optical disks, Compact Disc-Read Only Memories (CD-ROMs), magneto-optical disks, ROMs, RAMs, Erasable Programmable Read Only Memory (EPROMs), Electrically Erasable Programmable Read Only Memory (EEPROMs), magnetic or optical cards, flash memories, or other types of media / machine-readable media suitable for storing machine-executable instructions.
[0259] Further, embodiments may be downloaded as a computer program product, where the program may be transferred from a remote computer (e.g., a server) to a requesting computer (e.g., a client) via one or more data signals embodied and / or modulated on a carrier wave or other propagation medium over a communications link (e.g., a modem and / or network connection).
[0260] The following items and / or examples relate to further embodiments or examples. Particulars in the examples may be used anywhere in one or more embodiments. Various features of different embodiments or examples may be combined in various ways, with some features included and other features excluded, to suit a variety of different applications. Examples may include subject matter such as a method, a means for performing the acts of the method, at least one machine-readable medium including instructions that, when executed by a machine, cause the machine to perform the acts of the method, or an apparatus or system for facilitating hybrid communication according to the embodiments and examples described herein.
[0261] Some embodiments relate to Example 1, which is an apparatus for facilitating machine learning matrix processing, comprising: a memory for storing matrix data; one or more processors for executing instructions to examine a message descriptor included in the instruction to determine a type of matrix layout manipulation operation to be performed, examine a message header included in the instruction having a plurality of parameters defining a two-dimensional (2D) memory plane to be retrieved, and retrieve one or more blocks of matrix data from the memory based on the plurality of parameters; and a register file including a plurality of registers, wherein the one or more blocks of matrix data are stored in a first set of the plurality of registers.
[0262] Example 2 includes the subject matter of example 1, where the plurality of parameters includes an array length attribute that indicates the amount of the block to retrieve from memory.
[0263] Example 3 includes the subject matter of examples 1 and 2, wherein the plurality of parameters further includes attributes defining a width, a height, and a pitch of each of the one or more blocks.
[0264] Example 4 includes the subject matter of examples 1-3, where the message descriptor indicates that a 2D block read with a transpose operation is to be performed.
[0265] Example 5 includes the subject matter of examples 1-4, wherein the one or more processors perform a transpose operation on each of the one or more blocks of matrix data and store the transposed matrix in a first set of registers.
[0266] Example 6 includes the subject matter of examples 1-5, wherein the message descriptor indicates that a 2D block read with a transform operation is to be performed.
[0267] Example 7 includes the subject matter of examples 1-6, wherein the one or more processors perform a transform operation on each of the one or more blocks of matrix data and store the transformed matrix in a first set of registers.
[0268] Example 8 includes the subject matter of examples 1-7, wherein the message descriptor indicates that a 2D block read with transpose and transform operations is to be performed.
[0269] Some embodiments relate to Example 9, a method for facilitating machine learning matrix processing, comprising: examining a message descriptor included in an instruction to determine a type of matrix layout manipulation operation to be performed; examining a message header included in the instruction having a plurality of parameters defining a two-dimensional (2D) memory plane to be retrieved; retrieving one or more blocks of matrix data from memory based on the plurality of parameters; and storing the one or more blocks of matrix data in a first set of a plurality of registers.
[0270] Example 10 includes the subject matter of example 9, in which the plurality of parameters includes an array length attribute indicating the amount of blocks to retrieve from memory.
[0271] Example 11 includes the subject matter of example 9 and example 10, wherein the plurality of parameters further includes attributes defining a width, a height, and a pitch of each of the one or more blocks.
[0272] Example 12 includes the subject matter of examples 9-11, wherein the message descriptor indicates that a 2D block read with a transpose operation is to be performed.
[0273] Example 13 includes the subject matter of examples 9-12, wherein the transpose operation includes performing a transpose operation on each of the one or more blocks of matrix data and storing the transposed matrix in a first set of registers.
[0274] Example 14 includes the subject matter of examples 9-13, wherein the message descriptor indicates that a 2D block read with a transform operation is to be performed.
[0275] Example 15 includes the subject matter of examples 9-14, wherein the transform operation includes performing a transform operation on each of the one or more blocks of matrix data and storing the transformed matrix in a first set of registers.
[0276] Some embodiments relate to Example 16, which is at least one non-transitory computer-readable medium having instructions stored thereon that, when executed by one or more processors, cause the processor to examine a message descriptor included in the instructions to determine a type of matrix layout manipulation operation to be performed, examine a message header included in the instructions having a plurality of parameters defining a two-dimensional (2D) memory plane to be retrieved, retrieve one or more blocks of matrix data from memory based on the plurality of parameters, and store the one or more blocks of matrix data in a first set of a plurality of registers.
[0277] Example 17 includes the subject matter of example 16, in which the plurality of parameters includes an array length attribute indicating the amount of blocks to retrieve from memory.
[0278] Example 18 includes the subject matter of example 16 and example 17, wherein the plurality of parameters further includes attributes defining a width, a height, and a pitch of each of the one or more blocks.
[0279] Example 19 includes the subject matter of examples 16-18, wherein the message descriptor indicates that a 2D block read with a transpose operation is to be performed.
[0280] Example 20 includes the subject matter of examples 16-19, wherein the message descriptor indicates that a 2D block read with a transform operation is to be performed.
[0281] The foregoing description and drawings are to be understood in an illustrative and not a restrictive sense. Those skilled in the art will appreciate that various modifications and changes may be made to the embodiments described herein without departing from the broader spirit and scope of the present invention as set forth in the appended claims. Other possible items (Item 1) An apparatus for facilitating machine learning matrix processing, comprising: A memory for storing matrix data; Examining a message descriptor included in the instruction to determine the type of matrix layout manipulation operation to be performed; examining a message header included in the command having a number of parameters defining a two-dimensional (2D) memory plane to be retrieved; one or more processors executing instructions to retrieve one or more blocks of matrix data from said memory based on said parameters; a register file including a plurality of registers; The one or more blocks of the matrix data are stored in a first set of the plurality of registers. (Item 2) The apparatus described in Item 1, wherein the multiple parameters include an array length attribute indicating the amount of blocks to retrieve from memory. (Item 3) The apparatus described in Item 2, wherein the plurality of parameters further includes attributes defining a width, height, and pitch of each of the one or more blocks. (Item 4) The apparatus described in Item 3, wherein the message descriptor indicates that a 2D block read with a transpose operation is to be performed. (Item 5) The apparatus described in Item 4, wherein the one or more processors perform a transpose operation on each of the one or more blocks of the matrix data and store the transposed matrix in the first set of registers. (Item 6) The apparatus described in Item 3, wherein the message descriptor indicates that a 2D block read accompanied by a transform operation is to be performed. (Item 7) The apparatus described in Item 6, wherein the one or more processors perform a transform operation on each of the one or more blocks of the matrix data and store the transformed matrix in the first set of registers. (Item 8) The apparatus described in Item 3, wherein the message descriptor indicates that a 2D block read with transpose and transform operations is to be performed. 9. A method for facilitating machine learning matrix processing, comprising: examining a message descriptor included in an instruction to determine a type of matrix layout manipulation operation to be performed; examining a message header included in the command having a number of parameters defining a two-dimensional (2D) memory plane to be retrieved; Retrieving one or more blocks of matrix data from said memory based on said parameters; and storing the one or more blocks of the matrix data in a first set of the plurality of registers. (Item 10) The method described in Item 9, wherein the multiple parameters include an array length attribute indicating the amount of blocks to retrieve from memory. (Item 11) The method of item 10, wherein the plurality of parameters further includes attributes defining a width, height, and pitch of each of the one or more blocks. (Item 12) The method described in Item 9, wherein the message descriptor indicates that a 2D block read with a transpose operation is to be performed. (Item 13) The above transposition operation is performing a transpose operation on each of said one or more blocks of said matrix data; and storing the transposed matrix in the first set of registers. (Item 14) The method described in Item 11, wherein the message descriptor indicates that a 2D block read accompanied by a transform operation is to be performed. (Item 15) The above conversion operation is performing a transform operation on each of said one or more blocks of said matrix data; and storing the transformed matrix in the first set of registers. At least one non-transitory computer-readable medium having instructions stored thereon that, when executed by one or more processors, cause the processors to: examining a message descriptor included in the instruction to determine the type of matrix layout manipulation operation to be performed; examining a message header included in said command having a number of parameters defining a two-dimensional (2D) memory plane to be retrieved; Retrieving one or more blocks of matrix data from memory based on said plurality of parameters; A computer-readable medium that causes the one or more blocks of the matrix data to be stored in a first set of a plurality of registers. (Item 17) The computer-readable medium of item 16, wherein the multiple parameters include an array length attribute indicating the amount of blocks to retrieve from memory. (Item 18) The computer-readable medium of item 17, wherein the plurality of parameters further includes attributes defining a width, height and pitch of each of the one or more blocks. (Item 19) The computer-readable medium of item 18, wherein the message descriptor indicates that a 2D block read with a transpose operation is to be performed. (Item 20) The computer-readable medium of item 18, wherein the message descriptor indicates that a 2D block read accompanied by a transform operation is to be performed.
Claims
1. 1. An apparatus for facilitating machine learning matrix processing, comprising: A memory for storing matrix data; Examining a message descriptor included in the instruction to determine the type of matrix layout manipulation operation to be performed; examining a message header included in the command having a number of parameters defining a two-dimensional (2D) memory plane to be retrieved; one or more processors executing instructions to retrieve one or more blocks of the matrix data from the memory based on the parameters; a register file including a plurality of registers; the one or more blocks of matrix data are stored in a first set of the plurality of registers.
2. The apparatus of claim 1 , wherein the plurality of parameters includes an array length attribute indicating an amount of a block to retrieve from memory.
3. The apparatus of claim 2 , wherein the plurality of parameters further comprises attributes defining a width, height, and pitch of each of the one or more blocks.
4. The apparatus of claim 3 , wherein the message descriptor indicates that a 2D block read with a transpose operation is to be performed.
5. 5. The apparatus of claim 4, wherein the one or more processors perform a transpose operation on each of the one or more blocks of the matrix data and store the transposed matrix data in the first set of the plurality of registers.
6. The apparatus of claim 3 , wherein the message descriptor indicates that a 2D block read with a transform operation is to be performed.
7. 7. The apparatus of claim 6, wherein the one or more processors perform a transform operation on each of the one or more blocks of the matrix data and store the one or more blocks of transformed matrix data in the first set of the plurality of registers.
8. The apparatus of claim 3 , wherein the message descriptor indicates that a 2D block read with transpose and transform operations is to be performed.
9. 1. A method for facilitating machine learning matrix processing, comprising: examining a message descriptor included in the instruction to determine the type of matrix layout manipulation operation to be performed; examining a message header included in the command having a number of parameters defining a two-dimensional (2D) memory plane to be retrieved; Retrieving one or more blocks of matrix data from memory based on the plurality of parameters; storing the one or more blocks of matrix data in a first set of a plurality of registers.
10. The method of claim 9 , wherein the plurality of parameters includes an array length attribute that indicates an amount of a block to retrieve from memory.
11. The method of claim 10 , wherein the plurality of parameters further includes attributes defining a width, height, and pitch of each of the one or more blocks.
12. The method of claim 9 , wherein the message descriptor indicates that a 2D block read with a transpose operation is to be performed.
13. The transposition operation is performing a transpose operation on each of the one or more blocks of matrix data; and storing the one or more blocks of the transposed matrix data in the first set of the plurality of registers.
14. The method of claim 11 , wherein the message descriptor indicates that a 2D block read with a transform operation is to be performed.
15. the transform operation comprising: performing a transform operation on each of the one or more blocks of the matrix data; and storing the one or more blocks of the transformed matrix data in the first set of the plurality of registers.
16. A program for a computer, examining a message descriptor included in the instruction to determine the type of matrix layout manipulation operation to be performed; examining a message header included in said command having a number of parameters defining a two-dimensional (2D) memory plane to be retrieved; retrieving one or more blocks of matrix data from a memory based on said plurality of parameters; The one or more blocks of matrix data are stored in a first set of registers.
17. 17. The program of claim 16, wherein the plurality of parameters includes an array length attribute that indicates an amount of a block to retrieve from memory.
18. 20. The method of claim 17, wherein the plurality of parameters further comprises attributes defining a width, height and pitch of each of the one or more blocks.
19. 20. The program of claim 18, wherein the message descriptor indicates that a 2D block read with a transpose operation is to be performed.
20. 20. The program of claim 18, wherein the message descriptor indicates that a 2D block read with a transform operation is to be performed.
21. A computer readable medium storing a program according to any one of claims 16 to 20.
Citation Information
Patent Citations
Data processor
JP1994110446A
Image processing engine and image processing system including the same
JP2008003708A
Data transfer device
JP2008077590A
System and method for extensible wide-operand instructions
JP2017520873A
Arithmetic processing unit and method for controlling arithmetic processing unit
JP2018132901A
