Dot Product Multiplier Mechanism

Abstract

To provide a device and the like which facilitates matrix multiplication operation.SOLUTION: An apparatus comprises multiplication hardware to operate in a dot product mode. A multiplication stage included in the multiplication hardware is configured as a dot product of a plurality of bit vectors (N) so as to perform N×N multiplication operations on a plurality of multiplicands and perform addition operations on results of the N×N multiplication operations.SELECTED DRAWING: Figure 19

Description

[Technical Field]

[0001] Embodiments relate generally to data processing, and more particularly to data processing by general purpose graphics processing units. [Background technology]

[0002] Deep learning algorithms are currently implemented in various machine learning applications such as speech / video recognition, video summarization, etc. Various forms of neural networks (e.g., convolutional neural networks (CNNs), recurrent neural networks (RNNs), long short-term memory (LSTM), etc.) are applied to perform such workloads due to their highly parallel nature. Machine learning applications typically implement matrix multiplication workloads with multipliers.

[0003] So that the above-described features of the present embodiments may be understood in detail, a more particular description of the above briefly summarized embodiments can be had by reference to embodiments, some of which are illustrated in the accompanying drawings, which should be noted that the accompanying drawings depict only exemplary embodiments and are not to be considered limiting of the scope thereof. [Brief explanation of the drawings]

[0004] [Figure 1] FIG. 1 is a block diagram of a processing system according to an embodiment. [Figure 2A] 1 illustrates a computer system and graphics processor provided by embodiments described herein. [Figure 2B] 1 illustrates a computer system and graphics processor provided by embodiments described herein. [Figure 2C] 1 illustrates a computer system and graphics processor provided by embodiments described herein. [Figure 2D] 1 illustrates a computer system and graphics processor provided by embodiments described herein. [Figure 3A] 1 illustrates a block diagram of a further graphics processor and computation accelerator architecture provided by an embodiment. [Figure 3B] 1 illustrates a block diagram of a further graphics processor and computation accelerator architecture provided by an embodiment. [Figure 3C] 1 illustrates a block diagram of a further graphics processor and computation accelerator architecture provided by an embodiment. [Figure 4] FIG. 2 is a block diagram of a graphics processing engine of a graphics processor according to some embodiments. [Figure 5A] 5 illustrates thread execution logic 500 including an array of processing elements used in a graphics processor according to an embodiment. [Figure 5B] 5 illustrates thread execution logic 500 including an array of processing elements used in a graphics processor according to an embodiment. [Figure 6] 6 illustrates a further execution unit 600 according to an embodiment. [Figure 7] FIG. 2 is a block diagram illustrating a graphics processor instruction format according to some embodiments. [Figure 8] FIG. 2 is a block diagram of a graphics processor according to another embodiment. [Figure 9A] 1 illustrates a graphics processor command format and command sequence according to some embodiments. [Figure 9B] 1 illustrates a graphics processor command format and command sequence according to some embodiments. [Figure 10] 1 illustrates an example of a graphics software architecture for a data processing system according to some embodiments. [Figure 11A] 1 illustrates an integrated circuit package assembly according to an embodiment. [Figure 11B] 1 illustrates an integrated circuit package assembly according to an embodiment. [Figure 11C]1 illustrates an integrated circuit package assembly according to an embodiment. [Figure 11D] 1 illustrates an integrated circuit package assembly according to an embodiment. [Figure 12] FIG. 1 is a block diagram illustrating an example of a system on a chip integrated circuit according to an embodiment. [Figure 13A] FIG. 10 is a block diagram illustrating a further exemplary graphics processor. [Figure 13B] FIG. 10 is a block diagram illustrating a further exemplary graphics processor. [Figure 14] 1 illustrates a machine learning software stack according to an embodiment. [Figure 15A] Represents a layer in an example deep neural network. [Figure 15B] Represents a layer in an example deep neural network. [Figure 16] Represents an example recurrent neural network. [Figure 17] Represents training and deploying deep neural networks. [Figure 18] FIG. 1 is a block diagram illustrating distributed learning. [Figure 19] 1 illustrates one embodiment of a computing device that uses an accelerator. [Figure 20A] 1 represents a conventional multiplier. [Figure 20B] 1 represents a conventional multiplier. [Figure 21] 1 illustrates one embodiment of a dot product multiplier. [Figure 22] 10 illustrates another embodiment of a dot product multiplier. [Figure 23] FIG. 10 is a flow diagram illustrating one embodiment of a process for performing a multiplication operation. [Figure 24] FIG. 10 is a flow diagram illustrating one embodiment of a process for performing a dot-product multiplication operation. DETAILED DESCRIPTION OF THE INVENTION

[0005] In an embodiment, the accelerator has multiplication hardware that operates in either conventional mode or dot product mode, where multiplication stages within the multiplication hardware are configured as dot products of multiple N-bit vectors to perform addition operations on results of N×N multiplication operations.

[0006] In the following description, numerous specific details are set forth to provide a more thorough understanding. However, as will be apparent to one skilled in the art, the embodiments described herein may be practiced without one or more of these specific details. In other instances, well-known technology has not been described so as not to obscure the details of the embodiments.

[0007] [System Overview] 1 is a block diagram of a processing system 100 according to an embodiment. System 100 may include multiple processors 102 or processor cores 107 and may be used in a single-processor desktop system, a multiprocessor workstation system, or a server system. In one embodiment, system 100 is a processing platform embedded in a system-on-a-chip (SoC) integrated circuit for use in mobile, portable, or embedded devices, such as in Internet of Things (IoT) devices with wired or wireless connectivity to local or wide area networks.

[0008] In one embodiment, system 100 may include, be coupled to, or be incorporated into a server-based gaming platform; a gaming console, including game and media consoles; a mobile gaming console, a handheld gaming console, or an online gaming console. In some embodiments, system 100 is part of a mobile phone, a smartphone, a tablet computing device, or a mobile Internet-connected device, such as a laptop with low internal storage capacity. Processing system 100 may also include, be coupled to, or be incorporated into a wearable device, such as a smartwatch wearable device; augmented reality (AR) or virtual reality (VR)-augmented smart eyewear or clothing that provides visual, audio, or haptic output to supplement a real-world visual, audio, or haptic experience, or otherwise provides text, audio, graphics, video, holographic images or video, or haptic feedback; other augmented reality (AR) devices; or other virtual reality (VR) devices. In some embodiments, processing system 100 includes or is part of a television or set-top box device. In one embodiment, system 100 may include, be coupled to, or be incorporated into an autonomous vehicle, such as a bus, a tractor-trailer, an automobile, a motorcycle or electric bicycle, an airplane or glider (or any combination thereof), etc. The autonomous vehicle may use system 100 to process the sensed environment around the vehicle.

[0009] In some embodiments, one or more processors 102 each include one or more processor cores 107 to process instructions that, when executed, perform operations for system or user software. In some embodiments, at least one of the one or more processor cores 107 is configured to process a particular instruction set 109. In some embodiments, the instruction set 109 may facilitate Complex Instruction Set Computing (CISC), Reduced Instruction Set Computing (RISC), or Very Long Instruction Word (VLIW) computations. One or more processor cores 107 may process another instruction set 109, which may include instructions that facilitate emulation of other instruction sets. The processor cores 107 may also include other processing devices, such as a digital signal processor (DSP).

[0010] In some embodiments, processor 102 includes cache memory 104. Depending on the architecture, processor 102 may have a single internal cache or multiple levels of internal cache. In some embodiments, cache memory is shared among various components of processor 102. In some embodiments, processor 102 also uses an external cache (e.g., a level 3 (L3) cache or a last level cache (LLC)) that may be shared among processor cores 107 using known cache coherency techniques. Processor 102 may also include register file 106. Register file 106 may include various types of registers that store various types of data (e.g., integer registers, floating-point registers, status registers, and instruction pointer registers). Some registers may be general-purpose registers, while other registers may be specific to the design of processor 102.

[0011] In some embodiments, one or more processors 102 are coupled to one or more interface buses 110 to carry communication signals, such as address, data, or control signals, between the processors 102 and other components in the system 100. The interface bus 110, in one embodiment, can be a processor bus such as a version of a Direct Media Interface (DMI) bus. However, the processor bus is not limited to a DMI bus and may include one or more Peripheral Component Interconnect buses (e.g., PCI, PCI Express), memory buses, or other types of interface buses. In one embodiment, the processor 102 includes an integrated memory controller 116 and a platform controller hub 130. The memory controller 116 facilitates communication between memory devices and other components of the system 100, while the platform controller hub (PCH) 130 provides connectivity to I / O devices via a local I / O bus.

[0012] Memory device 120 can be a dynamic random access memory (DRAM) device, a static random access memory (SRAM) device, a flash memory device, a phase-change memory device, or other memory device with suitable performance to serve as process memory. In one embodiment, memory device 120 can operate as system memory for system 100, storing data 122 and instructions 121 used when one or more processors 102 execute applications or processes. Memory controller 116 also couples to an optional external graphics processor 118. External graphics processor 118 may communicate with one or more graphics processors 108 within processor 102 to perform graphics and media operations. In some embodiments, graphics, media, and computation operations may be assisted by accelerator 112, which is a co-processor that can be configured to perform specialized sets of graphics, media, or computation operations. For example, in one embodiment, accelerator 112 is a matrix multiplication accelerator used to optimize machine learning or computation operations. In one embodiment, accelerator 112 is a ray tracing accelerator that may be used to perform ray tracing operations in conjunction with graphics processor 108. In one embodiment, external accelerator 119 may be used instead of or in conjunction with accelerator 112.

[0013] In some embodiments, a display device 111 may be connected to the processor 102. The display device 111 may be one or more of an internal display device such as found in a mobile electronic device or laptop device, or an external display device attached via a display interface (e.g., DisplayPort, etc.). In one embodiment, the display device 111 may be a head-mounted display (HMD), such as a stereoscopic display device used in virtual reality (VR) or augmented reality (AR) applications.

[0014] In some embodiments, the platform controller hub 130 allows peripherals to connect to the memory device 120 and the processor 102 via a high-speed I / O bus. The I / O peripherals include, without limitation, an audio controller 146, a network controller 134, a firmware interface 128, a wireless transceiver 126, a touch sensor 125, and a data storage device 124 (e.g., non-volatile memory, volatile memory, hard disk drive, flash memory, NAND, 3D NAND, 3D XPoint, etc.). The data storage device 124 can be connected via a storage interface (e.g., SATA) or via a peripheral bus such as a Peripheral Component Interconnect bus (PCI, PCI Express). The touch sensor 125 can include a touchscreen sensor, a pressure sensor, or a fingerprint sensor. The wireless transceiver 126 can be a Wi-Fi transceiver, a Bluetooth transceiver, or a mobile network transceiver such as a 3G, 4G, 5G, or Long Term Evolution (LTE) transceiver. Firmware interface 128 enables communication with system firmware and can be, for example, a Unified Extensible Firmware Interface (UEFI). Network controller 134 can enable network connectivity to a wired network. In some embodiments, a high-performance network controller (not shown) couples to interface bus 110. Audio controller 146, in one embodiment, is a multi-channel high-definition audio controller. In one embodiment, system 100 includes an optional legacy I / O controller 140 that couples legacy (e.g., Personal System 2 (PS / 2)) devices to the system. Platform controller hub 130 can also connect to one or more universal serial bus (USB) controllers 142 connected input devices, such as a keyboard and mouse 143 combination, a camera 144, or other USB input devices.

[0015] It will be understood that the illustrated system 100 is exemplary and not limiting, as other types of data processing systems configured differently may also be used. For example, instances of memory controller 116 and platform controller hub 130 may be incorporated into a separate external graphics processor, such as external graphics processor 118. In one embodiment, platform controller hub 130 and / or memory controller 116 may be external to one or more processors 102. For example, system 100 may include external memory controller 116 and platform controller hub 130, which may be configured as a memory controller hub and peripheral controller hub within a system chipset that communicates with the processor.

[0016] For example, circuit boards ("sleds"), upon which components such as CPUs, memory, and other components are located, are designed for improved thermal performance. In some examples, processing components such as processors are located on the top surface of the sled, while near memory such as DIMMs are located on the bottom surface of the sled. As a result of the enhanced airflow provided by this design, components can operate at higher frequencies and power levels in a typical system, improving performance. Furthermore, sleds are configured to blindly mate with power and data communication cables in a rack, thereby enhancing their ability to be quickly removed, upgraded, reinstalled, and / or replaced. Similarly, individual components located on a sled, such as processors, accelerators, memory, and data storage drives, are configured to be easily upgraded by being widely spaced from one another. In an illustrative embodiment, the components further include hardware attestation mechanisms to prove their authenticity.

[0017] Data centers can utilize a single network architecture (a "fabric") that supports multiple other network architectures, including Ethernet and Omni-Path. Threads can be coupled to switches via optical fiber. Optical fiber provides higher bandwidth and lower latency than typical twisted pair cabling (Category 5, Category 5e, Category 6, etc.). With a high-bandwidth, low-latency interconnect and network architecture, data centers can pool resources such as memory, accelerators (e.g., GPUs, graphics accelerators, FPGAs, ASICs, neural network and / or artificial intelligence accelerators, etc.), and data storage drives that are physically disaggregated during use and provide them to computing resources (e.g., processors) as needed, allowing the computing resources to access the pooled resources as if they were local.

[0018] A power supply or source can provide voltage and / or current to system 100 or any component or system described herein. In one example, the power supply includes an AC / DC (alternating current to direct current) adapter that plugs into a wall outlet input. Such AC power can be a renewable energy (e.g., solar) source. In one example, the power source includes a DC power source, such as an external AC / DC converter. In one example, the power source or power supply includes wireless charging hardware that charges due to proximity to a charging field. In one example, the power source can include an internal battery, an AC supply, a motion-based power supply, a solar power source, or a fuel cell power source.

[0019] 2A-2D depict computer systems and graphics processors provided by embodiments described herein. Elements in Figures 2A-2D that have the same reference numbers (or names) as elements in any other figure in the present application can operate or function similarly as described elsewhere herein, but are not limited to such.

[0020] FIG. 2A is a block diagram of an embodiment of a processor 200 that includes one or more processor cores 202A-202N, an integrated memory controller 214, and an integrated graphics processor 208. The processor 200 may include additional cores, up to additional core 202N, represented by a dashed box. The processor cores 202A-202N include one or more internal cache units 204A-204N. In some embodiments, each processor core also has access to one or more shared cache units 206. The internal cache units 204A-204N and the shared cache unit 206 represent a cache memory hierarchy within the processor 200. The cache memory hierarchy may include at least one level of instruction and data cache within each processor core and one or more levels of shared mid-level cache, such as a level 2 (L2), level 3 (L3), level 4 (L4), or other level cache, with the highest level cache before external memory being classified as LLC. In some embodiments, cache coherency logic maintains coherency between the various cache units 206 and 204A-204N.

[0021] In some embodiments, processor 200 may also include a set of one or more bus controller units 216 and a system agent core 210. The one or more bus controller units 216 manage a set of peripheral buses, such as one or more PCI or PCI Express buses. System agent core 210 provides management functions for various processor components. In some embodiments, system agent core 210 includes one or more integrated memory controllers 214 to manage access to various external memory devices (not shown).

[0022] In some embodiments, one or more of processor cores 202A-202N include support for synchronous multithreading. In such embodiments, system agent core 210 includes components that coordinate and operate cores 202A-202N during multithreaded processing. System agent core 210 may further include a power control unit (PCU). The PCU includes logic and components that adjust the power state of processor cores 202A-202N and graphics processor 208.

[0023] In some embodiments, processor 200 further includes a graphics processor 208 to perform graphics processing operations. In some embodiments, graphics processor 208 couples to a set of shared cache unit 206 and a system agent core 210 that includes one or more integrated memory controllers 214. In some embodiments, system agent core 210 also includes a display controller 211 to drive graphics processor output to one or more coupled displays. In some embodiments, display controller 211 may be a separate module coupled to the graphics processor via at least one interconnect, or may be incorporated within graphics processor 208.

[0024] In some embodiments, a ring-based interconnect unit 212 is used to couple the internal components of processor 200. However, alternative interconnect units, such as point-to-point interconnects, switched interconnects, or other technologies, including those well known in the art, may be used. In some embodiments, graphics processor 208 couples to ring interconnect 212 via I / O link 213.

[0025] Exemplary I / O link 213 represents at least one of a wide variety of I / O interconnects, including a package I / O interconnect that facilitates communication between various processor components and a high-performance embedded memory module 218, such as an eDRAM module. In some embodiments, each of processor cores 202A-202N and graphics processor 208 can use embedded memory module 218 as a shared last-level cache.

[0026] In some embodiments, processor cores 202A-202N are homogenous cores that execute the same instruction set architecture. In other embodiments, processor cores 202A-202N are heterogeneous with respect to instruction set architecture (ISA), where one or more of processor cores 202A-202N execute a first instruction set, while at least one of the remaining cores executes a subset of the first instruction set or a different instruction set. In one embodiment, processor cores 202A-202N are heterogeneous with respect to microarchitecture, where one or more cores having relatively higher power consumption are combined with one or more cores having relatively lower power consumption. In one embodiment, processor cores 202A-202N are heterogeneous with respect to computational capability. Furthermore, processor 200 may be implemented on one or more chips or as an SoC integrated circuit having the depicted components in addition to other components.

[0027] FIG. 2B is a block diagram of hardware logic for a graphics processor core 219 according to some embodiments described herein. Elements of 2B having the same reference number (or name) as elements in any other figure herein can operate or function similarly as described elsewhere herein, but are not limited to such. Graphics processor core 219, sometimes referred to as a core slice, can be one or more graphics cores within a modular graphics processor. Graphics processor core 219 is an example of one graphics core slice; the graphics processors described herein may include multiple graphics core slices based on target power and performance envelopes. Each graphics processor core 219 may include a fixed-function block 230 coupled with multiple sub-cores 221A-221F, also referred to as sub-slices, that include modular blocks of general-purpose fixed-function logic.

[0028] In some embodiments, fixed function block 230 includes a geometry / fixed function pipeline 231 that may be shared by all sub-cores within graphics processor core 219, for example, in lower performance and / or lower power graphics processor implementations. In various embodiments, geometry / fixed function pipeline 231 includes a 3D fixed function pipeline (e.g., 3D pipeline 312 seen in FIGS. 3 and 4, described below), a video front end unit, a thread spawner and thread dispatcher, and a unified return buffer manager that manages a unified return buffer (e.g., unified return buffer 418 in FIG. 4, described below).

[0029] In one embodiment, fixed function block 230 also includes a graphics SoC interface 232, a graphics microcontroller 233, and a media pipeline 234. Graphics SoC interface 232 provides an interface between graphics processor core 219 and other processor cores within the SoC integrated circuit. Graphics microcontroller 233 is a programmable sub-processor that can be configured to manage various functions of graphics processor core 219, including thread dispatch, scheduling, and pre-emption. Media pipeline 234 (e.g., media pipeline 316 in FIGS. 3 and 4) includes logic that facilitates decoding, encoding, pre-processing, and / or post-processing of multimedia data, including image and video data. Media pipeline 234 implements media operations by requesting computation or sampling logic within sub-cores 221A-221F.

[0030] In one embodiment, SoC interface 232 enables graphics processor core 219 to communicate with general-purpose application processor cores (e.g., CPUs) and / or other components within the SoC, including memory hierarchy elements such as shared last-level cache memory, system RAM, and / or embedded on-chip or on-package DRAM. SoC interface 232 may also enable communication with fixed-function devices within the SoC, such as a camera imaging pipeline, and enable and / or implement the use of global memory atomics that may be shared between graphics processor core 219 and CPUs within the SoC. SoC interface 232 may also implement power management controls for graphics processor core 219 and enable an interface between the clock domain of graphics processor core 219 and other clock domains within the SoC. In one embodiment, SoC interface 232 enables receipt of command buffers from a command streamer and global thread dispatcher configured to supply commands and instructions to each of one or more graphics cores within the graphics processor. Commands and instructions can be dispatched to the media pipeline 234 when a media operation is to be performed, or to the geometry and fixed function pipelines (e.g., geometry and fixed function pipeline 231, geometry and fixed function pipeline 237) when a graphics processing operation is to be performed.

[0031] Graphics microcontroller 233 may be configured to perform various scheduling and management tasks for graphics processor core 219. In one embodiment, graphics microcontroller 233 can perform graphics and / or compute workload scheduling to various graphics parallel engines within execution unit (EU) arrays 222A-222F, 224A-224F within sub-cores 221A-221F. In this scheduling model, host software executing on a CPU core of an SoC including graphics processor core 219 can submit a workload to one of multiple graphics processor docks, which invokes scheduling operations on the appropriate graphics engine. Scheduling operations include determining which workload to run next, submitting the workload to the command streamer, preempting existing workloads running on the engines, monitoring the progress of the workloads, and notifying host software when the workloads are complete. In one embodiment, graphics microcontroller 233 can also facilitate low power or idle states for graphics processor core 219, giving graphics processor core 219 the ability to save and restore registers within graphics processor core 219 across low power state transitions independent of the operating system and / or graphics driver software on the system.

[0032] Graphics processor core 219 may have up to N modular sub-cores, more or less than the depicted sub-cores 221A-221F. For each set of N sub-cores, graphics processor core 219 may also include shared function logic 235, shared and / or cache memory 236, geometry / fixed function pipeline 237, and additional fixed function logic 238 to accelerate various graphics and computational operations. Shared function logic 235 may include logic units associated with the shared function logic of FIG. 4 (e.g., sampler, math, and / or inter-thread communication logic) that may be shared by each of the N sub-cores in graphics processor core 219. Shared and / or cache memory 236 may be a last-level cache for the set of N sub-cores 221A-221F in graphics processor core 219 and may also function as a shared memory accessible by multiple sub-cores. Geometry / fixed-function pipeline 237 may be included in place of geometry / fixed-function pipeline 231 in fixed-function block 230 and may include the same or similar logic units.

[0033] In one embodiment, graphics processor core 219 includes additional fixed-function logic 238, which can include various fixed-function acceleration logic used by graphics processor core 219. In one embodiment, additional fixed-function logic 238 includes an additional geometry pipeline used in position-only shading. In position-only shading, there are two geometry pipelines: a full geometry pipeline within geometry / fixed-function pipelines 237, 231, and a cull pipeline, an additional geometry pipeline that can be included within additional fixed-function logic 238. In one embodiment, the cull pipeline is a subdivided version of the full geometry pipeline. The full pipeline and the cull pipeline can run different instances of the same application, with each instance having a separate context. Position-only shading can complete shading faster in some cases because it can hide long cull runs of discarded triangles. For example, in one embodiment, the cull pipeline logic within additional fixed-function logic 238 can execute position shaders in parallel with the main application, generally producing significant results faster than the full pipeline. This is because the cull pipeline only fetches and shades vertex position attributes without rasterizing and rendering pixels to the frame buffer. The cull pipeline can use the generated nontrivial results to calculate visibility information for all triangles, regardless of whether they are culled or not. The full pipeline (which in this example may also be called the replay pipeline) can consume the visibility information to skip culled triangles, so as to shade only visible triangles that are eventually passed to the rasterization phase.

[0034] In one embodiment, the additional fixed function logic 238 may also include machine learning acceleration logic, such as fixed function matrix multiplication logic, for implementations that include optimizations for machine learning training or inference.

[0035] Within each graphics sub-core 221A-221F is a set of execution resources that can be used to perform graphics, media, and compute operations in response to requests by the graphics pipeline, media pipeline, or shader pipeline. The graphics sub-cores 221A-221F include a plurality of EU arrays 222A-222F, 224A-224F, thread dispatch and inter-thread communication (TD / IC) logic 223A-223F, 3D (e.g., texture) samplers 225A-225F, media samplers 206A-206F, shader processors 227A-227F, and shared local memories (SLMs) 228A-228F. The EU arrays 222A-222F, 224A-224F each include multiple execution units, which are general-purpose graphics processing units capable of performing floating-point and integer / fixed-point logic operations in service of graphics, media, or compute operations, including graphics, media, or compute shader programs. The TD / IC logic 223A-223F performs local thread dispatch and thread control operations for the execution units within the sub-core and facilitates communication between threads executing on the sub-core's execution units. The 3D samplers 225A-225F can load textures or other 3D graphics-related data into memory. The 3D samplers can read texture data differently based on the texture format and configured sample state associated with a given texture. The media samplers 206A-206F can perform similar read operations based on the type and format associated with the media data. In one embodiment, each graphics sub-core 221A-221F can alternatively include an integrated 3D and media sampler. Threads executing in each execution unit of sub-cores 221A-221F can use shared local memory 228A-228F within each sub-core, so that threads executing within a thread group can execute using a common pool of on-chip memory.

[0036] 2C depicts a graphics processing unit (GPU) 239 that includes a dedicated set of graphics processing resources arranged in multi-core groups 240A-240N. Although details of a single multi-core group 240A are given, it will be understood that other multi-core groups 240B-240N may be equipped with the same or similar sets of graphics processing resources.

[0037] As shown, multi-core group 240A may include a set of graphics cores 243, a set of tensor cores 244, and a set of ray tracing cores 245. Scheduler / dispatcher 241 schedules and dispatches graphics threads to execute on the various cores 243, 244, 245. A set of register files 242 stores operand values ​​used by cores 243, 244, 245 when executing the graphics threads. These may include, for example, integer registers for storing integer values, floating-point registers for storing floating-point values, vector registers for storing packed data elements (integer and / or floating-point data elements), and tile registers for storing tensor / matrix values. In one embodiment, the tile registers are implemented as a combined set of vector registers.

[0038] One or more combined level 1 (L1) cache and shared memory units 247 store graphics data such as texture data, vertex data, pixel data, ray data, bounding volume data, etc. locally within each multicore group 240A. One or more texture units 247 may also be used to perform texturing operations such as texture mapping and sampling. A level 2 (L2) cache 253, shared by all or some of the multicore groups 240A-240N, stores graphics data and / or instructions for multiple concurrent graphics threads. As shown, the L2 cache 253 may be shared across multiple multicore groups 240A-240N. One or more memory controllers 248 couple the GPU 239 to memory 249. The memory 249 may be system memory (e.g., DRAM) and / or dedicated graphics memory (e.g., GDDR6 memory).

[0039] Input / output (I / O) circuitry 250 couples GPU 239 to one or more I / O devices 252, such as a digital signal processor (DSP), a network controller, or a user input device. An on-chip interconnect may be used to couple I / O device 252 to GPU 239 and memory 249. One or more I / O memory management units (IOMMUs) 251 of I / O circuitry 250 directly couple I / O device 252 to system memory 249. In one embodiment, IOMMU 251 manages multiple sets of page tables to map virtual addresses to physical addresses in system memory 249. In this embodiment, I / O device 252, CPU 246, and GPU 239 may share the same virtual address space.

[0040] In one implementation, IOMMU 251 supports virtualization. In this case, it may manage a first set of page tables that map guest / graphics virtual addresses to guest / graphics physical addresses and a second set of page tables that map guest / graphics physical addresses to system / host physical addresses (e.g., in system memory 249). The base addresses of each of the first and second sets of page tables are stored in control registers and may be swapped out on a context switch (e.g., so that a new context is given access to the associated set of page tables). Although not shown in FIG. 2C , each of cores 243, 244, 245 and / or multicore groups 240A-240N may include a translation lookaside buffer (TLB) that caches guest-virtual-to-guest-physical, guest-physical-to-host-physical, and guest-virtual-to-host-physical translations.

[0041] In one embodiment, CPU 246, GPU 239, and I / O devices 252 are integrated onto a single semiconductor chip and / or chip package. Memory 249 is shown as being integrated onto the same chip or may be coupled to memory controller 248 via an off-chip interface. In one embodiment, memory 249 comprises GDDR6 memory that shares the same virtual address space as other physical system-level memory. However, the underlying principles of the present invention are not limited to this specific implementation.

[0042] In one embodiment, tensor cores 244 include multiple execution units specifically designed to perform matrix operations, which are fundamental computational operations used to perform deep learning computations. For example, simultaneous matrix multiplication operations may be used for neural network training and inference. Tensor cores 244 may perform matrix processing using a variety of operand precisions, including single-precision floating-point (e.g., 32 bits), half-precision floating-point (e.g., 16 bits), integer word (16 bits), byte (8 bits), and half-byte (4 bits). In one embodiment, the neural network implementation extracts features from each rendered scene, potentially combining details from multiple frames to construct a high-quality final image.

[0043] In deep learning implementations, parallel matrix computation work may be scheduled for execution on tensor cores 244. Neural network training, in particular, requires efficient matrix dot product operations. To process the inner product formulation of N×N×N matrix multiplication, tensor cores 244 may include at least N dot product processing elements. Before the matrix multiplication begins, one entire matrix is ​​loaded into a tile register, and at least one column of a second matrix is ​​loaded for each of the N cycles. For each cycle, there are N dot products to be processed.

[0044] Matrix elements may be stored in different precisions depending on the particular implementation, including 16-bit words, 8-bit bytes (e.g., INT8), and 4-bit half-bytes (e.g., INT4). Different precision modes may be specified for tensor cores 244 to ensure that the most efficient precision is used for different workloads (e.g., inferring workloads that can tolerate quantization to bytes and half-bytes, etc.).

[0045] In one embodiment, ray tracing core 245 accelerates ray tracing operations for both real-time and non-real-time ray tracing implementations. In particular, ray tracing core 245 includes ray traversal / intersection circuitry that performs ray traversal using a bounding volume hierarchy (BVH) and identifies intersections between rays and primitives within the BVH volume. Ray tracing core 245 may also include circuitry that performs depth testing and culling (e.g., using a Z-buffer or similar arrangement). In one implementation, ray tracing core 245 performs traversal and intersection operations in conjunction with the image denoising techniques described herein. At least a portion of the image denoising techniques may be executed by tensor core 244. For example, in one embodiment, tensor core 244 implements a deep learning neural network to perform denoising of frames generated by ray tracing core 245. It should be noted that the CPU 246, graphics core 243, and / or ray tracing core 245 may also implement all or part of the denoising and / or deep learning algorithms.

[0046] Additionally, as described above, a distributed approach to denoising may be used, where GPU 239 is in a computing device coupled to other computing devices via a network or high-speed interconnect. In this embodiment, the interconnected computing devices share neural network learning / training data to perform denoising for different types of image frames and / or different graphics applications, improving the speed at which the entire system learns.

[0047] In one embodiment, ray tracing core 245 handles all BVH traversal and ray primitive intersections, preventing the graphics core 243 from becoming overloaded with thousands of instructions per ray. In one embodiment, each ray tracing core 245 includes a first set of dedicated circuitry to perform bounding box tests (e.g., for traversal operations) and a second set of dedicated circuitry to perform ray triangle intersection tests (traversed intersecting rays). Thus, in one embodiment, multi-core group 240A can simply launch ray probes, and ray tracing core 245 independently performs ray traversal and intersection and returns hit data (e.g., hit, no hit, multiple hits, etc.) to the thread context. While ray tracing core 245 performs traversal and intersection operations, the other cores 243, 244 are freed to perform other graphics or computational work.

[0048] In one embodiment, each ray tracing core 245 includes a traversal unit that performs BVH test operations and an intersection unit that performs ray primitive intersection tests. The intersection unit generates a "hit," "no hit," or "multi-hit" response, which is delivered to the appropriate thread. During traversal and intersection operations, the execution resources of other cores (e.g., graphics core 243 and tensor core 244) are freed to perform other forms of graphics work.

[0049] In one particular embodiment described below, a hybrid rasterization / ray tracing approach is used, with work distributed between the graphics core 243 and the ray tracing core 245.

[0050] In one embodiment, the ray tracing core 245 (and / or other cores 243, 244) includes hardware support for a ray tracing instruction set such as Microsoft's DirectX Ray Tracing (DXR), including ray-generation, closest-hit, any-hit, and miss shaders and a DispatchRays command, which enables allocation of a unique set of shaders and textures per object. Another ray tracing platform that may be supported by the ray tracing core 245, graphics core 243, and tensor core 244 is Vulkan 1.1.85. Note that the principles underlying the present invention are not limited to any particular ray tracing ISA.

[0051] In general, the various cores 245, 244, 243 may support a ray tracing instruction set that includes instructions / functions for ray-generation, closest-hit, any-hit, ray-primitive intersection, per-primitive and hierarchical bounding box construction, miss, visit, and exceptions. More specifically, one embodiment includes ray tracing instructions that perform the following functions:

[0052] ray-generation: Ray-generation instructions can be executed for each pixel, sample, or other user-defined allocation of work.

[0053] closest-hit: The closest-hit command can be executed to locate the closest intersection of the ray with a primitive in the scene.

[0054] any-hit: The any-hit directive identifies multiple intersections between primitives and rays in the scene to potentially identify a new closest intersection point.

[0055] The intersection:intersection command performs a ray-primitive intersection test and outputs the result.

[0056] Per-primitive bounding box construction: This instruction creates a bounding box around a given primitive or group of primitives (e.g., when constructing a new BVH or other acceleration data structure).

[0057] miss: indicates that the ray misses all geometry in the scene or a specialized region of the scene.

[0058] visit: indicates the child volume that the ray traverses.

[0059] exceptions: Indicates different types of exception handlers (e.g., called for different error conditions).

[0060] FIG. 2D illustrates a general purpose graphics processing unit (GPGPU) that may be configured as a graphics processor and / or a computational accelerator in accordance with embodiments described herein. 2 is a block diagram of GPGPU 270. GPGPU 270 can be interconnected with a host processor (e.g., one or more CPUs 246) and memories 271, 272 via one or more system and / or memory buses. In one embodiment, memory 271 is system memory that can be shared with one or more CPUs 246, while memory 272 is device memory that is dedicated to GPGPU 270. In one embodiment, components within GPGPU 270 and device memory 272 can be mapped to memory addresses accessible by one or more CPUs 246. Access to memories 271 and 272 can be facilitated via memory controller 268. In one embodiment, memory controller 268 includes an internal direct memory access (DMA) controller 269 or can include logic to perform operations that would otherwise be performed by a DMA controller.

[0061] The GPGPU 270 includes multiple cache memories, including an L2 cache 253, an L1 cache 254, an instruction cache 255, and a shared memory 256, some of which may also be partitioned as cache memory. The GPGPU 270 also includes multiple compute units 260A-260N. Each compute unit 260A-260N includes vector registers 261, scalar registers 262, a vector logic unit 263, and a scalar logic unit 264. The compute units 260A-260N may also include a local shared memory 265 and a program counter 266. The compute units 260A-260N may be coupled with a constant cache 267, which may be used to store constant data that does not change during execution of a kernel or shader program executed by the GPGPU 270. In one embodiment, the constant cache 267 is a scalar data cache, and cached data may be fetched directly into the scalar registers 262.

[0062] During operation, one or more CPUs 246 can write commands to registers or memory within GPGPU 270 that are mapped to the accessible address space. Command processor 257 can read commands from registers or memory and determine how those commands are processed within GPGPU 270. Thread dispatcher 258 can then be used to dispatch threads to compute units 260A-260N to execute those commands. Each compute unit 260A-260N can execute threads independently of the other compute units. Additionally, each compute unit 260A-260N can be independently configured for conditional computation and can conditionally output the results of the computation to memory. Command processor 257 can interrupt one or more CPUs 246 when a submitted command completes.

[0063] 3A-3C depict block diagrams of further graphics processor and computation accelerator architectures provided by embodiments described herein. Elements in Figures 3A-3C having the same reference numbers (or names) as elements in any other figure in the present application can operate or function similarly as described elsewhere herein, but are not limited to such.

[0064] 3A is a block diagram of graphics processor 300. Graphics processor 300 may be a discrete graphics processing unit or may be a graphics processor integrated with multiple processing cores or other semiconductor devices, such as, without limitation, a memory device or a network interface. In some embodiments, the graphics processor communicates with registers on the graphics processor through a memory-mapped I / O interface and with commands placed in processor memory. In some embodiments, graphics processor 300 may interface to local memory, one or more internal caches, one or more shared external caches, and / or system memory.

[0065] In some embodiments, graphics processor 300 also includes a display controller 302 that drives display output data to a display device 318. Display controller 302 includes hardware for one or more overlay planes for displaying and compositing multiple layers of video or user interface elements. Display device 318 can be an internal or external display device. In one embodiment, display device 318 is a head-mounted display device, such as a virtual reality (VR) display device or an augmented reality (AR) display device. In some embodiments, graphics processor 300 includes a video codec engine 306 to encode media to, decode from, or transcode media between one or more media coding formats, including but not limited to, Moving Picture Experts Group (MPEG) formats such as MPEG-2, Advanced Video Coding (AVC) formats such as H.264 / MPEG-4 AVC, H.265 / HEVC, Alliance for Open Media (AOMedia) VP8, VP9, ​​and Society of Motion Picture & Television Engineers (SMPTE) 421M / VC1, and Joint Photographic Experts Group (JPEG) formats such as JPEG and Motion JPEG (MJPEG) formats.

[0066] In some embodiments, graphics processor 300 includes a block image transfer (BLIT) engine 304 to perform two-dimensional (2D) rasterizer operations, including, for example, bit-boundary block transfers. Note that in one embodiment, 2D graphics operations are performed using one or more components of a graphics processing engine (GPE) 310. In some embodiments, GPE 310 is a computation engine for performing graphics operations, including three-dimensional (3D) graphics operations and media operations.

[0067] In some embodiments, GPE 310 includes a 3D pipeline 312 for performing 3D operations such as rendering three-dimensional images and scenes using processing functions that operate on 3D primitive shapes (e.g., rectangles, triangles, etc.). 3D pipeline 312 includes programmable fixed-function elements that spawn execution threads to 3D / media subsystem 315 and / or perform various tasks within the elements. While 3D pipeline 312 can be used to perform multiple operations, embodiments of GPE 310 also include a media pipeline 316 that is specifically used to perform media operations such as video post-processing and image enhancement.

[0068] In some embodiments, the media pipeline 316 includes a fixed function or programmable logic unit to perform one or more specialized media operations, such as video decode acceleration, video deinterlacing, and video encode acceleration, instead of or for the video codec engine 306. In some embodiments, the media pipeline 316 further includes a thread generation unit to generate threads for execution in the 3D / media subsystem 315. The generated threads perform computations for the media operations in one or more graphics execution units included in the 3D / media subsystem 315.

[0069] In some embodiments, the 3D / media subsystem 315 includes logic for executing threads created by the 3D pipeline 312 and the media pipeline 316. In one embodiment, the pipelines send thread execution requests to the 3D / media subsystem 315. The 3D / media subsystem 315 includes thread dispatch logic that arbitrates and dispatches various requests to available thread execution resources. The execution resources include an array of graphics execution units that process the 3D and media threads. In some embodiments, the 3D / media subsystem 315 includes one or more internal caches for thread instructions and data. In some embodiments, the subsystem also includes shared memory, including registers and addressable memory, for supplying data between threads and for storing output data.

[0070] FIG. 3B illustrates a graphics processor 320 having a tiled architecture according to an embodiment described herein. In one embodiment, the graphics processor 320 includes a graphics processing engine cluster 322 having multiple instances of the graphics processing engine 310 of FIG. 3 within graphics engine tiles 310A-310D. Each graphics engine tile 310A-310D may be interconnected via a set of tile interconnects 323A-323F. Each graphics engine tile 310A-310D may also be connected to memory modules or memory devices 326A-326D via memory interconnects 325A-325D. The memory devices 326A-326D may use any graphics memory technology. For example, the memory devices 326A-326D may be graphics double data rate (GDDR) memory. In one embodiment, memory devices 326A-326D are high-bandwidth memory (HMB) modules that may be on-die with their respective graphics engine tiles 310A-310D. In one embodiment, memory devices 326A-326D are stacked memory devices that may be stacked on top of their respective graphics engine tiles 310A-310D. In one embodiment, each graphics engine tile 310A-310D and associated memory 326A-326D resides on a separate chiplet that is bonded to a base die or base substrate, as described in further detail in FIGS. 11B-11D.

[0071] The graphics processing engine cluster 322 may be connected to an on-chip or on-package fabric interconnect 324. The fabric interconnect 324 may enable communication between the graphics engine tiles 310A-310D and components such as the video codec engine 306 and one or more copy engines 304. The copy engine 304 may be used to move data to, from, and between the memory devices 326A-326D and memory external to the graphics processor 320 (system memory). The fabric interconnect 324 may also be used to interconnect the graphics engine tiles 310A-310D. The graphics processor 320 may optionally include a display controller 302 to enable connection to an external display device 318. The graphics processor may also be configured as a graphics or computation accelerator. In an accelerator configuration, the display controller 302 and display device 318 may be omitted.

[0072] Graphics processor 320 may be connected to a host system via host interface 328. Host interface 328 may enable communication between graphics processor 320, system memory, and / or other system components. Host interface 328 may be, for example, a PCI Express bus or other type of host system interface.

[0073] FIG. 3C illustrates a compute accelerator 330 according to an embodiment described herein. The compute accelerator 330 may include architectural similarities to the graphics processor 320 of FIG. 3B and is optimized for compute acceleration. The compute engine cluster 332 may include a set of compute engine tiles 340A-340D that include execution logic optimized for parallel or vector-based general-purpose computational operations. In some embodiments, the compute engine tiles 340A-340D do not include fixed-function graphics processing logic, but in one embodiment, one or more of the compute engine tiles 340A-340D may include logic to perform media acceleration. The compute engine tiles 340A-340D may be connected to memories 326A-326D via memory interconnects 325A-325D. The memories 326A-326D and memory interconnects 325A-325D may be of similar technology to that found in the graphics processor 320, or may be different. The graphics compute engine tiles 340A-340D may also be interconnected via a set of tile interconnects 323A-323F, which may be connected to and / or interconnected by the fabric interconnect 324. In one embodiment, the compute accelerator 330 includes a large L3 cache 336, which may be configured as a device-wide cache. The compute accelerator 330 may also be connected to a host processor and memory via a host interface 328, similar to the graphics processor 320 of FIG. 3B.

[0074] [Graphics processing engine] FIG. 4 is a block diagram of a graphics processing engine 410 of a graphics processor according to some embodiments. In one embodiment, the graphics processing engine (GPE) 410 is a variation of the GPE 310 shown in FIG. 3A and may represent the graphics engine tiles 310A-310D of FIG. 3B. Elements of FIG. 4 that have the same reference numbers (or names) as elements of any other figure herein can operate or function similarly as described elsewhere herein, but are not limited to such. For example, the 3D pipeline 312 and media pipeline 316 of FIG. 3A are depicted. The media pipeline 316 is optional in some embodiments of the GPE 410 and may not be explicitly included within the GPE 410. For example, in at least one embodiment, a separate media and / or image processor is coupled to the GPE 410.

[0075] In some embodiments, the GPE 410 is coupled to or includes a command streamer 403. The command streamer 403 provides a command stream to the 3D pipeline 312 and / or the media pipeline 316. In some embodiments, the command streamer 403 is coupled to memory. The memory can be system memory or one or more of an internal cache memory and a shared cache memory. In some embodiments, the command streamer 403 receives commands from memory and sends the commands to the 3D pipeline 312 and / or the media pipeline 316. The commands are instructions fetched from a ring buffer that stores commands for the 3D pipeline 312 and the media pipeline 316. In one embodiment, the ring buffer can further include a batch command buffer that stores batches of multiple commands. Commands for the 3D pipeline 312 can also include references to data stored in memory, such as, without limitation, vertex and geometry data for the 3D pipeline 312 and / or image data and memory objects for the media pipeline 316. The 3D pipeline 312 and the media pipeline 316 process commands and data by executing operations through logic within the respective pipelines or by dispatching one or more execution threads to the graphics core array 414. In one embodiment, the graphics core array 414 includes one or more blocks of graphics cores (e.g., graphics core 415A, graphics core 415B), each block including one or more graphics cores. Each graphics core includes a set of graphics execution resources including general-purpose and graphics-specific execution logic for performing graphics and computational operations, along with fixed-function texture processing and / or machine learning and artificial intelligence acceleration logic.

[0076] In various embodiments, 3D pipeline 312 may include fixed-function and programmable logic to process one or more shader programs, such as vertex shaders, geometry shaders, pixel shaders, fragment shaders, compute shaders, or other shader programs, by processing instructions and dispatching execution threads to graphics core array 414. Graphics core array 414 provides a unified block of execution resources used in processing those shader programs. Multipurpose execution logic (e.g., execution units) within graphics cores 415A-415B of graphics core array 414 includes support for various 3D API shader languages ​​and can execute multiple concurrent threads of execution associated with multiple shaders.

[0077] In some embodiments, graphics core array 414 includes execution logic that performs media functions such as video and / or image processing. In one embodiment, the execution unit includes general-purpose logic that is programmable to perform parallel general-purpose computing operations in addition to graphics processing operations. The general-purpose logic can perform processing operations in parallel with or together with the general-purpose logic in processor core 107 of FIG. 1 or cores 202A-202N of FIG. 2A.

[0078] Output data generated by threads executing in graphics core array 414 can output data to memory in unified return buffer (URB) 418. URB 418 can store data for multiple threads. In some embodiments, URB 418 can be used to route data between different threads executing in graphics core array 414. In some embodiments, URB 418 can also be used for synchronization between threads in graphics core array 414 and fixed function logic in shared function logic 420.

[0079] In some embodiments, graphics core array 414 is scalable such that the array includes a variable number of graphics cores, each having a variable number of execution units based on the target power and performance levels of GPE 410. In one embodiment, execution resources are dynamically scalable such that execution resources can be enabled or disabled as needed.

[0080] Graphics core array 414 couples to shared function logic 420, which includes multiple resources shared among the graphics cores in the graphics core array. The shared functions in shared function logic 420 are hardware logic units that provide specialized auxiliary functions to graphics core array 414. In various embodiments, shared function logic 420 includes, without limitation, sampler 421, mass 422, and inter-thread communication (ITU) 423 logic. Additionally, some embodiments implement one or more caches 425 within shared function logic 420.

[0081] Shared functionality is implemented at least when the demand for a given specialized function is insufficient for the instructions in the graphics core array 414. Instead, a single instantiation of that specialized function is implemented as a standalone entity in the shared function logic 420 and shared among the execution resources in the graphics core array 414. The exact set of functionality shared among and included within the graphics core array 414 varies with the embodiment. In some embodiments, certain shared functionality in the shared function logic 420 that is widely used by the graphics core array 414 may be included in the shared function logic 416 in the graphics core array 414. In various embodiments, the shared function logic 416 in the graphics core array 414 may include some or all of the logic in the shared function logic 420. In one embodiment, all of the logic elements in the shared function logic 420 may be duplicated in the shared function logic 416 of the graphics core array 414. In one embodiment, the shared functionality logic 420 is removed in favor of the shared functionality logic 416 of the graphics core array 414 .

[0082] [Execution Unit] 5A-5B illustrate thread execution logic 500 including an array of processing elements for use in a graphics processor according to embodiments described herein. Elements in FIGS. 5A-5B having the same reference numbers (or names) as elements in any other figures herein can operate or function similarly as described elsewhere herein, but are not limited to such. FIGS. 5A-5B illustrate an overview of thread execution logic 500, which may represent the hardware logic represented by each sub-core 221A-221F in FIG. 2B. FIG. 5A illustrates an execution unit in a general-purpose graphics processor, while FIG. 5B illustrates an execution unit that may be used in a computational accelerator.

[0083] 5A , in some embodiments, thread execution logic 500 includes a shader processor 502, a thread dispatcher 504, an instruction cache 506, a scalable execution unit array including multiple execution units 508A-508N, a sampler 510, a shared local memory 511, a data cache 512, and a data port 514. In one embodiment, the scalable execution unit array can be dynamically scaled by enabling or disabling one or more execution units (e.g., any of execution units 508A, 508B, 508C, 508D through 508N-1 and 508N) based on the computational requirements of a workload. In one embodiment, the included components are interconnected via an interconnect fabric that provides links to each of the components. In some embodiments, thread execution logic 500 includes one or more connections to memory, such as system memory or cache memory, through instruction cache 506, data port 514, sampler 510, and execution units 508A-508N. In some embodiments, each execution unit (e.g., 508A) is a standalone, programmable, general-purpose computational unit capable of executing multiple simultaneous hardware threads, processing multiple data elements in parallel for each thread. In various embodiments, the array of execution units 508A-508N is scalable to include any number of individual execution units.

[0084] In some embodiments, the execution units 508A-508N are primarily used to execute shader programs. The shader processor 502 processes various shader programs, and execution threads associated with the shader programs can be dispatched by the thread dispatcher 504. In one embodiment, the thread dispatcher includes logic to arbitrate thread initiation requests from the graphics and media pipelines and instantiate the requested threads on one or more of the execution units 508A-508N. For example, the geometry pipeline can dispatch vertex, tessellation, or geometry shaders to thread execution logic for processing. In some embodiments, the thread dispatcher 504 can also handle runtime thread creation requests from executing shader programs.

[0085] In some embodiments, the execution units 508A-508N support an instruction set that includes native support for many standard 3D graphics shader instructions, allowing shader programs from the graphics fabric (Direct3D and OpenGL) to execute with minimal translation. The execution units support vertex and geometry processing (e.g., vertex programs, geometry programs, vertex shaders), pixel processing (e.g., pixel shaders, fragment shaders), and general-purpose processing (e.g., compute and media shaders). Each of the execution units 508A-508N is capable of multi-issue single instruction multiple data (SIMD) execution, and multithreaded operation enables an efficient execution environment in the face of longer latency memory accesses. Each hardware thread within each execution unit has a dedicated high-bandwidth register file and associated independent thread state. Execution is multi-issue per clock for a pipeline capable of integer, single- and double-precision floating-point operations, SIMD branch functions, logical operations, transcendental operations, and various other operations. While waiting for data from memory or one of the shared functions, subordinate logic within execution units 508A-508N causes a waiting thread to suspend activity until the requested data is returned. While the waiting thread is paused, hardware resources may be dedicated to processing other threads. For example, during the delay associated with a vertex shader operation, the execution unit may execute operations for other types of shader programs, including pixel shaders, fragment shaders, or other vertex shaders. Various embodiments may be adapted to use execution through the use of single instruction multiple threads (SIMT) as an alternative to or in addition to the use of SIMD. References to SIMD cores or operations may also apply to SIMT or SIMD in addition to SIMT.

[0086] Each execution unit in execution units 508A-508N operates on an array of data elements. The number of data elements is the "execution size," or the number of channels for the instruction. An execution channel is a logical unit of execution for data element access, masking, and flow control within the instruction. The number of channels is independent of the number of physical ALUs (Arithmetic Logic Units) or FPUs (Floating Point Units) for a particular graphics processor. In some embodiments, execution units 508A-508N support integer and floating point data types.

[0087] The execution unit instruction set includes SIMD instructions. Various data elements may be stored as packed data types in registers, and the execution unit processes the various elements based on the data size of the elements. For example, when operating on a 256-bit wide vector, the 256 bits of the vector are stored in a register, and the execution unit operates on the vector as four separate 54-bit packed data elements (quadword (QW) size data elements), eight separate 32-bit packed data elements (doubleword (DW) size data elements), sixteen separate 16-bit packed data elements (word (W) size data elements), or thirty-two separate 8-bit data elements (byte (B) size data elements). Note that different vector widths and register sizes are possible.

[0088] In one embodiment, one or more execution units may be grouped into fused execution units 509A-509N with thread control logic (507A-507N) common to the fused EUs. Multiple EUs may be fused into an EU group. Each EU in a fused EU group may be configured to execute a separate SIMD hardware thread. The number of EUs in a fused EU group may vary depending on the embodiment. Additionally, various SIMD widths may be implemented per EU, including, without limitation, SIMD8, SIMD16, and SIMD32. Each fused graphics execution unit 509A-509N includes at least two execution units. For example, the fused execution units 509A-509N include a first EU 508A, a second EU 508B, and thread control logic 507A common to the first EU 508A and the second EU 508B. Thread control logic 507A controls the threads executed in fused graphics execution unit 509A, and enables each EU in fused execution units 509A-509N to execute using a common instruction pointer register.

[0089] One or more internal instruction caches (e.g., 506) are included in the thread execution logic 500 for caching thread instructions for the execution units. In some embodiments, one or more data caches (e.g., 502) are included for caching thread data during thread execution. Threads executing in the execution logic 500 can also store explicitly managed data in shared local memory 511. In some embodiments, a sampler 510 is included to provide texture sampling for 3D operations and media sampling for media operations. In some embodiments, the sampler 510 includes specialized texture or media sampling functions to process texture or media data during the sampling process before providing the sampled data to the execution units.

[0090] During execution, the graphics and media pipeline sends thread start requests to the thread execution logic 500 via thread creation and dispatch logic. Once a group of geometric objects has been processed and rasterized into pixel data, pixel processor logic (e.g., pixel shader logic, fragment shader logic, etc.) within the shader processor 502 is invoked to further compute output information and cause the results to be written to an output surface (e.g., a color buffer, a depth buffer, a stencil buffer, etc.). In some embodiments, the pixel shader or fragment shader computes values ​​for various vertex attributes that are to be interpolated across the rasterized objects. In some embodiments, the pixel processor logic within the shader processor 502 then executes pixel or fragment shader programs supplied by an application programming interface (API). To execute shader programs, the shader processor 502 dispatches threads to execution units (e.g., 508A) via the thread dispatcher 504. In some embodiments, the shader processor 502 uses texture sampling logic within the sampler 510 to access texture data within texture maps stored in memory. Arithmetic operations on the texture data and input geometry data compute pixel color data for each geometric fragment or discard one or more pixels from further processing.

[0091] In some embodiments, data port 514 provides a memory access mechanism for thread execution logic 500 to output processed data to memory for further processing in the graphics processor output pipeline. In some embodiments, data port 514 includes or couples to one or more cache memories (e.g., data cache 512) that cache data for memory access via the data port.

[0092] In one embodiment, the execution logic 500 may also include a ray tracer 505 that may provide ray tracing acceleration functionality. The ray tracer 505 may support a ray tracing instruction set that includes instructions / functions for ray generation. The ray tracing instruction set may be similar to or different from the ray tracing instruction set supported by the ray tracing core 245 of FIG. 2C.

[0093] 5B illustrates internal details of execution unit 508 according to an embodiment. GSU 508 may include an instruction fetch unit 537, a general purpose register file array (GRF) 524, an architectural register file array (ARF) 526, a thread arbiter 522, a send unit 530, a branch unit 532, a set of SIMD floating-point units (FPUs) 534, and, in one embodiment, a set of dedicated integer SIMD ALUs 535. GRF 524 and ARF 526 include a set of general purpose register files and architectural register files associated with each concurrent hardware thread that may be active in GSU 508. In one embodiment, per-thread architectural state is maintained in ARF 526, while data used during thread execution is stored in GRF. Execution state of each thread, including the instruction pointer for each thread, may be maintained in thread-specific registers within ARF 526.

[0094] In one embodiment, the graphics execution unit 508 has an architecture that is a combination of simultaneous multithreading (SMT) and fine-grained interleaved multithreading (IMT). The architecture has a modular configuration that can be tuned at design time based on the target number of simultaneous threads and the number of registers per execution unit, and execution unit resources are divided across the logic used to execute multiple simultaneous threads. The number of logical threads that can be executed by the graphics execution unit 508 is not limited to the number of hardware threads, and multiple logical threads can be assigned to each hardware thread.

[0095] In one embodiment, the graphics execution unit 508 can co-issue multiple instructions, each of which can be a different instruction. The thread arbiter 522 of the graphics execution unit thread 508 can dispatch instructions to one of the send unit 530, branch unit 532, or SIMD FPU 534 for execution. Each execution thread can access 128 general-purpose registers in the GRF 524, each of which can store 32 bytes accessible as a SIMD 8-element vector of 32-bit data elements. In one embodiment, each execution unit thread has access to 4K bytes in the GRF 524, although embodiments are not so limited and more or less register resources may be provided in other embodiments. In one embodiment, the graphics execution unit 508 is partitioned into seven hardware threads that can independently perform computational operations, although the number of threads per execution unit can also vary according to the embodiment. For example, in one embodiment, up to 16 hardware threads are supported. In an embodiment where seven threads can access 4K bytes, GRF 524 can store a total of 28K bytes. If 16 threads can access 4K bytes, GRF 524 can store a total of 64K bytes. Flexible addressing modes can allow registers to be addressed together to effectively build wider registers or to represent strided rectangular block data structures.

[0096] In one embodiment, memory operations, sampler operations, and other longer latency system communications are dispatched via "send" instructions executed by a message-passing send unit 530. In one embodiment, branch instructions are dispatched to a dedicated branch unit 532 to expedite SIMD branching and eventual convergence.

[0097] In one embodiment, the graphics execution unit 508 includes one or more SIMD floating-point units (FPUs) 534 to perform floating-point operations. In one embodiment, the FPUs 534 also support integer calculations. In one embodiment, the FPUs 534 can SIMD execute up to M 32-bit floating-point (or integer) operations or up to 2M 16-bit integer or 16-bit floating-point operations. In one embodiment, at least one of the FPUs provides extended math capabilities to support high-throughput transcendental math functions and double-precision 54-bit floating-point. In some embodiments, a set of 8-bit integer SIMD ALUs 535 are also present and may be specifically optimized to perform operations related to machine learning calculations.

[0098] In one embodiment, an array of multiple instances of graphics execution unit 508 may be instantiated in a graphics sub-core grouping (e.g., a sub-slice). For scalability, product designers can choose the exact number of execution units per sub-core grouping. In one embodiment, execution unit 508 can execute instructions across multiple execution channels. In a further embodiment, each thread executing in graphics execution unit 508 executes on a different channel.

[0099] FIG. 6 illustrates a further execution unit 600 according to an embodiment. Execution unit 600 may be, for example, a compute-optimized execution unit used in compute engine tiles 340A-340D of FIG. 3C, but is not limited as such. Variations of execution unit 600 may also be used in graphics engine tiles 310A-310B of FIG. 3B. In one embodiment, execution unit 600 includes a thread control unit 601, a thread state unit 602, an instruction fetch / prefetch unit 603, and an instruction decode unit 604. Execution unit 600 further includes a register file 606 that stores registers that may be allocated to hardware threads within the execution unit. Execution unit 600 further includes a send unit 607 and a branch unit 608. In one embodiment, send unit 607 and branch unit 608 may operate similarly to send unit 530 and branch unit 532 of graphics execution unit 508 of FIG. 5B.

[0100] Execution unit 600 also includes compute unit 610, which includes a variety of functional units. In one embodiment, compute unit 610 includes ALU unit 611, which includes an array of ALUs. ALU unit 611 may be configured to perform 64-bit, 32-bit, and 16-bit integer and floating-point operations. The integer and floating-point operations may be performed simultaneously. Compute unit 610 may also include systolic array 612 and math unit 613. Systolic array 612 includes a wide W and deep D network of data processing units that can be used to systolicly perform vector or other data-parallel operations. In one embodiment, systolic array 612 is configured to perform matrix operations, such as matrix dot-product operations. In one embodiment, systolic array 612 supports 16-bit floating-point operations along with 8-bit and 4-bit integer operations. In one embodiment, systolic array 612 may be configured to accelerate machine learning operations. In such an embodiment, systolic array 612 may be configured with support for the bfloat 16-bit floating-point format. In one embodiment, math unit 613 may be included to perform a specific subset of mathematical operations more efficiently and with less power than ALU unit 611. Math unit 613 may include a variation of math logic found in the shared functional logic of a graphics processing engine provided by other embodiments (e.g., math logic 422 of shared functional logic 420 of FIG. 4). In one embodiment, math logic 613 may be configured to perform 32-bit and 64-bit floating-point operations.

[0101] The thread control unit 601 contains logic to control the execution of threads within the execution units. The thread control unit 601 may include thread arbitration logic to start, stop, and preempt the execution of threads within the execution units 600. The thread state unit 602 may be used to store the thread states of threads assigned to execute in the execution units 600. Storing the thread states within the execution units 600 allows for fast preemption of threads if they become blocked or idle. The instruction fetch / prefetch unit 603 may fetch instructions from an instruction cache of higher-level execution logic (e.g., instruction cache 506 of FIG. 5A). The instruction fetch / prefetch unit 603 may also issue prefetch instructions to be loaded into the instruction cache based on analysis of currently executing threads. The instruction decode unit 604 may be used to decode instructions to be executed by the compute units. In one embodiment, the instruction decode unit 604 may be used as a secondary decoder to decode instructions into constituent micro-operations.

[0102] Execution unit 600 further includes a register file 606 that may be used by hardware threads executing in execution unit 600. The registers in register file 606 may be divided across logic used to execute multiple simultaneous threads within compute units 610 of execution unit 600. The number of logical threads that may be executed by graphics execution unit 600 is not limited to the number of hardware threads, and multiple logical threads may be assigned to each hardware thread. The size of register file 606 may vary from embodiment to embodiment based on the number of hardware threads supported. In one embodiment, register renaming may be used to dynamically assign registers to hardware threads.

[0103] 7 is a block diagram illustrating a graphics processor instruction format 700 according to some embodiments. In one or more embodiments, a graphics processor execution unit supports an instruction set having instructions in multiple formats. Solid boxes represent components that are typically included in execution unit instructions, while dashed lines include components that are optional or that are only included in a subset of instructions. In some embodiments, the instruction format 700 described and illustrated are macro-instructions in that they are supplied to the execution unit as the instruction is processed, as opposed to micro-operations that result from instruction decoding.

[0104] In some embodiments, the graphics processor execution units natively support instructions in a 128-bit instruction format 710. A 64-bit compressed instruction format 730 is available for some instructions based on the selected instruction, instruction options, and number of operands. While the native 128-bit instruction format 710 provides access to all instruction options, some options and operations are restricted in the 64-bit format 730. The native instructions available in the 64-bit format 730 vary depending on the embodiment. In some embodiments, instructions are partially compressed using a set of index values ​​in index field 713. The execution unit hardware references a set of compaction tables based on the index values ​​and uses the compaction table output to reconstruct the native instructions in the 128-bit instruction format 710. Instructions of other sizes and formats are available.

[0105] For each format, the instruction opcode 712 defines the operation to be performed by the execution unit. The execution unit executes each instruction in parallel across multiple data elements of each operand. For example, in response to an add instruction, the execution unit performs a simultaneous add operation across each color channel representing a texture or picture element. By default, the execution unit executes each instruction across all data channels of the operand. In some embodiments, the instruction control field 714 enables control over certain execution options, such as channel selection (e.g., predication) and data channel order (e.g., swizzle). For instructions in the 128-bit instruction format 710, the exec-size field 716 limits the number of data channels that are executed in parallel. In some embodiments, the exec-size field 716 is not available for use with the 64-bit compressed instruction format 730.

[0106] Some execution unit instructions have up to three operands, including two source operands SRC0 720, SRC1 722, and one destination 718. In some embodiments, the execution unit supports dual-destination instructions, where one of the destinations is implicit. Data manipulation instructions can have a third source operand (e.g., SRC2 724), and the instruction opcode 712 determines the number of source operands. The last source operand of an instruction can be an intermediate (hard-coded) value sent with the instruction.

[0107] In some embodiments, the 128-bit instruction format 710 includes an access / address mode field 726 that specifies, for example, whether direct or indirect register addressing mode is used. When direct register addressing mode is used, the register addresses of one or more operands are provided directly by bits in the instruction.

[0108] In some embodiments, the 128-bit instruction format 710 includes an access / address mode field 726 that specifies the address mode and / or access mode of the instruction. In one embodiment, the access mode is used to define the data access alignment of the instruction. Some embodiments support access modes including a 16-byte aligned access mode and a 1-byte aligned access mode, where the byte alignment of the access mode determines the access alignment of the instruction's operands. For example, when in a first mode, the instruction may use byte-aligned addressing for source and destination operands, and when in a second mode, the instruction may use 16-byte aligned addressing for all source and destination operands.

[0109] In one embodiment, the address mode portion of the access / address mode field 726 determines whether the instruction should use direct or indirect addressing. When the direct register addressing mode is used, bits in the instruction directly provide the register addresses of one or more operands. When the indirect register addressing mode is used, the register addresses of one or more operands may be calculated based on the address register value and the address intermediate field in the instruction.

[0110] In some embodiments, instructions are grouped based on a bit field in the opcode 712 to simplify opcode decoding 740. In the case of an 8-bit opcode, bits 4, 5, and 6 allow the execution unit to determine the type of opcode. The exact opcode grouping shown is only an example. In some embodiments, the move / logical opcode group 742 includes data movement instructions and logical instructions (e.g., move (mov), compare (cmp)). In some embodiments, the move and logical group 742 share five most significant bits (MSBs), with move (mov) instructions taking the form 0000xxxxb and logical instructions taking the form 0001xxxb. The flow control instruction group 744 (e.g., call, jump (jmp)) taking the form 0010xxxb (e.g., 0x20). The miscellaneous instruction group 746 contains a mix of instructions, including synchronization instructions (e.g., wait, send) in the form of 0011xxxb (e.g., 0x30). The parallel math instruction group 748 contains component-wise arithmetic instructions (e.g., add, multiply (mul)) in the form of 0100xxxb (e.g., 0x40). The parallel math instruction group 748 performs arithmetic operations in parallel across data channels. The vector math group 750 contains arithmetic instructions (e.g., dp4) in the form of 0101xxxb (e.g., 0x50). The vector math group 750 performs calculations such as dot product calculations on vector operands. The illustrated opcode decode 740, in one embodiment, can be used to determine which portion of the execution unit will be used to execute the decoded instruction. For example, some instructions may be designed as systolic instructions executed by a systolic array. Other instructions, such as ray tracing instructions (not shown), may be sent to a ray tracing core or ray tracing logic within a slice or partition of the execution logic.

[0111] [Graphics Pipeline] Figure 8 is a block diagram of another embodiment of a graphics processor 800. Elements in Figure 8 having the same reference numbers (or names) as elements in any other figure in this application can operate or function similarly as described elsewhere herein, but are not limited to such.

[0112] In some embodiments, graphics processor 800 includes a geometry pipeline 820, a media pipeline 830, a display engine 840, thread execution logic 850, and a render output pipeline 870. In some embodiments, graphics processor 800 is a graphics processor within a multi-core processing system that includes one or more general-purpose processing cores. The graphics processor is controlled by register writes to one or more control registers (not shown) or via commands issued to graphics processor 800 via ring interconnect 802. In some embodiments, ring interconnect 802 couples graphics processor 800 to other processing components, such as other graphics processors or general-purpose processors. Commands from ring interconnect 802 are interpreted by command streamer 803. Command streamer 803 supplies instructions to individual components of geometry pipeline 820 or media pipeline 830.

[0113] In some embodiments, command streamer 803 reads vertex data from memory and directs the operation of vertex fetcher 805, which executes vertex processing commands provided by command streamer 803. In some embodiments, vertex fetcher 805 provides vertex data to vertex shader 807, which performs coordinate space transformations and lighting operations on the extension points. In some embodiments, vertex fetcher 805 and vertex shader 807 execute vertex processing instructions by dispatching execution threads to execution units 852A-852B via thread dispatcher 831.

[0114] In some embodiments, the execution units 852A-852B are arrays of vertex processors with instruction sets for performing graphics and media operations. In some embodiments, the execution units 852A-852B have an attached L1 cache 851 that is local to each array or shared between the arrays. The cache can be configured as a data cache or an instruction cache, or a single cache that is partitioned to contain data and instructions in different partitions.

[0115] In some embodiments, geometry pipeline 820 includes a tessellation component to perform hardware-accelerated tessellation of 3D objects. In some embodiments, programmable hull shader 817 provides back-end evaluation of the tessellation output. Tessellator 813 operates under the direction of hull shader 811 and includes special-purpose logic that generates a set of detailed geometric objects based on a coarse geometric model provided as input to geometry pipeline 820. In some embodiments, the tessellation components (e.g., hull shader 811, tessellator 813, and domain shader 817) can be bypassed if tessellation is not used.

[0116] In some embodiments, a complete geometric object may be processed by the geometry shader 819 via one or more threads dispatched to execution units 852A-852B, or may proceed directly to the clipper 829. In some embodiments, the geometry shader operates on the entire geometric object, rather than on vertices or patches of vertices as found in earlier stages of the graphics pipeline. The geometry shader 819 receives input from the vertex shader 807 when tessellation is disabled. In some embodiments, the geometry shader 819 is programmable by a geometry shader program to perform geometry tessellation when the tessellation units are disabled.

[0117] Prior to rasterization, a clipper 829 processes the vertex data. The clipper 829 may be a fixed-function clipper or a programmable clipper with clipping and geometry shader functionality. In some embodiments, a rasterizer and depth test component 873 in the render output pipeline 870 dispatches pixel shaders to convert geometric objects into per-pixel representations. In some embodiments, pixel shader logic is included in the thread execution logic 850. In some embodiments, an application can bypass the rasterizer and depth test component 873 and access unrasterized vertex data through the stream output unit 823.

[0118] Graphics processor 800 includes an interconnect bus, interconnect fabric, or other interconnect mechanism that allows data and message passing between the major components of the processor. In some embodiments, execution units 825A-825B and associated logic units (e.g., L1 cache 851, sampler 854, texture cache 858, etc.) interconnect via data port 856 to perform memory accesses and communicate with the processor's render output pipeline components. In some embodiments, sampler 854, caches 851, 858, and execution units 852A-852B each have a separate memory access path. In one embodiment, texture cache 858 may also be configured as a sampler cache.

[0119] In some embodiments, the render output pipeline 870 includes a rasterizer and depth test component 873 that converts vertex-based objects into associated pixel-based representations. In some embodiments, the rasterizer logic includes a windower / marker unit to perform fixed-function triangle and line rasterization. An associated render cache 878 and depth cache 879 are also available in some embodiments. A pixel operations component 877 performs pixel-based operations on the data, although in some instances pixel operations related to 2D operations (e.g., bit-block image translation with blending) are performed by the 2D engine 841 or are replaced at display time by the display controller 843 using an overlay display surface. In some embodiments, a shared L3 cache 875 is available to all graphics components, allowing data sharing without using main system memory.

[0120] In some embodiments, graphics processor media pipeline 830 includes a media engine 837 and a video front end 834. In some embodiments, video front end 834 receives pipeline commands from command streamer 803. In some embodiments, media pipeline 830 includes a separate command streamer. In some embodiments, video front end 834 processes media commands before sending the commands to media engine 837. In some embodiments, media engine 837 includes a thread spawning function to spawn threads for dispatch to thread execution logic 850 via thread dispatcher 831.

[0121] In some embodiments, graphics processor 800 includes a display engine 840. In some embodiments, display engine 840 is external to processor 800 and couples to graphics processor 800 via ring interconnect 802 or other interconnect bus or fabric. In some embodiments, display engine 840 includes a 2D engine 841 and a display controller 843. In some embodiments, display engine 840 includes special-purpose logic that can operate independently of the 3D pipeline. In some embodiments, display controller 843 couples to a display device (not shown). The display device may be a display device integrated into the system, such as a laptop computer, or an external display device attached via a display device connector.

[0122] In some embodiments, the geometry pipeline 820 and the media pipeline 830 are configurable to perform operations based on multiple graphics and media programming interfaces and are not specific to any one application programming interface (API). In some embodiments, driver software for the graphics processor translates API calls that are specific to a particular graphics or media library into commands that can be processed by the graphics processor. In some embodiments, support is provided for the Open Graphics Library (OpenGL), Open Computing Language (OpenCL), and / or Vulkan graphics and compute APIs, all from the Khronos Group. In some embodiments, support may also be provided for the Direct3D library from Microsoft Corporation. In some embodiments, a combination of these libraries may be supported. Support may also be provided for the Open Source Computer Vision (OpenCV) library. Future APIs with compatible 3D pipelines may also be supported if a mapping can be made from the future API's pipeline to the graphics processor's pipeline.

[0123] [Graphics Pipeline Programming] Figure 9A is a block diagram illustrating a graphics processor command format 900 according to some embodiments. Figure 9B is a block diagram illustrating a graphics processor command sequence 910 according to an embodiment. The solid blocks in Figure 9A represent components that are typically included in graphics commands, while the dashed lines include components that are optional or that are only included in a subset of graphics commands. The example graphics processor command format 900 in Figure 9A includes data fields identifying a client 902, a command operation code (opcode) 904, and data for the command 906. A sub-opcode 905 and a command size 908 are also included in some commands.

[0124] In some embodiments, client 902 specifies a client unit of the graphics device that will process the command data. In some embodiments, a graphics processor command parser examines a client field in each command to condition further processing of the command and route the command data to the appropriate client unit. In some embodiments, the graphics processor client units include a memory interface unit, a render unit, a 2D unit, a 3D unit, and a media unit. Each client unit has a corresponding processing pipeline that processes the command. When a command is received by a client unit, the client unit reads opcode 904 and, if present, sub-opcode 905 to determine the operation to perform. The client unit executes the command using the information in field 906. For some commands, an explicit command size 908 is expected to specify the size of the command. In some embodiments, the command parser automatically determines the size of at least some of the commands based on the command opcode. In some embodiments, commands are aligned by multiples of doublewords. Other command formats are possible.

[0125] The flow diagram of Figure 9B depicts an example graphics processor command sequence 910. In some embodiments, software or firmware in a data processing system featuring a graphics processor embodiment uses variations of the command sequence shown to set up, execute, and terminate a set of graphics operations. The sample command sequence is shown and described solely as an example, as embodiments are not limited to these particular commands or to this command sequence. Furthermore, commands may be issued as a batch of commands within a command sequence, such that the graphics processor processes the sequence of commands at least partially concurrently.

[0126] In some embodiments, the graphics processor command sequence 910 may begin with a pipeline flush command 912 to force any active graphics pipelines to complete any currently pending commands for the pipeline. In some embodiments, the 3D pipeline 922 and the media pipeline 924 do not operate simultaneously. A pipeline flush is performed to force the active graphics pipelines to complete any pending commands. In response to the pipeline flush, the command parser for the graphics processor pauses command processing until the active drawing engines complete pending operations and the associated read caches are invalidated. Optionally, any data in the render cache that is marked as "dirty" may be flushed to memory. In some embodiments, the pipeline flush command 912 may be used for pipeline synchronization or before placing the graphics processor in a low-power state.

[0127] In some embodiments, the pipeline select command 913 is used when a command sequence requests the graphics processor to explicitly switch pipelines. In some embodiments, the pipeline select command 913 is only needed once within an execution context, before issuing a pipeline command, unless the context should issue commands for both pipelines. In some embodiments, the pipeline flush command 912 is needed immediately before a pipeline switch via the pipeline select command 913.

[0128] In some embodiments, pipeline control commands 914 are used to configure the graphics pipeline for operation and to program the 3D pipeline 922 and the media pipeline 924. In some embodiments, pipeline control commands 914 configure the pipeline state of the active pipeline. In one embodiment, pipeline control commands 914 are used for pipeline synchronization and to clear data from one or more cache memories in the active pipeline before processing a batch of commands.

[0129] In some embodiments, the return buffer state command 916 is used to configure the set of return buffers to which each pipeline writes data. Some pipeline operations require the allocation, selection, or configuration of one or more return buffers to which the operation writes intermediate data during processing. In some embodiments, the graphics processor also uses one or more return buffers to store output data and to perform cross-thread communication. In some embodiments, the return buffer state 916 includes selecting the size and number of return buffers to use for a set of pipeline operations.

[0130] The remaining commands in the command sequence differ based on the active pipeline for the operation. In pipeline decision 920, the command sequence is tailored for either the 3D pipeline 922 starting from 3D pipeline state 930 or the media pipeline 924 starting from media pipeline state 940.

[0131] Commands for configuring the 3D pipeline 922 include 3D set commands for vertex buffer state, vertex element state, constant color state, depth buffer state, and other state variables that should be configured before 3D primitive commands are processed. The values ​​of these commands are determined at least in part based on the particular 3D API in use. In some embodiments, the 3D pipeline state 930 commands can also selectively disable or bypass certain pipeline elements if those elements are not used.

[0132] In some embodiments, the 3D primitive 932 command is used to issue a 3D primitive to be processed by the 3D pipeline. The command and associated parameters sent to the graphics processor via the 3D primitive 932 command are forwarded to a vertex fetch function in the graphics pipeline. The vertex fetch function uses the 3D primitive 932 command to generate a vertex data structure. The vertex data structure is stored in one or more return buffers. In some embodiments, the 3D primitive 932 command is used to perform vertex operations on the 3D primitives by a vertex shader. To process the vertex shader, the 3D pipeline 922 dispatches shader execution threads to the graphics processor execution units.

[0133] In some embodiments, the 3D pipeline 922 is triggered by an execute 934 command or event. In some embodiments, a register write triggers command execution. In some embodiments, execution is triggered by a 'go' or 'kick' command in a command sequence. In one embodiment, command execution is triggered using a pipeline synchronization command to flush the command sequence through the graphics pipeline. The 3D pipeline performs geometry processing for 3D primitives. Once the operations are complete, the resulting geometric objects are rasterized and the pixel engine colors the resulting pixels. Additional commands to control pixel shading and pixel backend operations may also be included in the operations.

[0134] In some embodiments, the graphics processor command sequence 910 follows the media pipeline 924 path when performing media operations. In general, the specific use and manner of programming the media pipeline 924 depends on the media or computational operations being performed. Certain media decoding operations may be offloaded to the media pipeline during media decoding. In some embodiments, the media pipeline may also be bypassed, with media decoding performed in whole or in part using resources provided by one or more general-purpose processing cores. In one embodiment, the media pipeline also includes elements for general-purpose graphics processor unit (GPGPU) operation, where the graphics processor is used to perform SIMD vector operations using compute shader programs not explicitly related to rendering graphics primitives.

[0135] In some embodiments, the media pipeline 924 is configured similarly to the 3D pipeline 922. A set of commands for configuring the media pipeline state 940 is dispatched or placed in a command queue before the media object commands 942. In some embodiments, the commands for the media pipeline state 940 contain data that configures the media pipeline elements used to process the media object. This includes data that configures the video decoding and video encoding logic within the media pipeline, such as encoding or decoding formats. In some embodiments, the commands for the media pipeline state 940 also support the use of one or more pointers to "indirect" state elements that contain a batch of state settings.

[0136] In some embodiments, a media object command 942 provides a point to a media object for processing by the media pipeline. A media object includes a memory buffer containing video data to be processed. In some embodiments, all media pipeline state must be valid before issuing a media object command 942. Once the pipeline state is configured and the media object command 942 is queued, the media pipeline 924 is triggered by an execute command 944 or equivalent execution event (e.g., a register write). The output from the media pipeline 924 may then be post-processed by operations provided by the 3D pipeline 922 or the media pipeline 924. In some embodiments, GPGPU operations are configured and executed similarly to media operations.

[0137] [Graphics Software Architecture] 10 depicts an example graphics software architecture for a data processing system 1000 according to some embodiments. In some embodiments, the software architecture includes a 3D graphics application 1010, an operating system 1020, and at least one processor 1030. In some embodiments, the processor 1030 includes a graphics processor 1032 and one or more general-purpose processor cores 1034. The graphics application 1010 and the operating system 1020 each execute programs in the system memory 1050 of the data processing system.

[0138] In some embodiments, the 3D graphics application 1010 includes one or more shader programs that include shader instructions 1012. The shader language instructions may be in a high-level shader language such as Direct3D's High-Level Shader Language (HLSL), OpenGL Shader Language (GLSL), etc. The application also includes executable instructions 1014 in a machine language suitable for execution by a general-purpose processor core 1034. The application also includes graphics objects 1016 defined by vertex data.

[0139] In some embodiments, the operating system 1020 is a Microsoft® Windows® operating system from Microsoft Corporation, a proprietary UNIX-like operating system, or an open-source UNIX-like operating system that uses a variant of the Linux® kernel. The operating system 1020 can support a graphics API 1022, such as the Direct3D API, the OpenGL API, or the Vulkan API. If Direct3D is in use, the operating system 1020 uses a front-end shader compiler 1024 to compile any instructions in HLSL 1012 into a lower-level shader language. The compilation can be a just-in-time (JIT) compilation, or the application can perform shader precompilation. In some embodiments, high-level shaders are compiled into lower-level shaders during compilation of the 3D graphics application 1010. In some embodiments, the shader instructions 1012 are provided in an intermediate form, such as a variant of the Standard Portable Intermediate Representation (SPIR) used by the Vulkan API.

[0140] In some embodiments, user mode graphics driver 1026 includes a backend shader compiler 1027 to convert shader instructions 1012 into a hardware-specific representation. If the OpenGL API is in use, shader instructions 1012 in the GLSL high-level language are sent to user mode graphics driver 1026 for compilation. In some embodiments, user mode graphics driver 1026 uses operating system kernel mode functions 1028 to communicate with kernel mode graphics driver 1029. In some embodiments, kernel mode graphics driver 1029 communicates with graphics processor 1032 to dispatch commands and instructions.

[0141] [IP core implementation] One or more aspects of at least one embodiment may be implemented by representative code stored on a machine-readable medium that represents and / or defines logic within an integrated circuit, such as a processor. For example, the machine-readable medium may include instructions that represent various logic within a processor. When read by a machine, the instructions cause the machine to assemble the logic to perform the techniques described herein. Such representations, known as "IP cores," are reusable units of logic for an integrated circuit that may be stored on a tangible machine-readable medium as a hardware model that describes the structure of the integrated circuit. The hardware model may be provided to various customers of a manufacturing facility who install the hardware model in their manufacturing machines to produce the integrated circuit. The integrated circuit may be assembled such that the circuit performs the operations described in connection with any of the embodiments described herein.

[0142] FIG. 11A is a block diagram illustrating an IP core development system 1100 that can be used to fabricate an integrated circuit to perform operations according to an embodiment. The IP core development system 1100 can be used to generate modular, reusable designs that can be incorporated into a larger design or used to construct an entire integrated circuit (e.g., a SOC integrated circuit). A design facility 1130 can generate a software simulation 1110 of the IP core design in a high-level programming language (e.g., C / C++). The software simulation 1110 can be used to design, test, and verify the behavior of the IP core using a simulation model 1112. The simulation model 1112 may include functional, behavioral, and / or timing simulations. A register transfer level (RTL) design 1115 can then be created or synthesized from the simulation model 1112. The RTL design 1115 is an abstraction of the behavior of the integrated circuit that models the flow of digital signals between hardware registers, including associated logic that is implemented using the modeled digital signals. In addition to the RTL design 1115, sub-designs at the logic or transistor level may also be created, designed, or synthesized. As such, the specific details of the initial design and simulation may vary.

[0143] The RTL design 1115 or equivalent may be further synthesized by a design facility into a hardware model 1120. The hardware model 1120 may be in a hardware description language (HDL) or other representation of the physical design data. The HDL may be further simulated or tested to verify the IP core design. The IP core design may be stored using non-volatile memory 1140 (e.g., a hard disk, flash memory, or any non-volatile storage medium) for delivery to a third-party manufacturing facility 1165. Alternatively, the IP core design may be transmitted via a wired connection 1150 or a wireless connection 1160 (e.g., via the Internet). The manufacturing facility 1165 may then manufacture an integrated circuit based at least in part on the IP core design. The manufactured integrated circuit may be configured to perform operations in accordance with at least one embodiment described herein.

[0144] 11B illustrates a cross-sectional side view of an integrated circuit package assembly 1170 according to some embodiments described herein. The integrated circuit package assembly 1170 represents an implementation of one or more processors or accelerators described herein. The package assembly 1170 includes multiple units of hardware logic 1172, 1174 connected to a substrate 1180. The logic 1172, 1174 may be at least partially implemented in configurable logic or fixed-function logic hardware and may include one or more portions of a processor core, graphics processor, or any of the other accelerator devices described herein. Each unit of logic 1172, 1174 may be implemented within a semiconductor die and coupled to the substrate 1180 via interconnect structures 1173. The interconnect structures 1173 may be configured to route electrical signals between the logic 1172, 1174 and the substrate 1180 and may include, for example, without limitation, interconnects such as bumps or pillars. In some embodiments, interconnect structure 1173 may be configured to route electrical signals, such as input / output (I / O) signals and / or power or ground signals related to the operation of logic 1172, 1174. In some embodiments, substrate 1180 is an epoxy-based laminate substrate. Substrate 1180 may include other suitable types of substrates in other embodiments. Package assembly 1170 may be connected to other electrical devices via package interconnect 1183. Package interconnect 1183 may be coupled to a surface of substrate 1180 to route electrical signals to other electrical devices, such as a motherboard, another chipset, or a multi-chip module.

[0145] In some embodiments, the units of logic 1172, 1174 are electrically coupled to a bridge 1182 configured to route electrical signals between the logic 1172, 1174. The bridge 1182 may be a dense interconnect structure that provides a path for the electrical signals. The bridge 1182 may include a bridge substrate made of glass or a suitable semiconductor material. Electrical routing mechanisms may be formed on the bridge substrate to provide chip interconnection between the logic 1172, 1174.

[0146] Although two units of logic 1172, 1174 and bridge 1182 are shown, embodiments described herein may include more or less logic on one or more dies. One or more dies may be connected by zero or more bridges, such that bridge 1182 is omitted if the logic is contained on a single die. Alternatively, multiple dies or units of logic may be connected by one or more bridges. Furthermore, multiple logic, units, dies, and bridges may be connected together in other possible configurations, including three-dimensional configurations.

[0147] FIG. 11C illustrates a package assembly 1190 including multiple units of hardware logic chiplets connected to a substrate 1180 (e.g., a base die). Graphics processing units, parallel processors, and / or computational accelerators as described herein can be composed of multiple silicon chiplets fabricated separately. In this context, a chiplet is an at least partially packaged integrated circuit containing discrete units of logic that can be assembled with other chiplets into a larger package. Multiple sets of chiplets with different IP core logic can be assembled into a single device. Furthermore, chiplets can be integrated into a base die or base chiplet using active interposer technology. The concepts described herein enable interconnection and communication between different forms of IP within a GPU. IP cores can be fabricated using different process technologies and assembled during manufacturing. This avoids the complexity of aggregating multiple IPs into the same manufacturing process, especially on large-scale SoCs containing several types of IP. Enabling the use of multiple process technologies improves time to market and provides a cost-effective way to create multiple product SKUs. Additionally, non-collective IPs are more amenable to being power-gated independently, so that components not being used for a given workload can be powered off, reducing overall power consumption.

[0148] The hardware logic chiplets include special-purpose hardware logic chiplets 1172, logic or I / O chiplets 1174, and / or memory chiplets 1175. The hardware logic chiplets 1172 and logic or I / O chiplets 1174 may be implemented at least in part in configurable logic or fixed-function logic hardware and may include one or more portions of a processor core, a graphics processor, a parallel processor, or any of the other accelerator devices described herein. The memory chiplets 1175 can be DRAM (e.g., GDDR, HBM) memory or cache (SRAM) memory.

[0149] Each chiplet may be fabricated as a separate semiconductor die and coupled to substrate 1180 via interconnect structures 1173. Interconnect structures 1173 may be configured to route electrical signals between the various chiplets and logic within substrate 1180. Interconnect structures 1173 may include interconnects such as, for example, without limitation, bumps or pillars. In some embodiments, interconnect structures 1173 may also include, for example, input / output (I / O) associated with logic, I / O, and memory chiplet operation. ) signals and / or electrical signals such as power or ground signals.

[0150] In some embodiments, substrate 1180 is an epoxy-based laminate substrate. Substrate 1180 may include other suitable types of substrates in other embodiments. Package assembly 1190 may be connected to other electrical devices via package interconnect 1183. Package interconnect 1183 may be coupled to a surface of substrate 1180 to route electrical signals to other electrical devices, such as a motherboard, another chipset, or a multi-chip module.

[0151] In some embodiments, logic or I / O chiplets 1174 and memory chiplets 1175 may be electrically coupled via bridges 1187 configured to route electrical signals between the logic or I / O chiplets 1174 and the memory chiplets 1175. Bridges 1187 may be dense interconnect structures that provide paths for electrical signals. Bridges 1187 may include a bridge substrate made of glass or a suitable semiconductor material. Electrical routing mechanisms may be formed on the bridge substrate to provide chip interconnects between the logic or I / O chiplets 1174 and the memory chiplets 1175. Bridges 1187 may also be referred to as silicon bridges or interconnect bridges. For example, bridges 1187 may be embedded multi-die interconnect bridges (EMIBs) in some embodiments. In some embodiments, bridges 1187 may simply be direct connections from one chiplet to another.

[0152] Substrate 1180 can include hardware components for I / O 1191, cache memory 1192, and other hardware logic 1193. Fabric 1185 can be embedded in substrate 1180 to enable communication between various logic chiplets and logic 1191, 1193 within substrate 1180. In one embodiment, I / O 1191, fabric 1185, cache, bridges, and other hardware logic 1193 can be incorporated into a base die that is stacked on top of substrate 1180.

[0153] In various embodiments, package assembly 1190 can include more or fewer components and chiplets interconnected by fabric 1185 or one or more bridges 1187. Chilets within package assembly 1190 may be arranged in a 3D or 2.5D configuration. In general, bridge structure 1187 may be used to facilitate point-to-point connections between, for example, logic or I / O chiplets and memory chiplets. Fabric 1185 may be used to interconnect various logic and / or I / O chiplets (e.g., chiplets 1172, 1174, 1191, 1193) with other logic and / or I / O chiplets. In one embodiment, in-substrate cache memory 1192 can operate as a global cache for package assembly 1190, as part of a discrete global cache, or as a dedicated cache for fabric 1185.

[0154] FIG. 11D illustrates a package assembly 1194 including a replaceable chiplet 1195, according to an embodiment. The replaceable chiplet 1195 can be assembled into a standardized slot on one or more base chiplets 1196, 1198. The base chiplets 1196, 1198 can be coupled via a bridge interconnect 1197. The bridge interconnect 1197 can be similar to other bridge interconnects described herein, such as an EMIB. Memory chiplets can also be connected to logic or I / O chiplets via the bridge interconnect. I / O and logic chiplets can communicate via an interconnect fabric. The base chiplets can each support one or more slots in a standardized format for one of logic, I / O, or memory / cache.

[0155] In one embodiment, SRAM and power distribution circuitry may be fabricated in one or more of the base chiplets 1196, 1198. These may be fabricated using a different process technology than the replaceable chiplets 1195 stacked on top of the base chiplets. For example, the base chiplets 1196, 1198 may be fabricated using a larger-scale process technology, while the replaceable chiplets may be fabricated using a smaller-scale process technology. One or more of the replaceable chiplets 1195 may be memory (e.g., DRAM) chiplets. Different memory densities may be selected for the package assembly 1194 based on the performance and / or power targets of the product using the package assembly 1194. Furthermore, logic chiplets with a variety of functional units may be selected during assembly based on the performance and / or power targets of the product. Furthermore, chiplets containing various types of IP logic cores may be insertable into the replaceable chiplet slots, enabling hybrid processor designs that can mix and match IP blocks from different technologies.

[0156] [Example of a system on a chip integrated circuit] 12-13 illustrate examples of integrated circuits and associated graphics processors that may be fabricated using one or more IP cores in accordance with various embodiments described herein. In addition to what is depicted, other logic and circuitry may be included, including additional graphics processors / cores, peripheral interface controllers, or general-purpose processor cores.

[0157] 12 is a block diagram illustrating a system on a chip integrated circuit 1200 that may be fabricated using one or more IP cores, according to an embodiment. The example integrated circuit 1200 includes one or more application processors 1205 (e.g., CPUs) and at least one graphics processor 1210, and may also include an image processor 1215 and / or a video processor 1220, any of which may be modular IP cores from the same or multiple different design facilities. The integrated circuit 1200 includes a USB controller 1225, a UART controller 1230, an SPI / SDIO controller 1235, and an I / O controller 1240. 2 S / I 2 The integrated circuit 1200 includes peripheral or bus logic including a C controller 1240. Additionally, the integrated circuit 1200 may include a display device 1245 coupled to one or more of a High-Definition Multimedia Interface (HDMI®) controller 1250 and a Mobile Industry Processor Interface (MIPI) display interface 1255. Storage may be provided by a flash memory subsystem 1260 including a flash memory and a flash memory controller. A memory interface may be provided via a memory controller 1265 for access to an SDRAM or SRAM memory device. Some integrated circuits further include an embedded security engine 1270.

[0158] 13A-13B are block diagrams illustrating graphics processors used within an SoC in accordance with embodiments described herein. FIG. 13A illustrates a graphics processor 1310 in a system on a chip integrated circuit that may be fabricated using one or more IP cores in accordance with embodiments. FIG. 13B illustrates a further example of a graphics processor 1340 in a system on a chip integrated circuit that may be fabricated using one or more IP cores in accordance with embodiments. The graphics processor 1310 in FIG. 13A is an example of a low-power graphics processor core. The graphics processor 1340 in FIG. 13B is an example of a higher-performance graphics processor core. Each of the graphics processors 1310, 1340 may be a variation of the graphics processor 1210 in FIG. 12.

[0159] As shown in FIG. 13A , graphics processor 1310 includes vertex processor 1305 and one or more fragment processors 1315A-1315N (e.g., 1315A, 1315B, 1315C, 1315D, through 1315N-1 and 1315N). Graphics processor 1310 can execute different shader programs through separate logic, such that vertex processor 1305 is optimized to perform the operations of vertex shader programs, while one or more fragment processors 1315A-1315N perform the fragment (e.g., pixel) shading operations of fragment or pixel shader programs. Vertex processor 1305 performs the vertex processing stage of the 3D graphics pipeline, generating primitive and vertex data. Fragment processors 1315A-1315N use the primitive and vertex data generated by vertex processor 1305 to generate a frame buffer for display on a display device. In one embodiment, fragment processors 1315A-1315N are optimized to execute fragment shader programs such as those provided in the OpenGL API, which can be used to perform operations similar to pixel shader programs such as those provided in Direct3D.

[0160] The graphics processor 1310 further includes one or more memory management units (MMUs) 1320A-1320B, caches 1325A-1325B, and circuit interconnects 1330A-1330B. The one or more MMUs 1320A-1320B provide virtual to physical address mapping for the graphics processor 1310, including the vertex processor 1305 and / or fragment processors 1315A-1315N. The graphics processor 1310 may reference vertex or image / texture data stored in memory in addition to vertex or image / texture data stored in one or more caches 1325A-1325B. In one embodiment, the one or more MMUs 1320A-1320B may synchronize with other MMUs in the system, including one or more MMUs associated with one or more application processors 1205, image processor 1215, and / or video processor 1220 of FIG. 12. This allows each processor 1205-1220 to participate in a shared or unified virtual memory system. One or more circuit interconnects 1330A-1330B allow graphics processor 1310 to interface with other IP cores within the SoC via the SoC's internal bus or via direct connections, depending on the embodiment.

[0161] As shown in Figure 13B, graphics processor 1340 includes one or more MMUs 1320A-1320B, caches 1325A-1325B, and circuit interconnects 1330A-1330B of graphics processor 1310 of Figure 13A. Graphics processor 1340 includes one or more shader cores 1355A-1355N (e.g., 1355A, 1355B, 1355C, 1355D, 1355E, 1355F through 1355N-1, and 1355N) that provide a unified shader core architecture in which a single core or a single type of core can execute all types of programmable shader code, including vertex shaders, fragment shaders, and / or compute shaders. The exact number of shader cores present can vary among embodiments and implementations. Additionally, graphics processor 1340 includes an inter-core task manager 1345 that acts as a thread dispatcher to dispatch execution threads to one or more shader cores 1355A-1355N, and a tiling unit 1358 that accelerates tiling operations for tile-based rendering, in which rendering operations for a scene are subdivided in image space, for example, to exploit local spatial coherence within the scene or to optimize internal cache usage.

[0162] [Machine Learning Overview] Machine learning algorithms are algorithms that can learn based on a set of data. Embodiments of machine learning algorithms may be designed to model higher-level abstractions within a dataset. For example, an image recognition algorithm can be used to determine which of several categories a given input belongs to, a regression algorithm can output a numerical value given an input, and a pattern recognition algorithm can generate converted text or be used to convert text to speech and / or perform speech recognition.

[0163] An exemplary type of machine learning algorithm is a neural network. There are many types of neural networks, and a simple type of neural network is a feedforward network. A feedforward network may be implemented as an acyclic graph in which nodes are arranged in layers. Typically, a feedforward network topology includes an input layer and an output layer separated by at least one hidden layer. The hidden layer converts inputs received by the input layer into a representation that is useful for generating outputs at the output layer. Network nodes are fully connected via edges to nodes in adjacent layers, but there are no edges between nodes within each layer. Data received at nodes in the input layer of a feedforward network is propagated (i.e., fed forward) to nodes in the output layer via an activation function that calculates the state of the nodes in each successive layer in the network based on coefficients (“weights”) associated with each of the edges connecting the layers. Depending on the specific model represented by the algorithm being executed, output from a neural network algorithm can take a variety of forms.

[0164] Before a machine learning algorithm can be used to model a particular problem, the algorithm is trained using a training dataset. Training a neural network involves selecting a network topology, using a set of training data representative of the problem to be modeled by the network, and adjusting weights until the network model performs with minimal error for all instances of the training dataset. For example, during the supervised learning training process of a neural network, outputs generated by the network in response to inputs representing instances in the training dataset are compared with outputs labeled "correct" for that instance, an error signal representing the difference between the output and the labeled output is calculated, and weights associated with the connections are adjusted to minimize the error as the error signal propagates backward through the layers of the network. The network is considered "trained" when the error for each output generated from an instance of the training dataset is minimized.

[0165] The accuracy of machine learning algorithms can be significantly affected by the quality of the datasets used to train the algorithms. The training process can be computationally intensive and require a significant amount of time for conventional general-purpose processors. Therefore, parallel processing hardware is used to train many types of machine learning algorithms. This is particularly useful for optimizing the training of neural networks, as the calculations performed in adjusting the coefficients in a neural network naturally lend themselves to parallel implementation. In particular, many machine learning algorithms and software applications have been adapted to use parallel processing hardware within general-purpose graphics processing devices.

[0166] 14 is a generalized diagram of a machine learning software stack 1400. A machine learning application 1402 may be configured to train a neural network using a training dataset or to implement machine intelligence using a trained deep neural network. The machine learning application 1402 may include training and inference functions for the neural network and / or specialized software that may be used to train the neural network before deployment. The machine learning application 1402 may implement any type of machine intelligence, including, without limitation, image recognition, mapping and localization, autonomous navigation, speech synthesis, medical imaging, or language conversion.

[0167] Hardware acceleration for machine learning applications 1402 can be enabled through machine learning framework 1404. Machine learning framework 1404 can provide a library of machine learning primitives. Machine learning primitives are basic operations typically performed by machine learning algorithms. Without machine learning framework 1404, machine learning algorithm developers would be required to create and optimize the main computational logic associated with their machine learning algorithms and then re-optimize the computational logic as new parallel processors are developed. Instead, machine learning applications can be configured to perform the necessary computations using primitives provided by machine learning framework 1404. Example primitives include tensor convolution, activation functions, and pooling. These are computational operations performed while training a convolutional neural network (CNN). Machine learning framework 1404 can also provide primitives for implementing basic linear algebra subprograms performed by many machine learning algorithms, such as matrix and vector operations.

[0168] The machine learning framework 1404 can process input data received from the machine learning application 1402 and generate appropriate inputs to the computation framework 1406. The computation framework 1406 can abstract the primitives provided to the GPGPU hardware 1410 to enable the machine learning framework 1404 to take advantage of hardware acceleration via the GPGPU hardware 1410 without requiring the machine learning framework 1404 to have in-depth knowledge of the architecture of the GPGPU hardware 1410. Furthermore, the computation framework 1406 can enable hardware acceleration for the machine learning framework 1404 across different types and generations of GPGPU hardware 1410.

[0169] [Machine learning neural network implementation] The computer architecture provided by the embodiments described herein can be configured to perform a type of parallel processing that is particularly suited to training and deploying neural networks for machine learning. A neural network can be generalized as a network of functions with graph relationships. As known in the art, there are various types of neural network implementations used for machine learning. One example of a type of neural network is a feedforward network, as described above.

[0170] A second example of a type of neural network is the convolutional neural network (CNN). CNNs are specialized feedforward neural networks for processing data with a known grid topology, such as image data. Thus, CNNs are widely used for computational vision and image recognition applications, but they may also be used for other types of pattern recognition, such as speech and language processing. Nodes in a CNN input layer are organized into sets of "filters" (feature detectors inspired by receptive fields in the retina), and the output of each set of filters is propagated to nodes in successive layers of the network. CNN computation involves applying a convolution mathematical operation to each filter to generate that filter's output. Convolution is a special type of mathematical operation performed by two functions to generate a third function that is a modified version of one of the two original functions. In convolutional network terminology, the first function for a convolution may be called the input, while the second function may be called the convolution kernel. The output may be called a feature map. For example, the input to a convolutional layer may be a multidimensional array of data defining the various color components of the input image, and the convolution kernel may be a multidimensional array of parameters, which are adapted by the neural network training process.

[0171] Recurrent neural networks (RNNs) are a family of feedforward neural networks that contain feedback connections between layers. RNNs allow for the modeling of sequential data by sharing parameter data between different parts of the neural network. The architecture of RNNs involves recursion, meaning that at least a portion of the output data from the RNN is used as feedback to process subsequent inputs in the sequence, thereby representing the influence of the current value of that variable on the value of that variable at a future point in time. This feature makes RNNs particularly useful for language processing due to the variability in which language data can be constructed.

[0172] The following figures provide examples of feedforward, CNN, and RNN networks and describe the general process of training and deploying each of these types of networks, respectively. It will be understood that these descriptions are exemplary and not limiting with respect to any specific embodiment described herein, and that the concepts described may be applied generally to deep neural networks and machine learning techniques in general.

[0173] The above example neural networks can be used to perform deep learning, which is machine learning using deep neural networks. Deep neural networks used in deep learning are artificial neural networks that consist of multiple hidden layers, compared to shallow neural networks that contain only a single hidden layer. Deeper neural networks are generally more computationally intensive to train. However, the additional hidden layers of the network enable multi-stage pattern recognition, which can reduce output errors relative to shallow machine learning techniques.

[0174] Deep neural networks used in deep learning typically include a front-end model that performs feature recognition coupled to a back-end network that corresponds to a mathematical model that can perform an action (e.g., object classification, speech recognition, etc.) based on the feature representations provided to the model. Deep learning enables machine learning to be performed without the need for hand-crafted feature engineering to be performed for the model. Instead, deep neural networks can learn features based on statistical structures or correlations in the input data. The learned features can be provided to a mathematical model that can map the detected features to an output. The mathematical model used by the network is generally specialized to perform a specific task, and different models are used to perform different tasks.

[0175] Once a neural network is structured, a learning model can be applied to the network to train it to perform a specific task. The learning model describes how weights within the model should be adjusted to reduce the network's output error. Backpropagation is a common method for training neural networks. An input vector is presented to the network for processing. The network's output is compared to the desired output using a loss function, and an error value is calculated for each of the neurons in the output layer. The error values ​​are then propagated backward until each neuron has an associated error value that roughly represents its contribution to the original output. The network can then learn from these errors using a stochastic gradient descent algorithm to update the neural network's weights.

[0176] 15A-15B illustrate an example convolutional neural network. FIG. 15A illustrates various layers within a CNN. As shown in FIG. 15A, an example CNN used to model image processing can receive input 1502 describing the red, green, and blue (RGB) components of an input image. The input 1502 can be processed by multiple convolutional layers (e.g., a first convolutional layer 1504, a second convolutional layer 1506, etc.). The output from the multiple convolutional layers can optionally be processed by a set of fully connected layers 1508. Neurons in a fully connected layer have full connections to all activations in the previous layer, as described above for feedforward networks. The output from the fully connected layer 1508 can be used to generate an output result from the network. The activations in the fully connected layer 1508 can be calculated using matrix multiplication instead of convolution. Not all CNN implementations use fully connected layers 1508. For example, in some implementations, the second convolutional layer 1506 can generate the output of the CNN.

[0177] Convolutional layers are sparsely connected. This differs from traditional neural network configurations found in fully connected layers 1508. Traditional neural network layers are fully connected, such that every output unit interacts with every input unit. However, convolutional layers, as shown, are sparsely connected because the output of a field's convolution is input to the nodes of a subsequent layer (instead of the individual state values ​​of each of the nodes in the field). The kernels associated with a convolutional layer perform the convolution operation, and the output is sent to the next layer. The dimensionality reduction performed within convolutional layers is one aspect that allows CNNs to scale to process large images.

[0178] 15B shows an example of computational stages within a convolutional layer of a CNN. Input to convolutional layer 1512 of a CNN can be processed in three stages in convolutional layer 1514. The three stages can include a convolution stage 1516, a detection stage 1518, and a pooling stage 1520. Convolutional layer 1514 can then output data to successive convolutional layers. The final convolutional layer of the network can generate output feature map data or can provide input to a fully connected layer, for example, to generate a classification value for the input to the CNN.

[0179] The convolution stage 1516 performs several convolutions in parallel to generate a set of linear activations. The convolution stage 1516 can include affine transformations, which are any transformations that can be specified as linear transformations and translations. Affine transformations include rotations, translations, scaling, and combinations of these transformations. The convolution stage calculates the output of a function (e.g., a neuron) connected to a specific region in its input. The specific region can be determined as a local region associated with the neuron. The neuron calculates the dot product between the neuron's weight and the region in the local input to which the neuron is connected. The output from the convolution stage 1516 defines a set of linear activations that are processed by the convolution layer 1514.

[0180] The linear activations may be processed by a detection stage 1518. In the detection stage 1518, each linear activation is processed by a nonlinear activation function. The nonlinear activation function increases the nonlinear characteristics of the entire network without affecting the fields of each convolutional layer. Several types of nonlinear activation functions may be used. One particular type is the rectified linear unit (ReLU), which uses an activation function defined as f(x) = max(0,x), so that activations are thresholded at zero.

[0181] The pooling stage 1520 uses a pooling function that replaces the output of the second convolutional layer 1506 with a simplified statistic of neighboring outputs. Pooling functions can be used to introduce translation invariance into the neural network, so that small translations to the input do not change the pooled output. Invariance to local translations can be useful in scenarios where the presence of a feature in the input data is more important than the exact location of the feature. Various types of pooling functions can be used in the pooling stage 1520, including max pooling, average pooling, and l2-norm pooling. Furthermore, some CNN implementations do not include a pooling stage. Instead, such implementations substitute an additional convolutional stage with an increased stride relative to the previous convolutional stage.

[0182] The output from the convolutional layer 1514 may then be processed by the next layer 1522, which may be an additional convolutional layer or one of the fully connected layers 1508. For example, the first convolutional layer 1504 in FIG. 15A may output to the second convolutional layer 1506, which may in turn output to the first layer of the fully connected layer 1508.

[0183] FIG. 16 illustrates an example of a recurrent neural network. In a recurrent neural network (RNN), previous stages of the network influence the output of the current stage of the network. RNNs can be constructed in a variety of ways using a variety of functions. The use of RNNs generally involves recursively using mathematical models to predict the future based on previous input sequences. For example, RNNs may be used to perform statistical language modeling to predict upcoming words given previous word sequences. The depicted RNN 1600 can be described as having an input layer 1602 that receives input vectors, a hidden layer 1604 that implements a recurrent function, a feedback mechanism 1605 that allows 'memory' of previous stages, and an output layer 1606 that outputs the results. The RNN 1600 operates on a time-step basis. The state of the RNN at a given time step is influenced based on the previous time step via the feedback mechanism 1605. For a given time step, the state of the hidden layer 1604 is defined by the previous stage and the input at the current time step. The first input (x1) at the first time step may be processed by the hidden layer 1604. The second input (x2) may be processed by the hidden layer 1604 using state information determined during the processing of the first input (x1). A given state is s t =f(Ux t +Ws t-1 ), where U and W are parameter matrices. The function f is typically nonlinear, such as a variant of the hyperbolic tangent function (Tanh) or the rectifier function f(x)=max(0,x). However, the particular mathematical function used in the hidden layer 1604 can vary depending on the specific implementation details of the RNN 1600.

[0184] In addition to the basic CNN and RNN networks described, variations on these networks may be possible. One example of an RNN variation is the very long short-term memory (LSTM) RNN. LSTM RNNs can learn long-term dependencies that may be necessary for processing longer sequences of language. A variation on CC is the convolutional deep belief network, which has a structure similar to a CNN and is trained in a manner similar to a deep belief network. A deep belief network (DBN) is a generative neural network consisting of multiple layers of stochastic (random) variables. DBNs can be trained layer-by-layer using greedy unsupervised learning. The learned weights of a DBN can then be used to provide a pre-trained neural network by determining an optimal initial set of neural network weights.

[0185] FIG. 17 illustrates the training and deployment of a deep neural network. Once a given network is structured for a task, the neural network is trained using a training dataset 1702. Various training frameworks have been developed to enable hardware acceleration of the training process. For example, the machine learning framework 1404 of FIG. 14 may be configured as the training framework 1704. The training framework 1704 can hook into an untrained neural network 1706 to generate a trained neural network 1708, allowing the untrained neural network 1706 to be trained using parallel processing resources as described herein. To begin the training process, initial weights may be selected randomly or by pre-training with a deep belief network. The training cycle is then performed in either a supervised or unsupervised manner.

[0186] Supervised learning is a training method in which training is performed as an indirect operation, for example, when the training data set 1702 includes inputs paired with desired outputs for those inputs, or when the training data set 1702 includes inputs with known outputs and the neural network's outputs are manually scored. The network processes the inputs and compares the resulting outputs to a set of expected or desired outputs. Errors are then backpropagated through the system. The training framework 1704 can make adjustments to adjust the weights controlling the untrained neural network 1706. The training framework 1704 can provide tools to monitor how well the untrained neural network 1706 is converging to a model suitable for generating accurate answers based on known input data. The training process is iterative, with the network weights adjusted to refine the outputs generated by the neural network. The training process can continue until the neural network reaches a statistically desired accuracy associated with the trained neural network 1708. The trained neural network 1708 can then be deployed to implement any number of machine learning operations to generate inference results 1714 based on the input of new data 1712.

[0187] Unsupervised learning is a learning method in which a network attempts to train itself using unlabeled data. Thus, in unsupervised learning, a training dataset 1702 includes input data without any associated output data. The untrained neural network 1706 can learn groupings within the unlabeled inputs and determine how individual inputs relate to the entire dataset. Unsupervised learning can be used to generate self-organizing maps, a type of trained neural network 1708 that can perform operations useful for reducing the dimensionality of data. Unsupervised learning can also be used to perform anomaly detection, which allows for the identification of data points in an input dataset that deviate from normal patterns of data.

[0188] Variations on supervised and unsupervised learning may also be used. Semi-supervised learning is a technique in which the training data set 1702 contains a mixture of labeled and unlabeled data of the same distribution. Incremental learning is a variation of supervised learning in which input data is successively used to further train the model. Incremental learning allows the trained neural network 1708 to adapt to new data 1712 without forgetting the knowledge implanted in the network during initial training.

[0189] The training process, whether supervised or unsupervised, especially for deep neural networks, can be computationally intensive for a single computing node. Instead of using a single computing node, a distributed network of computing nodes can be used to accelerate the training process.

[0190] 18 is a block diagram illustrating distributed learning. Distributed learning is a training model that uses multiple distributed computing nodes to perform supervised or unsupervised learning of a neural network. Each distributed computing node can include one or more host processors and one or more general-purpose processing nodes. As shown, distributed learning can implement model parallelism 1802, data parallelism 1804, or a combination of model and data parallelism 1806.

[0191] In model parallelism 1802, different computational nodes in a distributed system can perform training computations for different portions of a single network. For example, each layer of a neural network can be trained by a different processing node of the distributed system. Advantages of model parallelism include the ability to scale to particularly large models. Separating the computations associated with different layers of a neural network enables the training of extremely large neural networks where all layer weights do not fit into the memory of a single computational node. In some cases, model parallelism can be particularly useful in performing unsupervised learning of large neural networks.

[0192] In data parallelism 1804, different nodes in the distributed network have a complete instance of the model, and each node receives a different portion of the data. The results from the different nodes are then combined. Different approaches to data parallelism are possible, but all data parallel training approaches require techniques to combine results and synchronize model parameters between each node. Examples of approaches to combining data include parameter averaging and update-based data parallelism. Parameter averaging trains each node on a subset of the training data and sets global parameters (e.g., weights, biases) to the average of the parameters from each node. Parameter averaging uses a central parameter server that holds parameter data. Update-based data parallelism is similar to parameter averaging, except that instead of transferring parameters from nodes to a parameter server, updates to the model are transferred. Additionally, update-based data parallelism can be performed in a decentralized manner, where updates are compressed and transferred between nodes.

[0193] The combined model and data parallelism 1806 can be implemented, for example, in a distributed system where each compute node contains multiple GPUs. Each node can have a complete instance of the model, and separate GPUs within each node are used to train different parts of the model.

[0194] Distributed training incurs additional overhead relative to training on a single machine, but the parallel processors and GPGPUs described herein can each implement various techniques to reduce the overhead of distributed training, including techniques that enable high-bandwidth inter-GPU data transfer and accelerated remote data synchronization.

[0195] [Example machine learning application] Machine learning can be applied to solve a variety of technical problems, including, without limitation, computer vision, autonomous driving and navigation, speech recognition, and language processing. Computer vision has traditionally been one of the most active research areas for machine learning applications. Applications of computer vision range from replicating human visual functions, such as face recognition, to creating new categories of visual functions. For example, a computer vision application may be configured to recognize sound waves from vibrations induced in visible objects in a video. Parallel processor-accelerated machine learning allows computer vision applications to be trained using significantly larger training datasets than previously feasible and enables inference systems to be deployed using low-power parallel processors.

[0196] Machine learning accelerated by parallel processors has autonomous driving applications including lane and road sign recognition, obstacle avoidance, navigation, and driving control. Accelerated machine learning techniques can be used to train driving models based on datasets that provide appropriate responses to specific training inputs. The parallel processors described herein can enable fast training of increasingly complex neural networks used in autonomous driving solutions and enable deployment of low-power inference processors on mobile platforms suitable for incorporation into autonomous vehicles.

[0197] Deep neural networks accelerated by parallel processors have enabled a machine learning approach to automatic speech recognition (ASR). ASR involves generating a function that calculates the most likely language sequence given an input acoustic sequence. Accelerated machine learning using deep neural networks has enabled the replacement of hidden Markov models (HMMs) and Gaussian mixture models (GMMs), which were previously used for ASR.

[0198] Parallel processor-accelerated machine learning can also be used to accelerate natural language processing. Automated learning procedures can use statistical inference algorithms to generate models that are robust to erroneous or unfamiliar input. Exemplary natural language processor applications include automatic machine translation between human languages.

[0199] Parallel processing platforms used for machine learning can be divided into training platforms and deployment platforms: training platforms are generally highly parallel and include optimizations to accelerate multi-GPU single-node training and multi-node multi-GPU training, while deployed machine learning (e.g., inference) platforms generally include low-power parallel processors suitable for use in products such as cameras, autonomous robots, and self-driving cars.

[0200] FIG. 19 illustrates one embodiment of a computing device 1900 employing an accelerator 1910. The computing device 1900 (e.g., a smart wearable device, a virtual reality (VR) device, a head-mounted display (HDM), a mobile computer, an Internet of Things (IoT) device, a laptop computer, a desktop computer, a server computer, etc.) may be the same as the data processing system 100 of FIG. 1 , and therefore, for the sake of brevity, clarity, and ease of understanding, many of the details described above with reference to FIGS. 1-18 will not be further discussed or repeated below. As illustrated, in one embodiment, the computing device 1900 is shown as hosting the accelerator 1910. While shown as a separate component, other embodiments may feature the accelerator 1910 being included within the GPU 1914. In yet other embodiments, the accelerator 1910 may be included within the CPU 1912.

[0201] Throughout this document, terms such as "graphics domain" may be synonymously referred to as "graphics processing unit," "graphics processor," or simply "GPU," and similarly, "CPU domain" or "host domain" may be synonymously referred to as "computer processing unit," "application processor," or simply "CPU."

[0202] Computing device 1900 may include any number and type of communications device, such as a large-scale computing system, such as a server computer, a desktop computer, etc., and may further include set-top boxes (e.g., Internet-based cable television set-top boxes, etc.), global positioning system (GPU)-based devices, etc. Computing device 1900 may also include mobile computing devices that function as communications devices, such as mobile phones including smartphones, personal digital assistants (PDAs), tablet computers, laptop computers, e-readers, smart televisions, television platforms, wearable devices (e.g., eyeglasses, watches, bracelets, smart cards, jewelry, clothing accessories, etc.), media players, etc. For example, in one embodiment, computing device 1900 may include a mobile computing device that uses a computing platform that hosts integrated circuits ("ICs"), such as a system-on-a-chip ("SoC" or "SOC") that incorporates various hardware and / or software components of computing device 1900 on a single chip.

[0203] As shown, in one embodiment, computing device 1900 may include any number and types of hardware and / or software components, such as (without limitation) a GPU 1914, a graphics driver (also referred to as a “GPU driver,” “graphics driver logic,” “driver logic,” “user-mode driver (UMD),” “UMD,” “user-mode driver framework (UMDF),” “UMDF,” or simply “driver”) 1916, a CPU 1912, memory 1908, network devices, drivers, etc., and input / output (I / O) sources 1904 such as a touchscreen, touch panel, touchpad, virtual or standard keyboard, virtual or standard mouse, ports, connectors, etc.

[0204] Computing device 1900 may include an operating system (OS) 1906 that serves as an interface between a user and the hardware and / or physical resources of computing device 1900. It is contemplated that CPU 1912 may include one or more processors, such as processor 102 of FIG. 1, while GPU 1914 may include one or more graphics processors (or multiple processors).

[0205] It should be noted that terms such as "node," "computer node," "server," "server device," "cloud computer," "cloud server," "cloud server computer," "machine," "host machine," "device," "computer device," "computer," "computer system," etc. may be used interchangeably throughout this document. It should also be noted that terms such as "application," "software application," "program," "software program," "package," "software package," etc. may be used interchangeably throughout this document. Also, terms such as "job," "input," "request," "message," etc. may be used interchangeably throughout this document.

[0206] 1-13, it is contemplated that some processes in the above-described graphics pipeline are implemented in software, while others are implemented in hardware. The graphics pipeline may be implemented in a graphics processor design in which a CPU 1912 is designed to operate with a GPU 1914, which may be included in or co-located with the CPU 1912. In one embodiment, the GPU 1914 may use any number and type of conventional software and hardware logic to perform conventional functions related to graphics rendering, as well as novel software and hardware logic to execute any number and type of instructions.

[0207] As described above, memory 1908 may include random access memory (RAM) that includes an application database with object information. A memory controller hub, such as memory hub 105 of FIG. 1, may access data in the RAM and transfer it to GPU 1914 for graphics pipeline processing. The RAM may include double data rate RAM (DDR RAM), extended data output RAM (EDO RAM), etc. CPU 1912 interacts with the hardware graphics pipeline to share graphics pipeline functions.

[0208] The processed data is stored in buffers within the hardware graphics pipeline, and state information is stored in memory 1908. The resulting image is then transferred to an I / O source 1904, such as a display component, for display of the image. The display device may be of various types, such as a cathode ray tube (CRT), thin film transistor (TFT), liquid crystal display (LCD), organic light emitting diode (OLED) array, etc., to display information to a user.

[0209] The memory 1908 may include a pre-allocated region of a buffer (e.g., a frame buffer), although those skilled in the art will appreciate that embodiments are not so limited and any memory accessible to the lower graphics pipeline may be used. The computing device 1900 may further include an input / output (I / O) control hub 130, referred to in FIG. 1, such as one or more I / O sources 1904.

[0210] The CPU 1912 may include one or more processors to execute instructions to perform whatever software routines the computer system implements. Instructions often involve some sort of operation performed on data. Both data and instructions are typically designed to have lower latency than the system memory 1908; for example, a cache may be integrated on the same silicon chip as the processor and / or comprise faster static RAM (SRAM) cells, while the system memory 1908 may comprise slower dynamic RAM (DRAM) cells. The tendency to store more frequently used instructions and data in the cache, as opposed to the system memory 1908, improves the overall performance efficiency of the computer device. In some embodiments, it is contemplated that the GPU 1914 may exist as part of the CPU 1912 (e.g., part of the physical CPU package), in which case the memory 1908 may be provided by the CPU 1912 and the GPU 1914 or may remain separate.

[0211] System memory 1908 may be made available to other components within computing device 1900. For example, any data (e.g., input graphics data) received from various interfaces into computing device 1900 (e.g., keyboard and mouse, printer port, local area network (LAN) port, modem port, etc.) or read from internal storage elements (e.g., hard disk drive) of computing device 1900 is often temporarily queued in system memory 1908 before being acted upon by one or more processors in executing a software program. Similarly, data that a software program determines should be sent from computing device 1900 to an external entity through one of the computer system interfaces or stored in an internal storage element is often temporarily queued in system memory 1908 before being transmitted or stored.

[0212] Further, for example, the ICH may be used to ensure that such data is properly passed between system memory 1908 and its appropriate corresponding computer system interfaces (and internal storage devices, if the computer system is so designed), and may have bidirectional point-to-point links between itself and the monitored I / O sources / devices 1904. Similarly, the platform control hub (PCH) may be used to manage various competing requests for access to system memory 1908 between the CPU 1912 and GPU 1914, interfaces, and internal storage elements that may occur close in time to one another.

[0213] I / O sources 1904 may include one or more I / O devices implemented for transferring data to and / or from computing device 1900 (e.g., a networking adapter) or for mass non-volatile storage (e.g., a hard disk drive) within computing device 1900. User input devices, including alphanumeric and other keys, may be used to send information and command selections to GPU 1914. Other types of user input devices are cursor controls, such as a mouse, trackball, touchscreen, touchpad, or cursor direction keys, that send directional information and command selections to GPU 1914 and control cursor movement on a display device. Camera and microphone arrays in computing device 1900 may be used to observe gestures, record audio and video, and send and receive visual and voice commands.

[0214] The computing device 1900 may further include a network interface to provide access to a network such as a LAN, a wide area network (WAN), a metropolitan area network (MAN), a personal area network (PAN), Bluetooth, a cloud network, a mobile network (e.g., third generation (3G), fourth generation (4G), etc.), an intranet, the Internet, etc. The network interface may include, for example, a wireless network interface with an antenna, which may correspond to one or more antennas. The network interface may also include, for example, a wired network interface to communicate with a remote device via a network cable, which may be, for example, an Ethernet cable, a coaxial cable, an optical fiber, a serial cable, or a parallel cable.

[0215] The network interface may provide access to a LAN, for example, by complying with the IEEE 802.11b and / or IEEE 802.11g standards, and / or the wireless network interface may provide access to a personal area network, for example, by complying with the Bluetooth standard. Other wireless network interfaces and / or protocols, including earlier and later versions of the standard, may also be supported. In addition to or instead of communicating via a wireless LAN standard, the network interface may provide wireless communication using, for example, a time division multiple access (TDMA) protocol, a Global Systems for Mobile communications (GSM) protocol, a code division multiple access (CDMA) protocol, and / or any other type of wireless communication protocol.

[0216] The network interface may include one or more communications interfaces such as a modem, a network interface card, or other well-known interface devices, e.g., those used to couple Ethernet, token ring, or other types of physical wired or wireless attachments intended to provide a communications link supporting a LAN or WAN. In this manner, the computer system may also be coupled to a number of peripherals, clients, control surfaces, consoles, or servers via conventional network infrastructures, including, for example, an intranet or the Internet.

[0217] Of course, a lesser or more equipped system than the above examples may be preferred for a particular implementation. Accordingly, the configuration of computing device 1900 may vary from implementation to implementation depending on numerous factors, such as price constraints, performance requirements, technological improvements, or other circumstances. Examples of electronic or computing device 1900 include (without limitation) a mobile device, a personal digital assistant, a mobile computing device, a smartphone, a mobile phone, a handset, a one-way pager, a two-way pager, a messaging device, a computer, a personal computer (PC), a desktop computer, a laptop computer, a notebook computer, a handheld computer, a tablet computer, a server, a server array or server farm, a web server, a network server, an Internet server, a workstation, a minicomputer, a mainframe computer, a supercomputer, a network appliance, a web appliance, a distributed computing system, a multiprocessor system, a processor-based system, a consumer electronics device, a programmable consumer electronics device, a television set, a digital television set, a set-top box, a wireless access point, a base station, a subscriber station, a mobile subscriber center, a wireless network controller, a router, a hub, a gateway, a bridge, a switch, a machine, or combinations thereof.

[0218] Embodiments may be implemented as one or a combination of one or more microchips or integrated circuits interconnected using a parent board, hardwired logic, software stored by a memory device and executed by a microprocessor, firmware, an application specific integrated circuit (ASIC), and / or a field programmable gate array (FPGA). The term "logic" may include, by way of example, software or hardware and / or a combination of software and hardware.

[0219] Embodiments may be provided as a computer program product, which may include one or more machine-readable media having stored thereon machine-executable instructions that, when executed by one or more machines, such as, for example, a computer, a network of computers, or other electronic devices, may cause the one or more machines to perform operations in accordance with the embodiments described herein. Machine-readable media may include, without limitation, floppy diskettes, optical disks, CD-ROMs (Compact Disc-Read Only Memories), and magnetic-optical disks, ROMs, RAMs, EPROMs (Erasable Programmable Read Only Memories), EEPROMs (Electrically Erasable Programmable Read Only Memories), magnetic or optical cards, flash memory, or other types of media / machine-readable media suitable for storing machine-readable instructions.

[0220] Furthermore, embodiments may be downloaded as a computer program product, where the program may be transferred from a remote computer (e.g., a server) to a requesting computer (e.g., a client) by one or more data signals carried and / or modulated by a carrier wave or other propagation medium over a communications link (e.g., a modem and / or network connection).

[0221] According to one embodiment, the accelerator 1910 includes an array 1913 of multiplication hardware (e.g., a systolic array) that performs machine learning matrix multiplication operations. In such an embodiment, operations may be performed on large matrices (e.g., 16x16) by dividing (splitting) the matrix into smaller matrices (e.g., 8x8). Thus, processing the matrix involves a significant number of matrix multiplications, including many multiply-and-accumulate operations.

[0222] Performing multiple matrix multiplications requires a tradeoff between performance and accuracy. For example, hardware may need to support both floating-point 16 (float16) and floating-point 8 (float8) applications. For the same memory bandwidth, float8 may perform twice as many multiply-accumulate operations, albeit with reduced accuracy. However, implementing separate circuits for each mode adds significant area and power costs.

[0223] FIG. 20A shows a conventional 2N-bit multiplier decomposed into four N-bit multipliers and an adder. As shown in FIG. 20A, the multiplication of terms results in the addition of A[N-1:0]×B[N-1:0], A[2N-1:N]×B[n-1:0], A[N-1:0]×B[2N-1:N], and A[2n-1:N]×B[2N-1:N] to the equal A[2N-1:0]×B[2N-1:0]. The same multiplier architecture can be used to calculate 2k×N-bit multiplications (or duplex mode) by using the 2N×2N multiplier shown in FIG. 20A to perform two N×N multiplications. FIG. 20B shows such a multiplication using a multiplier. As shown in FIG. 20B, the 2N×2N multiplier is modulated in duplex mode via an indicator signal for duplex operation (e.g., ∼duplex) to zero the intermediate terms (A2[N-1:0]×B1[N-1:0]×∼duplex and A1[N-1:0]×B2[N-1:0]×∼duplex). The non-zero terms are the products of two N×N multiplications, A2[N-1:0]×B2[N-1:0] and A1[N-1:0]×B1[N-1:0]. A2[N-1:0]×B2[N-1:0] and A1[N-1:0]×B1[N-1:0] are added in later stages by additional hardware. Thus, an additional adder is required to perform 2k×N-bit multiplications, which significantly increases the logic, area, and power required to implement 2N×2N multiplications.

[0224] According to one embodiment, array 1913 includes an optimized dot-product multiplier 1915 (FIG. 19) that supports multiple bit widths without the need to include additional adders. A dot product involves multiplication and addition. For example, given A and B, each M-bit wide, as inputs to a systolic multiplier, there are a number (k) of 2N-bit wide elements, so that in the first case (e.g., case 1):

number

[0225] In the second case (e.g., case 2) where the hardware application supports dot products of 2k N-bit wide elements:

number

[0226] As mentioned above, Case 2 has twice the number of multipliers compared to Case 1, and each individual multiplier is significantly smaller. Moreover, Case 2 has twice the number of elements to add relative to Case 1, even though each element in Case 2 has fewer bits.

[0227] FIG. 21 illustrates one embodiment of the dot product multiplier 1915. As shown in FIG. 21, the multiplier 1915 is a 2N×2N multiplier that includes a multiplication stage 2110 and an addition stage 2120. In one embodiment, the multiplier 1915 can operate in a dot product mode that performs 2k N-bit multiplications in addition to a conventional mode (e.g., performing 2N×2N multiplications). In the dot product mode, the multiplier 1915 is configured as a dot product (A2×B2+A1×B1) of multiple (e.g., 2N) bit vectors to perform an N×N multiplication operation and an addition operation of the N×N multiplication operation in the multiplication stage 2110, rather than implementing the addition operation at a later stage. In such an embodiment, the first and last terms of the multiplication operation are reduced to (or made zero) via an indicator signal (e.g., ~dotp) for the dot product multiplication. Therefore, (A1[N-1:0]×B1[N-1:0])×~dotp and (A2[N-1:0]×B2[N-1:0])×~dotp=0.

[0228] In a further embodiment, the most significant bit and least significant bit of one of the multiplicands (e.g., the first N bits and last N bits (B1 and B2) of input B) are swapped by multiplexing before multiplication. Such swapping allows for vertical multiplication of elements (e.g., A1B1 and A2B2) rather than diagonal multiplication of elements (e.g., A1B2 and A2B1). Adder stage 2120 then performs the addition of A2×(dotp?B2:B1) and A1×(dotp?B1:B2) to yield an output of A2×B2+A1×B1. Therefore, no separate adder is required to perform the operation.

[0229] In other embodiments, multiplier 1915 may generalize the dot product optimization where the received input is divided into K elements. Thus, given A[N×K−1:0] and B[N×K−1:0], where K=the number of piecewise inputs being added and N=the bit width of each of the multiplicands, the multiplier output can be generalized to the following equation for two modes of operation: Conventional Mode Output:

number

number

[0230] Based on the above, all other terms in multiplier 1915 are zeroed. Based on the example of K=3, the above equations are expanded as shown in Figure 22, where A0=a[N-1:0], A1=a[2N-1:N], A2=a[3N-1:2N], and B0=b[N-1:0], B1=b[2N-1:N], B2=b[3N-1:2N]. As shown in Figure 22, all products are set to 0 (or zeroed) except A2 x (dotp?B2:B0); A1 x B1; and A0(dotp?B0:B2).

[0231] 23 is a flow diagram illustrating one embodiment of a process for performing a multiplication operation. At processing block 2310, an input is received by multiplier 1915. At processing block 2320, the input is divided into K elements. At processing block 2330, a mode of operation (e.g., conventional or dot product) is determined. At processing block 2340, the multiplication operation is performed as indicated by the mode of operation (e.g., k 2N-bit multiplication, or 2k N-bit multiplication). In one embodiment, the instruction is provided by the received input. At processing block 2350, the result of the multiplication operation is returned.

[0232] 24 is a flow diagram illustrating one embodiment of a process for performing a dot-product 2k N-bit multiplication operation. At processing block 2410, the multiplier is configured as a dot product. At processing block 2420, the bits of one multiplicand are swapped. At processing block 2430, the multiplication operation is performed. At processing block 2440, the first and last terms of the multiplication operation are zeroed. At processing block 2450, the addition of the middle terms is performed.

[0233] The following sections and / or examples relate to further embodiments or examples. Details in the examples may be used anywhere in one or more embodiments. Various features of different embodiments or examples may be combined in various ways, such as including some features and excluding others, to suit a wide variety of applications. Examples may include objects such as a method, means for performing the operations of the method, at least one machine-readable medium containing instructions that, when executed by a machine, cause the machine to perform the operations of the method, or an apparatus or system that facilitates hybrid communication in accordance with the embodiments and examples described herein.

[0234] Some embodiments relate to Example 1, which includes an apparatus for facilitating matrix multiplication operations, the apparatus having multiplication hardware operating in a dot product mode, wherein a multiplication stage included in the multiplication hardware is configured as a dot product of a plurality of bit vectors (N) to perform N×N multiplication operations on a plurality of multiplicands and to perform addition operations on results of the N×N multiplication operations.

[0235] Example 2 includes the subject matter of example 1, wherein the multiplication stage further swaps the most significant bit and the least significant bit of a first multiplicand of the plurality of multiplicands before performing the multiplication operation.

[0236] Example 3 includes the subject of Examples 1 and 2, where the multiplication stage zeros out the first and last terms of the multiplication operation.

[0237] Example 4 includes the subject matter of examples 1-3, wherein the multiplication hardware further comprises an adder stage that performs an add operation on an intermediate term of the multiplication operation.

[0238] Example 5 includes the subject matter of examples 1-4, wherein the multiplication hardware is further configured to operate in a conventional mode to perform 2N matrix multiply-accumulate operations.

[0239] Example 6 includes the subject matter of examples 1-5, wherein the multiplication hardware receives an input and divides the input into multiple components.

[0240] Example 7 includes the subject of Examples 1 through 6, and determines whether the multiplication hardware should operate in conventional mode or dot product mode.

[0241] Some embodiments relate to Example 8, which includes a method for facilitating a matrix multiplication operation comprising operating multiplication hardware in a dot product mode, the method including configuring a multiplication stage as a dot product of a plurality of bit vectors (n) to perform an N×N multiplication operation on a plurality of multiplicands, and performing an addition operation on results of the N×N multiplication operations.

[0242] Example 9 includes the subject matter of example 8, further comprising swapping a most significant bit and a least significant bit of a first multiplicand of the plurality of multiplicands before performing the multiplication operation.

[0243] Example 10 includes the subject of examples 8 and 9, further comprising forcing the first and last terms of the multiplication operation to zero.

[0244] Example 11 includes the subject matter of examples 8-10, further comprising performing an addition operation on a middle term of the multiplication operation in an addition stage of multiplication hardware.

[0245] Example 12 includes the subject matter of examples 8-11, further comprising configuring the multiplication hardware to operate in a conventional mode to perform 2N matrix multiply-accumulate operations.

[0246] Example 13 includes the subject matter of examples 8-12, further comprising receiving an input with multiplication hardware and splitting the input into multiple components.

[0247] Example 14 includes the subject matter of examples 8-13, further comprising determining whether the multiplication hardware should operate in conventional mode or dot product mode.

[0248] Some embodiments relate to Example 15, which includes a hardware accelerator having a systolic array including multiplication hardware operating in a dot product mode, wherein the multiplication hardware includes multiplication stages configured as dot products of a plurality of bit vectors (N) to perform N×N multiplication operations on a plurality of multiplicands and to perform addition operations on results of the N×N multiplication operations.

[0249] Example 16 includes the subject matter of example 15, wherein the multiplication stage swaps the most significant bit and the least significant bit of a first multiplicand of the plurality of multiplicands before performing the multiplication operation.

[0250] Example 17 includes the subject of Examples 15 and 16, where the multiplication stage zeros out the first and last terms of the multiplication operation.

[0251] Example 18 includes the subject matter of examples 15-17, wherein the multiplication hardware further comprises an adder stage that performs an add operation on an intermediate term of the multiplication operation.

[0252] Example 19 includes the subject matter of examples 15-18, wherein the multiplication hardware is further configured to operate in a conventional mode to perform 2N matrix multiply-accumulate operations.

[0253] Example 20 includes the subject matter of Examples 15-19, and determines whether the multiplication hardware should operate in conventional mode or dot product mode.

[0254] The foregoing description and drawings should be regarded in an illustrative rather than a restrictive sense. Those skilled in the art will understand that various modifications and changes may be made to the embodiments described herein without departing from the broad spirit and scope of the invention as set forth in the appended claims. [Explanation of symbols]

[0255] 100 Processing Systems 102,200 processors 104 cache memory 106 Register File 107,202 processor cores 108,208 graphics processor 109 instruction set 110 Interface Bus 111 Display Devices 112,1910 Accelerators 116,214 Integrated Memory Controller 120 Memory Devices 121 Command 122 Data 130 Platform Controller Hub 204 Internal Cache Unit 206 Shared Cache Unit 219 Graphics Processor Core 330 Computational Accelerators 500 Execution Logic 508,600 execution units 612 Systolic Array 1900 Computer Devices 1915 dot product multiplier 2110 multiplication stage 2120 Addition Stage

Claims

1. 1. An apparatus for facilitating matrix multiplication operations, comprising: multiplication hardware operating in a dot product mode, the multiplication hardware comprising: obtaining a first multiplicand consisting of K consecutive N-bit elements and a second multiplicand consisting of K consecutive N-bit elements, where K is an integer equal to or greater than 2; reversing the order of the elements of the first multiplicand so that the most significant N bits become the least significant N bits and vice versa; performing an N×N multiplication operation on the second multiplicand and the first multiplicand with the elements in reverse order; The following formula: [Equation 1] and configured to perform an addition operation on a result of the N×N multiplication operation according to In the formula, a[x:y] represents each N-bit element of the second multiplicand from bit x to bit y, b[w:z] represents each N-bit element of the first multiplicand from bit w to bit z, and dotp?P:Q represents reversing the order of elements P to Q. Device.

2. the multiplication hardware is further configured to operate in a conventional mode to perform 2N matrix multiply-accumulate operations on the first multiplicand and the second multiplicand.

10. The apparatus of claim 1.

3. The multiplication hardware operating in the conventional mode performs the following operation: [Equation 2] performing the matrix multiply-accumulate operation according to 3. The apparatus of claim 2.

4. the multiplication hardware receives inputs and divides each of the inputs into a plurality of N-bit elements; 4. An apparatus according to any one of claims 1 to 3.

5. determining whether the multiplication hardware should operate in the conventional mode or the dot product mode; 4. The device according to claim 2 or 3.

6. 1. A method for facilitating a matrix multiplication operation comprising operating multiplication hardware in a dot product mode, comprising: obtaining a first multiplicand consisting of K consecutive N-bit elements and a second multiplicand consisting of K consecutive N-bit elements, where K is an integer greater than or equal to 2; reversing the order of the elements of the first multiplicand so that the most significant N bits become the least significant N bits and vice versa; performing an N×N multiplication operation on the second multiplicand and the first multiplicand with the elements in reverse order; The following formula: [Equation 3] performing an addition operation on the results of the N×N multiplication operations according to Including, In the formula, a[x:y] represents each N-bit element of the second multiplicand from bit x to bit y, b[w:z] represents each N-bit element of the first multiplicand from bit w to bit z, and dotp?P:Q represents reversing the order of elements P to Q. method.

7. further comprising configuring the multiplication hardware to operate in a conventional mode to perform 2N matrix multiply-accumulate operations on the first multiplicand and the second multiplicand. The method of claim 6.

8. The multiplication hardware operating in the conventional mode performs the following operation: [Equation 4] performing the matrix multiply-accumulate operation according to The method of claim 7.

9. receiving an input at said multiplication hardware; dividing each of said inputs into a plurality of N-bit elements; Further comprising:

9. The method according to any one of claims 6 to 8.

10. determining whether the multiplication hardware should operate in the conventional mode or the dot product mode.

9. The method according to claim 7 or 8.

11. a systolic array including multiplication hardware operating in a dot product mode, said multiplication hardware comprising: obtaining a first multiplicand consisting of K consecutive N-bit elements and a second multiplicand consisting of K consecutive N-bit elements, where K is an integer equal to or greater than 2; reversing the order of the elements of the first multiplicand so that the most significant N bits become the least significant N bits and vice versa; performing an N×N multiplication operation on the second multiplicand and the first multiplicand with the elements in reverse order; The following formula: [Equation 5] and performing an addition operation on a result of the N×N multiplication operation according to In the formula, a[x:y] represents each N-bit element of the second multiplicand from bit x to bit y, b[w:z] represents each N-bit element of the first multiplicand from bit w to bit z, and dotp?P:Q represents reversing the order of elements P to Q. Hardware accelerator.

12. the multiplication hardware is further configured to operate in a conventional mode to perform 2N matrix multiply-accumulate operations on the first multiplicand and the second multiplicand. The hardware accelerator of claim 11.

13. The multiplication hardware operating in the conventional mode performs the following operation: [Equation 6] performing the matrix multiply-accumulate operation according to The hardware accelerator of claim 12.

14. determining whether the multiplication hardware should operate in the conventional mode or the dot product mode; 14. The hardware accelerator according to claim 12 or 13.

Images