Sparse Optimization for Matrix Accelerator Architecture

Abstract

To provide a technique for performing arithmetic on sparse data with a systolic processing unit.SOLUTION: Embodiments described in the present application include software, firmware, and hardware logic that provide a technique for performing arithmetic on sparse data with a systolic processing unit. An embodiment described in the present application provides a technique for skipping computation operations on a zero-filling matrix and a sub-matrix. The embodiment further provides a technique for maintaining data compression up to a processing unit. The embodiment further provides architecture for a sparse aware logic unit.SELECTED DRAWING: Figure 30A

Description

Technical Field

[0001] [Cross - Reference to Related Applications] This application claims the benefit of priority under 35 U.S.C. 119(e) to U.S. Provisional Patent Application No. 62 / 819,337, filed on March 15, 2019, by Abhishek Appu et al. and titled "GRAPHICS PROCESSING" (Attorney Docket No. AC0271 - Z); U.S. Provisional Patent Application No. 62 / 819,435, filed on March 15, 2019, by Lakshminarayanan Striramassarma et al. and titled "GRAPHICS PROCESSING" (Attorney Docket No. AC0285 - Z); U.S. Provisional Patent Application No. 62 / 819,361, filed on March 15, 2019, by Subramaniam Maiyuran et al. and titled "SYSTEMS AND METHODS FOR PARTITIONING CACHE TO REDUCE CACHE ACCESS LATENCY" (Attorney Docket No. AC0286 - Z); and U.S. Provisional Patent Application No. 62 / 935,670, filed on November 15, 2019, by Abhishek Appu et al. and titled "SYSTOLIC ARITHMETIC ON SPARSE DATA" (Attorney Docket No. AC5197 - Z). These prior U.S. patent applications are hereby incorporated by reference in their entirety.

[0002] [Field] This application generally relates to data processing, and more specifically, to sparse data processing by a matrix accelerator of a general - purpose graphics processing unit.

Background Art

[0003] Current parallel graphics data processing includes systems and methods developed to perform certain operations on graphics data, such as, for example, linear interpolation, tessellation, rasterization, texture mapping, depth testing, and the like. Conventionally, graphics processors used fixed-function computing units to process graphics data, but more recently, portions of the graphics processor have become programmable, enabling such processors to support a wider variety of operations for processing vertex and fragment data.

[0004] To further improve performance, graphics processors typically implement processing techniques such as pipelining that attempt to process as much graphics data in parallel as possible across different parts of the graphics pipeline. A parallel graphics processor with a SIMT (single instruction, multiple thread) architecture is designed to maximize the amount of parallel processing in the graphics pipeline. In a SIMT architecture, a group of parallel threads attempts to execute program instructions together simultaneously as frequently as possible to increase processing efficiency. An overview of the software and hardware for a SIMT architecture can be found in Shane Cook, CUDA Programming Chapter 3, pages 37-51 (2013).

Brief Description of the Drawings

[0005] To enable a more detailed understanding of the above features of the present embodiment, the embodiments briefly summarized above are described more specifically with reference to the embodiments, some of which are illustrated in the accompanying drawings. It should be noted that the accompanying drawings represent only typical embodiments and should not be considered as limiting the scope thereof.

[0006]

Figure 1

Figure 2A

Figure 2B

Figure 2C

Figure 2D

Figure 3A

Figure 3B

Figure 3C

Figure 4A

Figure 4B

Figure 4C

Figure 4D

Figure 4E

Figure 4F

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9A

Figure 9B

Figure 10

Figure 11

Figure 12

Figure 13

Figure 14

Figure 15A

Figure 15B

Figure 15C

Figure 16A

Figure 16B

Figure 16C

Figure 17

Figure 18A

Figure 18B

Figure 19

Figure 20

Figure 21

Figure 22A

Figure 22B

Figure 23

Figure 24A

Figure 24B

Figure 24C

Figure 24D

Figure 25

Figure 26A

Figure 26B

Figure 27

Figure 28

Figure 29A

Figure 29B

Figure 30A

Figure 30B

Figure 31A

Figure 31B

Figure 32

Figure 33A

Figure 33B

Figure 34

Figure 35

Figure 36

Figure 37

Best Mode for Carrying Out the Invention

[0007] A graphics processing unit (GPU) is communicatively coupled to a host / processor core, for example, to accelerate graphics operations, machine learning operations, pattern analysis operations, and / or various general-purpose GPU (GPGPU) functions. The GPU may be communicatively coupled to the host processor / core via a bus or other interconnect (e.g., a high-speed interconnect such as PCIe or NVLink). Alternatively, the GPU may be integrated on the same package or die as the core and communicatively coupled to the core via an internal processor bus / interconnect (i.e., within the package or die). Regardless of how the GPU is connected, the processor core may assign work to the GPU in the form of a sequence of commands / instructions contained in a work descriptor. The GPU then uses dedicated circuitry / logic to efficiently process these commands / instructions.

[0008] Embodiments described herein include software, firmware, and hardware logic that provide techniques for performing arithmetic on sparse data by a systolic processing unit. Embodiments described herein provide techniques for skipping computational operations on zero-filled matrices and submatrices. Embodiments further provide techniques for maintaining data compression up to the processing unit. Embodiments further provide an architecture for a sparse-aware logic unit.

[0009] In the following description, numerous specific details are set forth in order to provide a more thorough understanding. However, as will be apparent to one of ordinary skill in the art, embodiments described herein may be practiced without one or more of these specific details. In other instances, well-known features have not been described in detail so as not to obscure the details of the embodiments.

[0010] [System Overview] FIG. 1 is a block diagram representing a computing system 100 configured to implement one or more aspects of the embodiments described in the present application. The computing system 100 includes a processing subsystem 101 having one or more processors 102 and a system memory 104 that communicate via an interconnect path. The interconnect path may include a memory hub 105. The memory hub 105 may be a separate component within a chipset component or may be incorporated within one or more processors 102. The memory hub 105 is coupled to an I / O subsystem 111 via a communication link 106. The I / O subsystem 111 includes an I / O hub 107 that can enable the computing system 100 to receive inputs from one or more input devices 108. Further, the I / O hub 107 can enable a display controller, which may be included in one or more processors 102, to supply outputs to one or more display devices 110A. In one embodiment, the one or more display devices 110A coupled to the I / O hub 107 can include local, built-in, or embedded display devices.

[0011] The processing subsystem 101 includes, for example, one or more parallel processors 112 coupled to a memory hub 105 via a bus or other communication link 113. The communication link 113 can be one of any number of communication link technologies or protocols based on, but not limited to, standards such as PCI Express, or alternatively, a vendor-specific communication interface or communication fabric. The one or more parallel processors 112 can form a computationally focused parallel or vector processing system that can include a number of processing cores and / or processing clusters, such as a MIC (many integrated core) processor. For example, the one or more parallel processors 112 can form a graphics processing system that can output pixels to one of one or more display devices 110A coupled via an I / O hub 107. The one or more parallel processors 112 can also include a display controller and a display interface (not shown) that enables direct connection to one or more display devices 110B.

[0012] Within the I / O subsystem 111, the system memory unit 114 is connectable to the I / O subsystem 111 to provide a storage mechanism for the computing system 100. The I / O switch 116 is used to provide an interface mechanism that enables connection between the I / O hub 107 and other components such as a network adapter 118 and / or a wireless network adapter 119 that can be integrated into the platform, and various other devices that can be added via one or more add-in devices 120. The add-in device 120 can also include, for example, one or more external graphics processor devices and / or computing accelerators. The network adapter 118 can be an Ehternet (registered trademark) adapter or other wired network adapter. The wireless network adapter 119 can include one or more of Wi-Fi, Bluetooth (registered trademark), near field communication (NFC), or other network devices that include one or more wireless radios.

[0013] Computing system 100 can include other components not shown, including USB or other port connections, and optical memory drivers, video capture devices, etc. may also be connected to I / O hub 107. The communication paths interconnecting the various components in FIG. 1 may be implemented using any suitable protocol, such as a protocol based on PCI (Peripheral Component Interconnect) (e.g., PCI-Express), or any other bus or point-to-point communication interface and / or protocol, such as an NV-Link high-speed interconnect or an interconnect protocol known in the art.

[0014] One or more parallel processors 112 may incorporate a circuit configuration optimized for graphics and video processing, including, for example, a video output circuit configuration, and constitute a graphics processing unit (GPU). Alternatively, or in addition, one or more parallel processors 112 may incorporate a circuit configuration optimized for general-purpose processing while retaining the underlying computational architecture, as described in more detail herein. The components of computing system 100 may be integrated with one or more other system elements on a single integrated circuit. For example, one or more parallel processors 112, memory hub 105, processor 102, and I / O hub 107 may be incorporated into a system-on-chip (SoC) integrated circuit. Alternatively, the components of computing system 100 may be incorporated into a single package to form a system-in-package (SIP) configuration. In one embodiment, at least some of the components of computing system 100 may be incorporated into a multi-chip module (MCM) that can be interconnected with other multi-chip modules within a modular computing system.

[0015] The computing system 100 shown in this application is an example, and it will be understood that modifications and variations are possible. The connection topology, including the number and arrangement of bridges, the number of processors 102, and the number of parallel processors 112, may be changed as desired. For example, the system memory 104 may be connected directly to the processor 102 rather than through a bridge, while other devices communicate with the system memory 104 via the memory hub 105 and the processor 102. In other alternative topologies, the parallel processor 112 may be connected directly to the I / O hub 107, or to one of the one or more processors 102, rather than to the memory hub 105. In other embodiments, the I / O hub 107 and the memory hub 105 may be incorporated into a single chip. Also, two or more sets of processors 102 may be attached via a plurality of sockets that can be coupled to two or more instances of the parallel processor 112.

[0016] Some of the specific components shown here are optional and may not be included in all implementations of the computing system 100. For example, any number of add-in cards or peripherals may be supported, or some components may be removed. Furthermore, some architectures may use different terms for components similar to those shown in FIG. 1. For example, the memory hub 105 may sometimes be referred to as a north bridge in some architectures, while the I / O hub 107 may sometimes be referred to as a south bridge.

[0017] FIG. 2A represents a parallel processor 200. The parallel processor 200 may be a GPU, GPGPU, etc. described in this application. The various components of the parallel processor 200 may be implemented using one or more integrated circuit devices such as programmable processors, application specific integrated circuits (ASICs), or field programmable gate arrays (FPGAs). The represented parallel processor 200 may be the parallel processor 112 shown in FIG. 1 or one of them.

[0018] The parallel processor 200 includes a parallel processing unit 202. The parallel processing unit 202 includes an I / O unit 204 that enables communication with other devices including other instances of the parallel processing unit 202. The I / O unit 204 may be directly connected to other devices. For example, the I / O unit 204 connects to other devices through the use of a hub or switch interface such as the memory hub 105. The connection between the memory hub 105 and the I / O unit 204 forms a communication link 113. Within the parallel processing unit 202, the I / O unit 204 is connected to a host interface 206 and a memory crossbar 216. The host interface 206 receives commands directed to performing processing operations, and the memory crossbar 216 receives commands directed to performing memory operations.

[0019] When host interface 206 receives command buffers via I / O unit 204, host interface 206 can direct work operations for executing those commands to front end 208. In one embodiment, front end 208 is coupled to scheduler 210, and scheduler 210 is configured to distribute commands or other work items to processing cluster array 212. Scheduler 210 ensures that processing cluster array 212 is properly configured and in a valid state before tasks are distributed to the processing clusters of processing cluster array 212. Scheduler 210 may be implemented by firmware logic executed on a microcontroller. Scheduler 210 implemented by a microcontroller is configured to perform complex scheduling and work distribution operations at coarse and fine granularities while allowing for rapid preemption and context switching of threads executing in processing array 212. Desirably, host software can demonstrate a workload for scheduling to processing array 212 by one of a plurality of graphics processing doorbells. The workload can then be automatically distributed across processing array 212 by the logic of scheduler 210 within the scheduler microcontroller.

[0020] The processing cluster array 212 can include up to "N" processing clusters (e.g., cluster 214A, cluster 214B, or cluster 214N). Each of the clusters 214A - 214N of the processing cluster array 212 can execute a number of simultaneous threads. The scheduler 210 can assign work to the clusters 214A - 214N of the processing cluster array 212 using various scheduling and / or work distribution algorithms that can vary depending on the workload generated for each type of program or calculation. The scheduling can be dynamically processed by the scheduler 210 or can be partially assisted by compiler logic during the compilation of the program logic configured for execution by the processing cluster array 212. Optionally, different clusters 214A - 214N of the processing cluster array 212 can be assigned to process different types of programs and to perform different types of calculations.

[0021] The processing cluster array 212 can be configured to perform various types of parallel processing operations. For example, the cluster array 212 can be configured to perform general-purpose parallel computing operations. For example, the processing cluster array 212 can include logic for performing processing tasks including filtering of video and / or audio data, execution of modeling operations including physical computations, and execution of data transformation.

[0022] The processing cluster array 212 is configured to execute parallel graphics processing operations. In such embodiments where the parallel processor 200 is configured to execute graphics processing operations, the processing cluster array 212 can include texture sampling logic that executes texture operations along with tessellation logic and other vertex processing logic, and can further include additional logic to support the execution of such graphics processing operations, which is not limited thereto. Furthermore, the processing cluster array 212 can be configured to execute graphics processing related shader programs such as, but not limited to, vertex shaders, tessellation shaders, geometry shaders, and pixel shaders. The parallel processing unit 202 can transfer data from the system memory through the I / O unit 204 for processing. During processing, the transferred data can be stored in on-chip memory (e.g., parallel processor memory 222) during processing and then written back to the system memory.

[0023] In embodiments where the parallel processing unit 202 is used to execute graphics processing, the scheduler 210 can be configured to divide the processing workload into tasks of approximately equal size to better enable the distribution of graphics processing operations to the multiple clusters 214A - 214N of the processing cluster array 212. In some of these embodiments, portions of the processing cluster array 212 can be configured to execute different types of processing. For example, to generate a rendered image for display, a first portion can be configured to execute vertex shading and topology generation, a second portion can be configured to execute tessellation and geometry shading, and a third portion can be configured to execute pixel shading or other screen space operations. Intermediate data generated by one or more of the clusters 214A - 214N can be stored in a buffer to enable the intermediate data to be transmitted between the clusters 214A - 214N for further processing.

[0024] During operation, the processing cluster array 212 can receive processing tasks to be executed via the scheduler 210, and the scheduler 210 receives commands defining the processing tasks from the front end 208. For graphics processing operations, the processing tasks can include an index of the data to be processed, e.g., surface (patch) data, primitive data, vertex data, and pixel data, along with state parameters and commands that define how the data is to be processed (e.g., which programs are to be executed). The scheduler 210 may be configured to fetch the index corresponding to the task, or may receive the index from the front end 208. The front end 208 may be configured to ensure that the processing cluster array 212 is set to an active state before the workload specified by an incoming command buffer (e.g., batch buffer, push buffer, etc.) is initiated.

[0025] Each of one or more instances of the parallel processing unit 202 can be coupled to a parallel processor memory 222. The parallel processor memory 222 can be accessed via a memory crossbar 216. The memory crossbar 216 can receive memory requests from the processing cluster array 212 and the I / O unit 204. The memory crossbar 216 can access the parallel processor memory 222 via a memory interface 218. The memory interface 218 can include a plurality of partition units (e.g., partition unit 220A, partition unit 220B, or partition unit 220N) to which each can be coupled to a portion (e.g., a memory unit) of the parallel processor memory 222. The number of partition units 220A - 220N can be configured to be equal to the number of memory units, such that the first partition unit 220A has a corresponding first memory unit 224A, the second partition unit 220B has a corresponding memory unit 224B, and the Nth partition unit 220N has a corresponding Nth memory unit 224N. In other embodiments, the number of partition units 220A - 220N may not be equal to the number of memory devices.

[0026] Memory units 224A - 224N can include various types of memory devices, including dynamic random access memory (DRAM) containing graphics double data rate (GDDR) memory or graphics random access memory, such as synchronous graphics random access memory (SGRAM). Optionally, memory units 224A - 224N can also include 3D stacked memory, including but not limited to high bandwidth memory (HBM). As will be apparent to those skilled in the art, memory units 224A - 224N can be varied and selectable from one of various conventional designs. Render targets, such as frame buffers or texture maps, can be stored across memory units 224A - 224N, enabling partition units 220A - 220N to write portions of each render target to efficiently use the available bandwidth of parallel processor memory 222. In some embodiments, a local instance of parallel processor memory 222 can support an integrated memory design that utilizes system memory along with local cache memory and can be excluded.

[0027] Optionally, any one of clusters 214A through 214N of the processing cluster array 212 is capable of processing data that is to be written to any one of memory units 224A through 224N within the parallel processor memory 222. The memory crossbar 216 can be configured to transfer the output of each of clusters 214A through 214N to any one of the partition units 220A through 220N or to another one of clusters 214A through 214N that can perform additional processing operations on the output. Each of clusters 214A through 214N can communicate with the memory interface 218 through the memory crossbar 216 to read from or write to various external memory devices. In one of the embodiments with the memory crossbar 216, the memory crossbar 216 has a connection to the memory interface 218 for communicating with the I / O unit 204 and a connection to a local instance of the parallel processor memory 222, enabling processing units within different processing clusters 214A through 214N to communicate with system memory or other memory that is not local to the parallel processing unit 202. Generally, the memory crossbar 216 may be able to use virtual channels, for example, to separate traffic streams between clusters 214A through 214N and partition units 220A through 220N.

[0028] Although a single instance of the parallel processing unit 202 is represented within the parallel processor 200, any number of instances of the parallel processing unit 202 may be included. For example, multiple instances of the parallel processing unit 202 may be provided on a single add-in card, or multiple add-in cards may be interconnected. Different instances of the parallel processing unit 202 may be configured to interoperate, even if the different instances have different numbers of processing cores, different amounts of local parallel processor memory, and / or other configuration differences. Optionally, some instances of the parallel processing unit 202 may include floating-point units with higher precision than other instances. A system incorporating one or more instances of the parallel processing unit 202 or the parallel processor 200 may be implemented in a variety of configurations and form factors, including, but not limited to, desktop, laptop, or portable personal computers, servers, workstations, game consoles, and / or embedded systems.

[0029] Figure 2B is a block diagram of partition unit 220. Partition unit 220 may be an instance of one of partition units 220A - 220N of Figure 2A. As shown, partition unit 220 includes an L2 cache 221, a frame buffer interface 225, and a ROP 226 (raster operations unit). The L2 cache 221 is a read / write cache configured to execute load and store operations received from the memory crossbar 216 and the ROP 226. Read misses and urgent write-back surfaces are output by the L2 cache 221 to the frame buffer interface 225 for processing. Updates may also be sent to the frame buffer via the frame buffer interface 225 for processing. In one embodiment, the frame buffer interface 225 interfaces with one of the memory units within a parallel processor memory, such as memory units 224A - 224N of Figure 2A (e.g., within parallel processor memory 222). Partition unit 220 may additionally or alternatively interface with one of the memory units within the parallel processor memory via a memory controller (not shown).

[0030] In a graphics application, the ROP 226 is a processing unit that performs raster operations such as stencil, z - test, blending, etc. The ROP 226 then outputs the processed graphics data, and the processed graphics data is stored in the graphics memory. In some embodiments, the ROP 226 includes compression logic that compresses depth or color data written to the memory and decompresses depth or color data read from the memory. The compression logic can be lossless compression logic that uses one or more of a plurality of compression algorithms. The type of compression performed by the ROP 226 can vary based on the statistical characteristics of the data to be compressed. For example, in one embodiment, delta color compression is performed on depth and color data on a tile - by - tile basis.

[0031] ROP226 may be included within each processing cluster (e.g., clusters 214A - 214N of FIG. 2A) rather than within partition unit 220. In such embodiments, read and write requests for pixel data are transmitted via memory crossbar 216 instead of pixel fragment data. The processed graphics data is displayed on a display device such as one of the one or more display devices 110 of FIG. 1, routed for further processing by processor 102, or routed for further processing by one of the processing entities within parallel processor memory 222 of FIG. 2A.

[0032] FIG. 2C is a block diagram of processing cluster 214 within the parallel processing unit. For example, the processing cluster is an instance of one of clusters 214A - 214N of FIG. 2A. Processing cluster 214 may be configured to execute multiple threads simultaneously. Here, the term "thread" refers to an instance of a particular program that executes on a particular set of input data. Optionally, SIMD (single-instruction, multiple-data) instruction issue techniques may be used to support parallel execution of multiple threads without providing a large number of independent instruction units. Alternatively, SIMT (single-instruction, multiple-thread) techniques may be used to support parallel execution of a large number of generally synchronized threads using a common instruction unit configured to issue instructions to a set of processing engines within each of the processing clusters. Unlike SIMD execution regimes where all processing engines typically execute the same instruction, SIMT execution allows different threads to more easily follow different execution paths through a given thread program. Those skilled in the art will understand that the SIMD processing regime represents a functional subset of the SIMT processing regime.

[0033] The operation of processing cluster 214 can be controlled via a pipeline manager 232 that distributes processing tasks to SIMT parallel processors. The pipeline manager 232 receives instructions from the scheduler 210 of FIG. 2 and manages the execution of these instructions by the graphics multiprocessor 234 and / or texture unit 236. The illustrated graphics multiprocessor 234 is an exemplary instance of a SIMT parallel processor. However, various types of SIMT parallel processors of different architectures may be included within processing cluster 214. One or more instances of the graphics multiprocessor 234 may be included within processing cluster 214. The graphics multiprocessor 234 can process data, and the data crossbar 240 can be used to distribute the processed data to one of a plurality of possible destinations including other shader units. The pipeline manager 232 can facilitate the distribution of the processed data by specifying a destination for the processed data to be distributed via the data crossbar 240.

[0034] Each graphics multiprocessor 234 within processing cluster 214 can include the same set of functional execution logic (e.g., arithmetic logic units, load-store units, etc.). The functional execution logic can be configured in a pipelined manner where new instructions can be issued before the previous instruction has completed. The functional execution logic supports various operations including integer and floating point arithmetic, comparison operations, boolean operations, bit shifts, and calculations of various algebraic functions. The same functional unit hardware is available for performing various operations, and any combination of functional units may exist.

[0035] The instructions transmitted to processing cluster 214 constitute threads. A set of threads that are executed across the parallel processing engines is a thread group. The thread group executes the same program for different input data. Each thread within the thread group can be assigned to a different processing engine within graphics multiprocessor 234. The number of threads included in the thread group may be less than the number of processing engines within graphics multiprocessor 234. When the number of threads included in the thread group is less than the number of processing engines, one or more of the processing engines may be in an idle state during the cycles in which the thread group is being processed. The number of threads included in the thread group may also be more than the number of processing engines within graphics multiprocessor 234. When the number of threads included in the thread group is more than the number of processing engines within graphics multiprocessor 234, the processing can be executed over consecutive clock cycles. Optionally, multiple thread groups can be executed simultaneously on graphics multiprocessor 234.

[0036] The graphics multiprocessor 234 may include an internal cache memory to perform load and store operations. Optionally, the graphics multiprocessor 234 may prefetch the internal cache and use the cache memory (e.g., L1 cache 248) within the processing cluster 214. Each graphics multiprocessor 234 also has access to the L2 cache within a partition unit (e.g., partition units 220A-220N of FIG. 2A) that can be shared across all processing clusters 214 and used to transfer data between threads. The graphics multiprocessor 234 may also access off-chip global memory. The off-chip global memory can include one or more of local parallel processor memory and / or system memory. Any memory outside the parallel processing unit 202 may be used as global memory. In embodiments where the processing cluster 214 includes multiple instances of the graphics multiprocessor 234, common instructions and data that can be stored in the L1 cache 248 can be shared.

[0037] Each processing cluster 214 may include an MMU 245 (memory management unit) configured to map virtual addresses to physical addresses. In other embodiments, one or more instances of the MMU 245 may be present within the memory interface 218 of FIG. 2A. The MMU 245 includes a set of page table entries (PTEs) used to map virtual addresses to the physical address of a tile and optionally a cache line index. The MMU 245 may include an address translation lookaside buffer (TLB) or cache that may be present within the graphics multiprocessor 234 or the L1 cache 248 or the processing cluster 214. The physical address is processed to disperse the locality of surface data access to enable efficient request interleaving between partition units. The cache line index may be used to determine whether a request for a cache line is a hit or a miss.

[0038] In graphics and computing applications, the processing cluster 214 may be configured such that each graphics multiprocessor 234 is coupled to a texture unit 236 for performing texture mapping operations, e.g., determining texture sample positions, reading texture data, and filtering texture data. The texture data may be read from an internal texture L1 cache (not shown) or, in some embodiments, from an L1 cache within the graphics multiprocessor 234 and, if necessary, fetched from an L2 cache, local parallel processor memory, or system memory. Each graphics multiprocessor 234 outputs the processed task to a data crossbar 240 such that the processed task is supplied to other processing clusters 214 for further processing or stored in an L2 cache, local parallel processor memory, or system memory via a memory crossbar 216. The preROP242 (pre-raster operations unit) is configured to receive data from the graphics multiprocessor 234 and direct the data to a ROP unit that can be positioned by a partitioning unit (e.g., partitioning units 220A - 220N of FIG. 2A) described herein. The preROP242 unit can perform optimizations for color mixing, compose pixel color data, and perform address translation.

[0039] The core architectures described in this application are examples and it will be understood that they can be changed and modified. Any number of processing units, for example, graphics multiprocessor 234, texture unit 236, preROP 242, etc. may be included in processing cluster 214. Further, although only one processing cluster 214 is shown, the parallel processing units described in this application may include any number of instances of processing cluster 214. Optionally, each processing cluster 214 may be configured to operate independently from other processing clusters 214 using separate and distinct processing units, L1 caches, etc.

[0040] FIG. 2D shows an example of a graphics multiprocessor 234 coupled to a pipeline manager 232 of a processing cluster 214. The graphics multiprocessor 234 includes an execution pipeline including, but not limited to, an instruction cache 252, an instruction unit 254, an address mapping unit 256, a register file 258, one or more general-purpose graphics processing unit (GPGPU) cores 262, and one or more load / store units 266. The GPGPU cores 262 and the load / store units 266 are coupled to cache memory 272 and shared memory 270 via a memory and cache interconnect 268. The graphics multiprocessor 234 may further include a tensor and / or ray tracing core 263 including hardware logic for accelerating matrix and / or ray tracing operations.

[0041] The instruction cache 252 may receive a stream of instructions from the pipeline manager 232. The instructions are cached in the instruction cache 252 and dispatched for execution by the instruction unit 254. The instruction unit 254 can dispatch instructions as a thread group (e.g., a warp), and each thread of the thread group is assigned to a different execution unit within the GPGPU core 262. Instructions can access any of the local, shared, or global address spaces by specifying an address within the unified address space. The address mapping unit 256 can be used to convert an address within the unified address space to a different memory address that can be accessed by the load / store unit 266.

[0042] The register file 258 provides a set of registers for the functional units of the graphics multiprocessor 234. The register file 258 provides temporary storage of operands connected to the data paths of the functional units of the graphics multiprocessor 234 (e.g., the GPGPU core 262, the load / store unit 266). The register file 258 may be divided among the respective functional units such that each functional unit is assigned a dedicated portion of the register file 258. For example, the register file 258 is divided among different warps executed by the graphics multiprocessor 234.

[0043] The GPGPU core 262 can include a floating-point unit (FPU) and / or an integer arithmetic logic unit (ALU) respectively used to execute the instructions of the graphics multiprocessor 234. In some implementations, the GPGPU core 262 can include hardware logic that would otherwise be present within the tensor and / or ray tracing core 263. The GPGPU cores 262 can have similar architectures or, alternatively, different architectures. For example, in one embodiment, the first part of the GPGPU core 262 includes a single-precision FPU and an integer ALU, while the second part of the GPGPU core 262 includes a double-precision FPU. Optionally, the FPU can implement IEEE 754-2008 for floating-point arithmetic or enable variable-precision floating-point arithmetic. The graphics multiprocessor 234 can further include one or more fixed or special functions to execute specific functions such as copy rectangle or pixel blend operations. One or more of the GPGPU cores can also include fixed or special function logic.

[0044] The GPGPU core 262 may include SIMD logic that can execute a single instruction on multiple sets of data. Optionally, the GPGPU core 262 can physically execute SIMD4, SIMD8, and SIMD16 instructions and logically execute SIMD1, SIMD2, and SIMD32 instructions. The SIMD instructions for the GPGPU core are generated at compile time by a shader compiler or can be automatically generated when executing a program written and compiled for a single program multiple data (SPMD) or SIMT architecture. Multiple threads of a program configured for the SIMT execution model are executable by a single SIMD instruction. For example, in one embodiment, eight SIMT threads performing the same or similar operations can be executed in parallel by a single SIMD8 logic unit.

[0045] The memory and cache interconnect 268 is an interconnect network that connects each of the functional units of the graphics multiprocessor 234 to the register file 258 and to the shared memory 270. For example, the memory and cache interconnect 268 is a crossbar interconnect that enables the load / store unit 266 to implement load and store operations between the shared memory 270 and the register file 258. Since the register file 258 can operate at the same frequency as the GPGPU core 262, the data transfer between the GPGPU core 262 and the register file 258 has a very small latency. The shared memory 270 can be used to enable communication between threads executed by functional units within the graphics multiprocessor 234. The cache memory 272 can be used as a data cache, for example, to cache texture data communicated between a functional unit and the texture unit 236. The shared memory 270 can also be used as a program that is cached and managed. Threads executed on the GPGPU core 262 can programmably store data in the shared memory in addition to the automatically cached data stored in the cache memory 272.

[0046] Figures 3A-3C represent additional graphics multiprocessors according to an embodiment. Figures 3A-3B are related to the graphics multiprocessor 234 of Figure 2C and represent graphics multiprocessors 325, 350 that may be used in place of one of them. Thus, any disclosure of features combined with the graphics multiprocessor 234 of the present application also discloses corresponding combinations with the graphics microprocessor 325, but is not limited thereto. Figure 3C represents a graphics processing unit (GPU) 380 that includes a dedicated set of graphics processing resources arranged in multi-core groups 365A-365N, and the GPU 380 corresponds to the graphics microprocessors 325, 350. The represented graphics multiprocessors 325, 350 and multi-core groups 365A-365N can be streaming multiprocessors (SMs) capable of simultaneous execution of a large number of execution threads.

[0047] The graphics multiprocessor 325 of Figure 3A includes a plurality of additional instances of execution resource units with respect to the graphics multiprocessor 234 of Figure 2D. For example, the graphics multiprocessor 325 can include multiple instances of instruction units 332A-332B, register files 334A-334B, and texture units 344A-344B. The graphics multiprocessor 325 also includes multiple sets of graphics or compute execution units (e.g., GPGPU cores 336A-336B, tensor cores 337A-337B, ray tracing cores 338A-338B) and multiple sets of load / store units 340A-340B. The execution resource units include a common instruction cache 330, texture and / or data cache memory 342, and shared memory 346.

[0048] Various components can communicate via an interconnect fabric 327. The interconnect fabric 327 may include one or more crossbar switches to enable communication between various components of the graphics multiprocessor 325. The interconnect fabric 327 may be a separate high-speed network fabric layer in which the components of the graphics multiprocessor 325 are stacked. The components of the graphics multiprocessor 325 communicate with remote components via the interconnect fabric 327. For example, GPGPU cores 336A - 336B, 337A - 337B, and 338A - 338B can each communicate with the shared memory 346 via the interconnect fabric 327. The interconnect fabric 327 can mediate communication within the graphics multiprocessor 325 to ensure fair bandwidth allocation between components.

[0049] The graphics multiprocessor 350 of FIG. 3B includes multiple sets of execution resources 356A - 356D, and each set of execution resources includes multiple instruction units, register files, GPGPU cores, and load / store units as represented in FIGS. 2D and 3A. The execution resources 356A - 356D can operate in response to texture units 360 - 360D for texture operations while sharing an instruction cache 354 and a shared memory 353. For example, the execution resources 356A - 356D can share the instruction cache 354 and the shared memory 353 along with multiple instances of texture and / or data cache memories 358A - 358B. Various components can communicate via an interconnect fabric 352 similar to the interconnect fabric 327 of FIG. 3A.

[0050] Those skilled in the art will understand that the architectures shown in FIGS. 1, 2A-2D, and 3A-3B are illustrative and not limiting of the scope of the present embodiment. Thus, the technology described in the present application may be implemented without limitation, without departing from the scope of the embodiments described in the present application, in any suitably configured processing unit including one or more mobile application processors, one or more desktop or server central processing units including multi-core CPUs, one or more parallel processing units such as the parallel processing unit 202 of FIG. 2A, and one or more graphics processors or special-purpose processing units.

[0051] The parallel processors or GPGPUs described in the present application may be communicatively coupled to a host / processor core to accelerate graphics operations, machine learning operations, pattern analysis operations, and various general-purpose GPU (GPGPU) functions. The GPU may be communicatively coupled to the host processor / core via a bus or other interconnect (e.g., a high-speed interconnect such as PCIe or NVLink). In other embodiments, the GPU may be integrated in the same package or chip as the core and communicatively coupled to the core via an internal processor bus / interconnect (i.e., within the package or chip). Regardless of how the GPU is connected, the processor core may assign work to the GPU in the form of a sequence of commands / instructions included in the work description. The GPU then uses dedicated circuitry / logic to effectively process those commands / instructions.

[0052] FIG. 3C represents a graphics processing unit (GPU) 380 that includes a dedicated set of graphics processing resources assigned to multi-core groups 365A-365N. Although only the details of one multi-core group 365A are provided, it will be understood that the other multi-core groups 365A-365N may be provided with the same or similar sets of graphics processing resources. The details described with respect to multi-core groups 365A-365N may also apply to any of the graphics multiprocessors 234, 325, 350 described in the present application.

[0053] As represented, the multi-core group 365A may include a set of graphics cores 370, a set of tensor cores 371, and a set of ray tracing cores 372. The scheduler / dispatcher 368 schedules and dispatches graphics threads executed on the various cores 370, 371, and 372. The set of register files 369 stores operand values used by the cores 370, 371, and 372 when executing graphics threads. These may include, for example, integer registers that store integer values, floating-point registers that store floating-point values, vector registers that store packed data elements (integer and / or floating-point data elements), and tile registers that store tensor / matrix values. The tile registers may be implemented as a combined set of vector registers.

[0054] One or more combined level 1 (L1) cache and shared memory units 373 locally store graphics data such as texture data, vertex data, pixel data, ray data, bounding volume data, etc. within each multi-core group 365A. One or more texture units 374 may also be used to perform texturing operations such as texture mapping and sampling. The level 2 (L2) cache 375 shared by all or a subset of the multi-core groups 365A - 365N stores graphics data and / or instructions for multiple concurrent graphics threads. As represented, the L2 cache 375 may be shared across multiple multi-core groups 365A - 365N. One or more memory controllers 367 couple the GPU 380 to the memory 366. The memory 366 may be system memory (e.g., DRAM) and / or dedicated graphics memory (e.g., GDDR6 memory).

[0055] The input / output (I / O) circuit configuration 363 couples the GPU 380 to one or more I / O devices 362, such as a digital signal processor (DSP), a network controller, or a user input device. An on-chip interconnect may be used to couple the I / O device 362 to the GPU 380 and the memory 366. One or more input / output memory management units (IOMMUs) 364 of the I / O circuit configuration 363 couple the I / O device 362 directly to the system memory 366. Optionally, the IOMMU 364 manages multiple sets of page tables to map virtual addresses to physical addresses within the system memory 366. The I / O device 362, the CPU 361, and the GPU 380 may share the same virtual address space.

[0056] In one implementation of the IOMMU 364, the IOMMU 364 supports virtualization. In this case, it may manage a first set of page tables that map guest / graphics virtual addresses to guest / graphics physical addresses and a second set of page tables that map guest / graphics physical addresses to system / host physical addresses (e.g., within the system memory 366). The base address of each of the first and second sets of page tables may be stored in a control register and swapped out on a context switch (e.g., such that a new context is given access to the relevant set of page tables). Although not shown in FIG. 3C, each of the cores 370, 371, 372 and / or the multicore groups 365A - 365N may include a translation lookaside buffer (TLB) to cache the translation from guest virtual to guest physical, from guest physical to host physical, and from guest virtual to host physical.

[0057] The CPU 361, GPU 380, and I / O device 362 may be integrated in a single semiconductor chip and / or chip package. The represented memory 366 may be integrated on the same chip or may be coupled to the memory controller 367 via an off-chip interface. In one implementation, the memory 366 has GDDR6 memory that shares the same virtual address space as other physical system-level memories, but the basic principles described in this application are not limited to this specific implementation.

[0058] The tensor core 371 may include a plurality of execution units specifically designed to perform matrix operations, which are the basic computational operations used to execute deep learning operations. For example, simultaneous matrix multiplication operations may be used for neural network training and inference. The tensor core 371 may perform matrix processing using various operand precisions including single-precision floating point (e.g., 32 bits), half-precision floating point (e.g., 16 bits), integer words (16 bits), bytes (8 bits), and nibbles (4 bits). For example, neural network implementations extract the features of each rendered scene that potentially combine details from multiple frames to compose a high-quality final image.

[0059] In deep learning implementations, parallel matrix multiplication operations may be scheduled for execution on the tensor core 371. Neural network training, in particular, requires a significant number of matrix inner product operations. To process the inner product formulation of an N×N×N matrix multiplication, the tensor core 371 may include at least N inner product processing elements. Before the matrix multiplication begins, an entire matrix is loaded into a tile register, and at least one column of the second matrix is loaded in each cycle of N cycles. There are N inner products processed per cycle.

[0060] The matrix elements may be stored with different precisions according to a particular implementation, including 16-bit words, 8-bit bytes (e.g., INT8), and 4-bit nibbles (e.g., INT4). Different precision modes may be specified for the set of tensor cores 371 to ensure that they are used for workloads with different most effective precisions (e.g., inferring workloads that can allow quantization to bytes and nibbles, etc.).

[0061] The ray tracing core 372 may accelerate ray tracing operations for both real-time and non-real-time ray tracing implementations. In particular, the ray tracing core 372 may include a ray traversal / intersection circuit that performs ray traversal using a bounding volume hierarchy (BVH) and identifies intersections between rays and primitives contained within the BVH volume. The ray tracing core 372 may also include a circuit configuration (e.g., using a Z-buffer or similar arrangement) for performing depth testing and selection. In one implementation, the ray tracing core 372 performs traversal and intersection operations in response to the image noise removal techniques described in this application. At least some of those techniques may be performed by the tensor cores 371. For example, the tensor cores 371 may implement a deep learning neural network to perform noise removal on frames generated by the ray tracing core 372. However, the CPU 361, the graphics core 370, and / or the ray tracing core 372 may also implement all or part of the noise removal and / or deep learning algorithms.

[0062] Furthermore, as described above, a distributed approach to noise removal may be used, and the GPU 380 may be within a computing device coupled to other computing devices via a network or a high-speed interconnect. In this distributed approach, interconnected computing devices may share neural network learning / training data to improve the rate at which the system as a whole learns to perform noise removal for different types of image frames and / or different graphics applications.

[0063] The ray tracing core 372 may handle all BVH traversals and ray primitive intersections so as to prevent the graphics core 370 from being overloaded with thousands of instructions per ray. For example, each ray tracing core 372 may include a first set of dedicated circuits that perform bounding box tests (e.g., for traversal operations) and a second set of dedicated circuits that perform ray-triangle intersection tests (e.g., for the intersecting rays being traversed). Thus, for example, the multi-core group 365A can easily initiate a ray probe, and the ray tracing core 372 independently performs ray traversal and intersection and returns hit data (e.g., hit, no hit, multiple hit, etc.) to the thread context. While the ray tracing core 372 performs traversal and intersection operations, the other cores 370, 371 can freely perform other graphics and computing tasks.

[0064] Optionally, each ray tracing core 372 may include a traversal unit that performs BVH test operations and / or an intersection unit that performs ray-primitive intersection tests. The intersection unit generates a "hit", "no hit", or "multiple hit" response and supplies it to the appropriate thread. During traversal and intersection operations, the execution resources of other cores (e.g., the graphics core 370 and the tensor core 371) can freely perform other forms of graphics work.

[0065] In one optional embodiment described below, a hybrid rasterization / ray tracing approach is used and the work is distributed between the graphics core 370 and the ray tracing core 372.

[0066] The ray tracing core 372 (and / or other cores 370, 371) may include ray tracing instructions such as Microsoft's DirectX Ray Tracing (DXR) that includes the DispatchRays command, and hardware support for ray generation, closest hit, any hit, and miss shaders that allows for the assignment of a unique set of shaders and textures per object. Other ray tracing platforms that may be supported by the ray tracing core 372, the graphics core 370, and the tensor core 371 are Vulkan 1.1.85. However, it should be noted that the basic principles described herein are not limited to any particular ray tracing ISA.

[0067] In general, the various cores 372, 371, 370 may support a ray tracing instruction set that includes instructions / functions for one or more of ray generation, closest hit, ray-primitive intersection, hierarchical bounding box construction per primitive, miss, visit, and exceptions. More specifically, a preferred embodiment includes ray tracing instructions for performing one or more of the following functions.

[0068] Ray Generation - Ray generation instructions may be executed for each pixel, sample, or other user-defined work assignment.

[0069] Closest Hit - Closest hit instructions may be executed to find the closest intersection of a ray and a primitive in the scene.

[0070] Any Hit - The any hit instruction potentially identifies multiple intersections between a ray and a primitive within a scene to identify a new closest intersection.

[0071] Intersection - The intersection instruction performs a ray-primitive intersection test and outputs the result.

[0072] Per-primitive Bounding box Construction - This instruction forms a bounding box around a given primitive or group of primitives (e.g., when forming a new BVH or other acceleration data structure).

[0073] Miss - Indicates that the ray misses all geometry within the scene or a specified region of the scene.

[0074] Visit - Indicates the children volumes that the ray is about to traverse.

[0075] Exceptions - Includes various types of exception handlers (e.g., called for various error conditions).

[0076] [Technology where the GPU hosts the processor interconnect] FIG. 4A depicts an exemplary architecture in which a plurality of GPUs 410-413, such as the parallel processor 200 shown in FIG. 2A, are communicatively coupled to a plurality of multi-core processors 405-406 via high-speed links 440A-440D (e.g., buses, point-to-point interconnects, etc.). The high-speed links 440A-440D support communication throughput of 4 GB / s, 30 GB / s, 80 GB / s or more, depending on the implementation. Various interconnect protocols may be used, including but not limited to PCIe 4.0 or 5.0 and NVLink 2.0. However, the basic principles described herein are not limited to any particular communication protocol or throughput.

[0077] Two or more of GPUs 410-413 may be interconnected via high-speed links 442A-442B that may be implemented using the same or different protocols / links as those used for high-speed links 440A-440D. Similarly, two or more of multi-core processors 405-406 may be connected via high-speed link 443. High-speed link 443 may be a symmetric multi-processor (SMP) bus operating at 20 GB / s, 30 GB / s, 120 GB / s or more. Alternatively, all communication between the various system components shown in FIG. 4A may be realized using the same protocol / link (e.g., via a communication interconnect fabric). However, as stated, the basic principles described herein are not limited to any particular type of interconnect technology.

[0078] Each multi-core processor 405-406 may be communicatively coupled to processor memories 401-402 via memory interconnects 430A-430B respectively, and each GPU 410-413 may be communicatively coupled to GPU memories 420-423 via GPU memory interconnects 450A-450D respectively. The memory interconnects 430A-430B and 450A-450D may utilize the same or different memory access technologies. By way of example, and not limitation, the processor memories 401-402 and the GPU memories 420-423 may be volatile memories such as dynamic random access memory (DRAM) (including stacked DRAM), graphics DDR SDRAM (GDDR) (e.g., GDDR5, GDDR6), high bandwidth memory (HBM), etc., and / or may be non-volatile memories such as 3D XPoint / Optance or Nano-Ram. For example, some portions of the memory may be volatile memory and other portions may be non-volatile memory (e.g., using a two-level memory (2LM) hierarchy).

[0079] As described below, the various processors 405-406 and GPUs 410-413 may be physically coupled to specific memories 401-402, 420-423 respectively, but an integrated memory architecture may also be implemented, in which the same virtual system address space (also referred to as the "effective address" space) is distributed among all of the various physical memories. For example, the processor memories 401-402 may each have a 64GB system memory address space, and the GPU memories 420-423 may each have a 32GB system memory address space (resulting in a total of 254GB of addressable memory in this example).

[0080] Figure 4B depicts any further details regarding the interconnection between the multi-core processor 407 and the graphics acceleration module 446. The graphics acceleration module 446 may include one or more GPU chips integrated on a line card coupled to the processor 407 via the high-speed link 440. Alternatively, the graphics acceleration module 446 may be integrated on the same package or chip as the processor 407.

[0081] The depicted processor 407 includes a plurality of cores 460A - 460D, each core having a translation lookaside buffer 461A - 461D and one or more caches 462A - 462D. The cores may include various other components for executing instructions and processing data, but these components are not shown so as not to obscure the underlying principles described in the present application (e.g., instruction fetch unit, branch prediction unit, decoder, execution unit, reorder buffer, etc.). The caches 462A - 462D may have levels 1 (L1) and 2 (L2). Further, one or more shared caches 456 may be included in the cache hierarchy and shared by groups of cores 460A - 460D. For example, one embodiment of the processor 407 includes 24 cores, each core having its own L1 cache, 12 shared L2 caches, and 12 shared L3 caches. In this embodiment, one of the L2 and L3 caches is shared by two adjacent cores. The processor 407 and the graphics acceleration module 446 are connected to the system memory 441, which may include processor memories 401 - 402.

[0082] For the data and instructions stored in the various caches 462A - 462D, 456, and the system memory 441, coherence is maintained via inter - core communication on the coherence bus 464. For example, each cache may have cache coherence logic / circuitry associated with it and communicate on the coherence bus 464 in response to detecting a read or write to a particular cache line. In one implementation, a cache snooping protocol is implemented on the coherence bus 464 to snoop cache accesses. Cache snooping / coherence techniques are well understood by those skilled in the art and are not described in detail here so as not to obscure the basic principles described in this application.

[0083] A proxy circuit 425 may be provided that communicatively couples the graphics acceleration module 446 to the coherence bus 464 so that the graphics acceleration module 446 can participate in the cache coherence protocol as a peer of the core. In particular, the interface 435 provides a connection to the proxy circuit 425 via a high - speed link 440 (e.g., PCIe bus, NVLink, etc.), and the interface 437 connects the graphics acceleration module 446 to the high - speed link 440.

[0084] In one implementation, the accelerator integrated circuit 436 provides cache management, memory access, context management, and interrupt management services in place of the plurality of graphics processing engines 431, 432, N of the graphics acceleration module 446. The graphics processing engines 431, 432, N may each have a separate graphics processing unit (GPU). Alternatively, the graphics processing engines 431, 432, N may have different types of graphics processing engines within a GPU such as graphics execution units, media processing engines (e.g., video encoders / decoders), samplers, and bit engines. In other words, the graphics acceleration module may be a GPU with a plurality of graphics processing engines 431-432, N, or the graphics processing engines 431-432, N may be individual GPUs integrated on a common package, line card, or chip.

[0085] Accelerator integrated circuit 436 may include a memory management unit (MMU) 439 that executes various memory management functions such as virtual - physical memory conversion (also called effective - physical memory conversion), and a memory access protocol for accessing system memory 441. MMU 439 may also include a translation lookaside buffer (TLB) (not shown) for caching virtual / effective - physical / substantive address translations. In one implementation, cache 438 stores commands and data for valid accesses by graphics processing engines 431, 432, N. Data stored in cache 438 and graphics memories 433 - 434, M can be kept coherent with core caches 462A - 462D, 456 and system memory 441. As described, this can be achieved via proxy circuit 425 that participates in the cache coherence mechanism in place of cache 438 and memories 433 - 434, M (e.g., sending updates to cache 438 related to changes / accesses of cache lines on processor caches 462A - 462D, 456 and receiving updates from cache 438).

[0086] A set of registers 445 stores context data of threads executed by graphics processing engines 431 - 432, N, and context management circuit 448 manages the thread context. For example, context management circuit 448 may perform save and restore operations to save and restore the context of various threads during a context switch (e.g., the first thread is saved and the second thread is stored, whereby the second thread is executable by the graphics processing engine). For example, during a context switch, context management circuit 448 may store the current register values in a specified area in memory (e.g., identified by a context pointer). It may then restore the register values when returning to the context. Interrupt management circuit 447 may receive and process interrupts received from system devices, for example.

[0087] In one implementation, the virtual / effective address from the graphics processing engine 431 is translated by the MMU 439 into a physical / substantive address within the system memory 441. Optionally, the accelerator integrated circuit 436 supports a plurality (e.g., 4, 8, 16) of graphics accelerator modules 446 and / or other accelerator devices. The graphics accelerator module 446 may be dedicated to a single application executed by the processor 407 or may be shared among multiple applications. Optionally, a virtualized graphics execution environment is provided, in which the resources of the graphics processing engines 431-432, N are shared among multiple applications or virtual machines (VMs). The resources may be subdivided into "slices" that are assigned to different VMs and / or applications based on the processing requirements and priorities associated with the VMs and / or applications.

[0088] Thus, the accelerator integrated circuit 436 operates as a system bridge for the graphics acceleration module 446 and provides address translation and system memory cache services. In one embodiment, to facilitate the bridging functionality, the accelerator integrated circuit 436 may also include shared I / O 497 (e.g., PCIe, USB) and hardware that enables system control of voltage, clocking, performance, heat, and security. Additionally, the accelerator integrated circuit 436 may provide a virtualization facility for the host processor to manage virtualization, interrupts, and memory management of the graphics processing engine.

[0089] The hardware resources of the graphics processing engines 431-432, N are explicitly mapped to the virtual address space seen by the host processor 407, so any host processor can address those resources directly using virtual address values. Any function of the accelerator integrated circuit 436 is the physical separation of the graphics engines 431-432, N, whereby they appear to the system as independent units.

[0090] One or more graphics memories 433-434, M may each be coupled to a respective one of the graphics processing engines 431-432, N. The graphics memories 433-434, M store instructions and data processed by respective ones of the graphics processing engines 431-432, N. The graphics memories 433-434, M may be volatile memories such as DRAM (including stacked DRAM), GDDR memory (e.g., GDDR5, GDDR6), or HBM, and / or may be non-volatile memories such as 3D XPoint / Optance or Nano-Ram.

[0091] To reduce data traffic on the high-speed link 440, biasing techniques are used to ensure that the data stored in the graphics memories 433-434, M is most frequently used by the graphics processing engines 431-432, N and desirably not (or at least not as frequently) used by the cores 460A-460D. Similarly, the biasing mechanism attempts to hold and use in the core caches 462A-462D, 456 and the system memory 441 the data required by the cores (desirably not required by the graphics processing engines 431-432, N).

[0092] According to the variant shown in FIG. 4C, the accelerator integrated circuit 436 is integrated within the processor 407. The graphics processing engines 431-432, N communicate directly on the high-speed link 440 with the accelerator integrated circuit 436 via the interface 437 and the interface 435 (again, any form of bus or interface protocol may be used). The accelerator integrated circuit 436 can perform the same operations as described with respect to FIG. 4B, but potentially with higher throughput considering its close proximity to the coherence bus 464 and the caches 462A-462D, 456.

[0093] The described embodiments may support different programming models including a dedicated process programming model (non-graphics acceleration module virtualization) and a shared programming model (by virtualization). The latter may include a programming model controlled by the accelerator integrated circuit 436 and a programming model controlled by the graphics acceleration module 446.

[0094] In embodiments of the dedicated process model, the graphics processing engines 431-432, N may be dedicated to a single application or process under a single operating system. The single application can direct other application requests to the graphics engines 431-432, N to provide virtualization within the VM / partition.

[0095] In the dedicated process programming model, the graphics processing engines 431-432, N may be shared by multiple VM / application partitions. The shared model requires the system hypervisor to virtualize the graphics processing engines 431-432, N to enable access by each operating system. For a single partition system without a hypervisor, the graphics processing engines 431-432, N are owned by the operating system. In either case, the operating system can virtualize the graphics processing engines 431-432, N to provide access to each process or application.

[0096] For the shared programming model, the graphics acceleration module 446 or individual graphics processing engines 431-432, N select process elements using a process handle. The process elements may be stored in the system memory 441 and may be addressable using the virtual-to-physical address translation techniques described in this application. The process handle may be an implementation-specific value supplied to the host process when registering its context with the graphics processing engines 431-432, N (i.e., calling system software to add the process element to the list linked to that process element). The lower 16 bits of the process handle may be the offset of the process element within the list linked to the process element.

[0097] Figure 4D represents an example accelerator integrated slice 490. As used in the present application, a "slice" has a designated portion of the processing resources of the accelerator integrated circuit 436. The application effective address space 482 in the system memory 441 stores the process element 483. The process element 483 may be stored in response to a GPU launch 481 from the application 480 being executed by the processor 407. The process element 483 includes the process state of the corresponding application 480. The work descriptor (WD) 484 included in the process element 483 can be a single job requested by the application, or may include a pointer to a queue of jobs. In the latter case, the WD 484 is a pointer to the job request queue within the application's address space 482.

[0098] The graphics acceleration module 446 and / or the individual graphics processing engines 431 - 432, N can be shared by all or a subset of the processes within the system. For example, the techniques described in the present application may include an infrastructure that sets up the process state, sends the WD 484 to the graphics acceleration module 446, and starts a job in a virtualized environment.

[0099] In one implementation, the dedicated process programming model is implementation - specific. In this model, a single process owns the graphics acceleration module 446 or the individual graphics processing engine 431. Since the graphics acceleration module 446 is owned by a single process, the hypervisor initializes the accelerator integrated circuit 436 for the owning partition, and the operating system initializes the accelerator integrated circuit 436 for the owning partition when the graphics acceleration module 446 is assigned.

[0100] During operation, the WD fetch unit 491 in the accelerator integrated circuit 436 fetches the next WD 484 that contains instructions for work to be performed by one of the graphics processing engines of the graphics acceleration module 446. The data from the WD 484 is stored in the register 445 and may be used by the MMU 439, the interrupt management circuit 447, and / or the context management circuit 448 as represented. For example, the MMU 439 may include a segment / page walk circuit for accessing the segment / page table 486 in the OS virtual address space 485. The interrupt management circuit 447 may process the interrupt event 492 received from the graphics acceleration module 446. When executing a graphics operation, the effective address 493 generated by the graphics processing engines 431 - 432, N is translated to a physical address by the MMU 439.

[0101] The same set of registers 445 may be replicated for each of the graphics processing engines 431 - 432, N and / or the graphics acceleration module 446 and may be initialized by the hypervisor or the operating system. Each of those replicated registers may be included in the accelerator integrated slice 490. Exemplary registers that may be initialized by the hypervisor are shown in Table 1.

Table 1

[0102] Exemplary registers that may be initialized by the operating system are shown in Table 2.

Table 2

[0103] Each WD484 may be specific to a particular graphics acceleration module 446 and / or graphics processing engines 431-432, N. It may contain all the information that the graphics processing engines 431-432, N need to perform their operations, or it can be a pointer to a memory location that sets up a command queue for the work to be done by the application.

[0104] Figure 4E shows further optional details of the shared model. It includes the hypervisor physical address space 498 in which the process element list 499 is stored. The hypervisor physical address space 498 is accessible via a hypervisor 496 that virtualizes the graphics acceleration module engine for the operating system 495.

[0105] The shared programming model enables all or a subset of processes from all or a subset of partitions within the system to use the graphics acceleration module 446. There are two programming models in which the graphics acceleration module 446 is shared by multiple processes and partitions, namely, time slice sharing and graphics-oriented sharing.

[0106] In this model, the system hypervisor 496 owns the graphics acceleration module 446 and makes its functions available to all operating systems 495. To support virtualization by the system hypervisor 496, the graphics acceleration module 446 may comply with the following requirements: 1) The job requests of the application must be autonomous (i.e., the state does not need to be maintained between jobs), or the graphics acceleration module 446 must provide a context save and restore mechanism. 2) The job requests of the application are guaranteed to be completed within a specified amount of time, including any conversion failures, by the graphics acceleration module 446, or the graphics acceleration module 446 must provide the ability to preempt the processing of the job. 3) The graphics acceleration module 446 must ensure fairness between processes when operating in a directed shared programming model.

[0107] For the shared model, the application 480 may be required to make system calls to the operating system 495 using the type of the graphics acceleration module 446, the work descriptor (WD), the authority mask register (AMR) value, and the context save / restore area pointer (CSRP). The type of the graphics acceleration module 446 describes the acceleration function to be the target of the system call. The type of the graphics acceleration module 446 may be a system-specific value. The WD is specifically formatted for the graphics acceleration module 446 and can take the form of a command of the graphics acceleration module 446, a valid address pointer to a user-defined structure, a valid address pointer to a command queue, or any other data structure that describes the work to be done by the graphics acceleration module 446. In one embodiment, the AMR value is the AMR state to be used for the current process. The value passed to the operating system is the same as that of the application that sets the AMR. If the implementation of the accelerator integrated circuit 436 and the graphics acceleration module 446 does not support the User Authority Mask Override Register (UAMOR), the operating system may apply the current UAMOR value to the AMR value before passing the AMR in the hypervisor call. The hypervisor 496 may optionally apply the current AMOR (Authority Mask Override Register) value before placing the AMR in the process element 483. The CSRP may be one of the registers 445 that contains the valid address of an area within the application's address space 482 for the graphics acceleration module 446 to save and restore the context state.This pointer is optional if the state does not need to be saved between jobs or if the job is preempted. The context save / restore area may be pinned system memory.

[0108] Upon receiving a system call, the operating system 495 may verify that the application 480 is registered and has been granted the right to use the graphics acceleration module 446. The operating system 495 then calls the hypervisor 496 using the information shown in Table 3.

Table 3

[0109] Upon receiving a hypervisor call, the hypervisor 496 verifies that the operating system 495 is registered and has been granted the right to use the graphics acceleration module 446. The hypervisor 496 then places the process element 483 in a list linked to the process elements for the type of the corresponding graphics acceleration module 446. The process element may contain the information shown in Table 4.

Table 4

[0110] The hypervisor may initialize the slice 445 of a plurality of accelerator integration slices 490.

[0111] As shown in FIG. 4F, in any one implementation, an integrated memory addressable via a common virtual memory address space used to access physical processor memories 401-402 and GPU memories 420-423 is used. In this implementation, operations executed on GPUs 410-413 utilize the same virtual / effective memory address space to access processor memories 401-402, and vice versa, thereby simplifying programmability. A first portion of the virtual / effective address space may be allocated to processor memory 401, a second portion may be allocated to second processor memory 402, a third portion may be allocated to GPU memory 420, and so on. The overall virtual / effective memory space (sometimes referred to as the effective address space) is thereby distributed across respective ones of processor memories 401-402 and GPU memories 420-423, enabling any processor or GPU to access any physical memory using the virtual address mapped to that memory.

[0112] Bias / coherence management circuits 494A-494E are provided in one or more of MMUs 439A-439E, which ensure cache coherence between the cache of a host processor (e.g., 405) and GPUs 410-413 and implement a biasing technique indicating the physical memory in which a particular type of data should be stored. While multiple instances of bias / coherence management circuits 494A-494E are shown in FIG. 4F, the bias / coherence circuit may be implemented within the MMU of one or more host processors 405 and / or within accelerator integration circuit 436.

[0113] The GPU-attached memories 420-423 are mapped as part of the system memory and can be accessed using shared virtual memory (SVM) technology without suffering from typical performance drawbacks associated with full system cache coherence. The ability of the GPU-attached memories 420-423 to be accessed as system memory without cumbersome cache coherence overhead provides an advantageous operating environment for GPU offloading. This arrangement allows the software of the host processor 405 to set up operands and access computation results without the overhead of conventional I / O DMA data copies. Such conventional copies require driver calls, interrupts, and memory mapped I / O (MMIO) accesses, all of which are inefficient for simple memory accesses. At the same time, the ability to access the GPU-attached memories 420-423 without cache coherence overhead can be important for the execution time of offloaded computations. When there is a significant amount of streaming write memory traffic, for example, the cache coherence overhead can significantly reduce the effective write bandwidth seen by the GPUs 410-413. The efficiency of operand setup, result access, and GPU computation all play a role in determining the effectiveness of GPU offloading.

[0114] The selection between GPU bias and host processor bias may be driven by a bias tracker data structure. For example, a bias table may be used. The bias table may be a page-granularity structure that includes one or two bits per GPU-attached memory page (i.e., controlled at the granularity of the memory page). The bias table may be implemented in the stolen memory ranges of one or more of the GPU-attached memories 420-423, regardless of the presence or absence of a bias cache within the GPUs 410-413 (e.g., for caching frequently / most recently used entries of the bias table). Alternatively, an entry bias table may be maintained within the GPU.

[0115] In one implementation, the bias table entries associated with each access to the GPU-attached memories 420-423 are accessed before the actual access to the GPU memory and cause the following operations. First, local requests from the GPUs 410-413 to find their pages in the GPU bias are transferred directly to the corresponding GPU memories 420-423. Local requests from the GPUs to find their pages in the host bias are transferred to the processor 405 (e.g., over the high-speed link described above). Optionally, requests from the processor 405 to find the requested page in the host processor bias complete the request as a normal memory read. Alternatively, requests directed to the GPU bias pages may be transferred to the GPUs 410-413. The GPU may then move the page to the host processor if the host processor is not currently using the page.

[0116] The bias state of a page can be changed by a software-based mechanism or a hardware-assisted software-based mechanism, or, in limited cases, simply by a hardware-based mechanism.

[0117] One mechanism for changing the bias state uses an API call (e.g., OpenCL). This then calls the GPU device driver, which then sends a message to the GPU instructing it to change the bias state (or enqueues a command descriptor) and, for some transitions, performs a cache flush operation on the host. The cache flush operation is required for the transition from the bias of the host processor 405 to the bias of the GPU, but not for the reverse transition.

[0118] Cache coherence can be maintained by temporarily making the GPU bias pages non-cacheable by the host processor 405. To access those pages, the processor 405 may request access from the GPU 410, which may or may not immediately grant access, depending on the implementation. Thus, to reduce communication between the host processor 405 and the GPU 410, it is beneficial to ensure that the GPU bias pages are pages that are needed by the GPU but not by the host processor 405, and vice versa.

[0119] [Graphics Processing Pipeline] FIG. 5 represents a graphics processing pipeline 500. Graphics multiprocessors such as the graphics multiprocessor 234 as seen in FIG. 2D, the graphics multiprocessor 325 as seen in FIG. 3A, the graphics multiprocessor 350 as seen in FIG. 3B, etc. can implement the represented graphics processing pipeline 500. The graphics multiprocessor may be related to the parallel processor 112 of FIG. 1 and may be included within the parallel processing subsystem described herein, such as the parallel processor 200 of FIG. 2A which may be used in place of one of them. Various parallel processing systems can implement the graphics processing pipeline 500 via one or more instances of parallel processing units (e.g., the parallel processing unit 202 of FIG. 2A) as described herein. For example, a shader unit (e.g., the graphics multiprocessor 234 of FIG. 2C) may be configured to execute one or more functions of the vertex processing unit 504, the tessellation control processing unit 508, the tessellation evaluation processing unit 512, the geometry processing unit 516, and the fragment / pixel processing unit 524. The functions of the data assembler 502, the primitive assemblers 506, 514, 518, the tessellation unit 510, the rasterizer 522, and the raster operation unit 526 may also be performed by other processing engines within a processing cluster (e.g., the processing cluster 214 of FIG. 2A) and corresponding partitioning units (e.g., the partitioning units 220A-220N of FIG. 2A). The graphics processing pipeline 500 may also be implemented using dedicated processing units for one or more functions. It is also possible for one or more portions of the graphics processing pipeline 500 to be executed by parallel processing logic within a general-purpose processor (e.g., a CPU). Optionally, one or more portions of the graphics processing pipeline 500 can access on-chip memory (e.g., the parallel processor memory as seen in FIG. 2A) via a memory interface 528 which may be an instance of the memory interface 218 of FIG. 2A.The graphics processor pipeline 500 may also be implemented via a multi-core group 365A as seen in FIG. 3C.

[0120] The data assembler 502 is a processing unit that can collect vertex data and primitives of a surface. The data assembler 502 then outputs vertex data including vertex attributes to the vertex processing unit 504. The vertex processing unit 504 is a programmable execution unit that executes a vertex shader program and writes and transforms the vertex data specified by the vertex shader program. The vertex processing unit 504 may be programmed to read data stored in a cache, local or system memory for use in processing the vertex data and transform the vertex data from an object-based coordinate representation to a world space coordinate space or a normalized device coordinate space.

[0121] A first instance of the primitive assembler 506 receives vertex attributes from the vertex processing unit 504. The primitive assembler 506 reads out stored vertex attributes as needed and constructs graphics primitives for processing by the tessellation control processing unit 508. Graphics primitives include triangles, line segments, points, patches, etc. supported by various graphics processing application programming interfaces (APIs).

[0122] The tessellation control processing unit 508 treats the input vertices as control points for geometric patches. The control points are converted from the input representation from the patch (e.g., patch-based) to a representation suitable for proper evaluation by the tessellation evaluation processing unit 512. The tessellation control processing unit 508 can also calculate the tessellation factors for the edges of the geometric patches. The tessellation factor is applied to a single edge and quantifies the view-dependent level of detail associated with the edge. The tessellation unit 510 is configured to mosaic the patch into a plurality of geometric primitives such as lines, triangles, or quadrilateral primitives. The plurality of geometric primitives are sent to the tessellation evaluation processing unit 512. The tessellation evaluation processing unit 512 acts on the parameterized coordinates of the subdivided patch to generate the surface representation and vertex attributes of each vertex associated with the geometric primitive.

[0123] A second instance of the primitive assembler 514 receives vertex attributes from the tessellation evaluation processing unit 512 that reads the stored vertex attributes as needed and constructs graphics primitives for processing by the geometry processing unit 516. The geometry processing unit 516 is a programmable execution unit that executes a geometry shader program to transform the graphics primitives received from the primitive assembler 514 as specified by the geometry shader program. The geometry processing unit 516 may be programmed to subdivide the graphics primitive into one or more new graphics primitives and calculate the parameters used to rasterize the new graphics primitives.

[0124] The geometry processing unit 516 may be capable of adding or removing elements within a geometry stream. The geometry processing unit 516 outputs parameters that identify a new graphics primitive and vertices to the primitive assembler 518. The primitive assembler 518 receives the parameters and vertices from the geometry processing unit 516 and constructs a graphics primitive for processing by the viewport scale, cull, and clip unit 520. The geometry processing unit 516 reads data stored in the parallel processor memory or the system memory for use in processing geometry data. The viewport scale, cull, and clip unit 520 performs clipping, culling, and viewport scaling and outputs the processed graphics primitive to the rasterizer 522.

[0125] The rasterizer 522 can perform depth culling and other depth-based optimizations. The rasterizer 522 also performs scan conversion on new graphics primitives so as to generate fragments and output these fragments and the associated coverage data to the fragment / pixel processing unit 524. The fragment / pixel processing unit 524 is a programmable execution unit configured to execute a fragment shader program or a pixel shader program. The fragment / pixel processing unit 524 transforms the fragments or pixels received from the rasterizer 522 as specified by the fragment or pixel shader program. For example, the fragment / pixel processing unit 524 may be programmed to perform operations including, but not limited to, texture mapping, shading, blending, texture correction, and perspective correction to generate shaded fragments or pixels that are output to the raster operation unit 526. The fragment / pixel processing unit 524 can read data stored either in the parallel processor memory or the system memory for use when processing fragment data. The fragment or pixel shader program may be configured to shade at a sample, pixel, tile, or other granularity corresponding to the sampling rate set for the processing unit.

[0126] The raster operation unit 526 performs raster operations including, but not limited to, stencil, z-test, blending, etc., and outputs the pixel data as processed graphics data to be stored in the graphics memory (e.g., the parallel processor memory 222 of FIG. 2A and / or the system memory 104 of FIG. 1), to be displayed on one or more display devices 110, or for further processing by one of the one or more processors 102 or parallel processors 112. The raster operation unit 526 may be configured to compress z or color data written to the memory and decompress z or color data read from the memory.

[0127] [Overview of Machine Learning] The above architecture can be applied to perform training and inference operations using a machine learning model. Machine learning has been successful in solving many types of tasks. The computations that occur when training and using machine learning algorithms (e.g., neural networks) are inherently useful for efficient parallel implementations. Thus, parallel processors such as general-purpose graphics processing units (GPGPUs) play an important role in the actual implementation of deep neural networks. A parallel graphics processor with a SIMT (single instruction, multiple thread) architecture is designed to maximize the amount of parallel processing in the graphics pipeline. In the SIMT architecture, a group of parallel threads attempts to execute program instructions synchronously together as frequently as possible to increase processing efficiency. The efficiency brought about by parallel machine learning algorithm implementations enables the use of high-capacity networks and allows those networks to be trained on larger datasets.

[0128] Machine learning algorithms can learn based on a set of data. For example, a machine learning algorithm can be designed to model high-level abstractions within a dataset. For example, an image recognition algorithm can be used to determine to which of several categories a given input belongs, a regression algorithm can output a numerical value when an input is given, and a pattern recognition algorithm can be used to generate transformed text or to perform speech recognition from and / or for text.

[0129] An example type of machine learning algorithm is a neural network. There are many types of neural networks, and a simple type of neural network is a feedforward network. A feedforward network may be implemented as an acyclic graph in which nodes are arranged in layers. Typically, a feedforward network topology includes an input layer and an output layer separated by at least one hidden layer. The hidden layer transforms the input received by the input layer into a representation that is useful for generating an output at the output layer. Network nodes are fully connected via edges to nodes in adjacent layers, but there are no edges between nodes within each layer. Data received at the nodes of the input layer of a feedforward network is propagated (i.e., sent forward) to the nodes of the output layer by an activation function that calculates the state of the nodes of each successive layer within the network based on coefficients ("weights") each associated with one of the edges connecting the layers. Depending on the specific model represented by the algorithm being executed, the output from a neural network algorithm can take various forms.

[0130] Before a machine learning algorithm can be used to model a particular problem, the algorithm is trained using a training data set. Training a neural network requires selecting a network topology, using a set of training data that represents the problem to be modeled by the network, and adjusting the weights until the network model performs with minimal error for all instances of the training data set. For example, during a supervised learning training process for a neural network, the output generated by the network in response to an input representing an instance within the training data set is compared to the "correct" labeled output for that instance, an error signal representing the difference between the output and the labeled output is calculated, and the weights associated with the connections are adjusted to minimize that error as the error signal is backpropagated through the network layers. The network is considered "trained" when the error for each of the outputs generated from the instances of the training data set is minimized.

[0131] The accuracy of a machine learning algorithm can be greatly affected by the quality of the data set used to train the algorithm. The training process is computationally intensive and may require a significant amount of time on conventional general-purpose processors. Accordingly, parallel processing hardware is used to train many types of machine learning algorithms. This is particularly useful for optimizing the training of neural networks in that the calculations performed when adjusting the coefficients in a neural network are inherently amenable to parallel implementation. Specifically, many machine learning algorithms and software applications are adapted to use the parallel processing hardware within general-purpose graphics processing devices.

[0132] FIG. 6 is a generalized view of a machine learning software stack 600. The machine learning application 602 can be configured to train a neural network using a training data set or to implement machine learning intelligence using a trained deep neural network. The machine learning application 602 can include training and inference functions for the neural network and / or specialized software that can be used to train the neural network prior to deployment. The machine learning application 602 can implement any type of machine intelligence including, but not limited to, image recognition, mapping and localization, autonomous navigation, speech synthesis, medical imaging, or language translation.

[0133] Hardware acceleration for the machine learning application 602 can be enabled via a machine learning framework 604. The machine learning framework 604 can provide a library of machine learning primitives. Machine learning primitives are the basic operations generally executed by machine learning algorithms. Without the machine learning framework 604, developers of machine learning algorithms would be required to create and optimize the main computational logic associated with the machine learning algorithm and then re-optimize the computational logic if a new parallel processor is developed. Instead, the machine learning application can be configured to execute the necessary computations using the primitives provided by the machine learning framework 604. Example primitives include tensor convolution, activation functions, and pooling, which are computational operations performed while training a convolutional neural network (CNN). The machine learning framework 604 can also supply primitives to implement basic linear algebra subprograms executed by many machine learning algorithms, such as matrix and vector operations.

[0134] The machine learning framework 604 can process the input data received from the machine learning application 602 and generate appropriate input to the computing framework 606. The computing framework 606 can abstract the underlying instructions supplied to the GPGPU driver 608 so as to enable the machine learning framework 604 to utilize hardware acceleration via the GPGPU hardware 610 without requiring the machine learning framework 604 to have in-depth knowledge of the architecture of the GPGPU hardware 610. Furthermore, the computing framework 606 can enable hardware acceleration for the machine learning framework 604 across various types and generations of the GPGPU hardware 610.

[0135] [GPGPU Machine Learning Acceleration] FIG. 7 depicts a general-purpose graphics processing unit 700 that may be the parallel processor 200 of FIG. 2A or the parallel processor 112 of FIG. 1. The general-purpose processing unit (GPGPU) 700 may be configured to be particularly efficient in processing types of computational workloads associated with the training of deep neural networks. Furthermore, the GPGPU 700 may be directly linked to other instances of GPGPU to form a multi-GPU cluster, particularly to improve the training speed for deep neural networks.

[0136] The GPGPU 700 includes a host interface 702 that enables connection to a host processor. The host interface 702 may be a PCI Express interface. However, the host interface can also be a vendor - specific communication interface or a communication fabric. The GPGPU 700 receives commands from the host processor and uses a global scheduler 704 to distribute the execution threads associated with those commands to a set of processing clusters 706A - 706H. The processing clusters 706A - 706H share a cache memory 708. The cache memory 708 can act as a higher - level cache for the cache memories within the compute clusters 706A - 706H. The illustrated processing clusters 706A - 706H may correspond to the processing clusters 214A - 214N as seen in Figure 2A.

[0137] The GPGPU 700 includes memories 714A - 714B coupled to the processing clusters 706A - 706H via a set of memory controllers 712A - 712B. The memories 714A - B can include various types of memory devices including dynamic random - access memory (DRAM) such as synchronous graphics random - access memory (SGRAM) including graphics double data rate (GDDR) memory or graphics random - access memory. The memories 714A - 714B can also include 3D stacked memories including but not limited to high - bandwidth memory (HBM).

[0138] Each of processing clusters 706A - 706H may include a set of graphics multiprocessors such as the graphics multiprocessor 234 of FIG. 2D, the graphics multiprocessor 325 of FIG. 3A, the graphics multiprocessor 350 of FIG. 3B, or may include multi - core groups 365A - 365N as seen in FIG. 3C. The graphics multiprocessors of the compute cluster include multiple types of integer and floating - point logic units capable of performing compute operations within a range of precision including those suitable for machine - learning computations. For example, at least a subset of the floating - point units in each of processing clusters 706A - 706H may be configured to perform 16 - bit or 32 - bit floating - point operations, while another subset of the floating - point units may be configured to perform 64 - bit floating - point operations.

[0139] Multiple instances of GPGPU700 may be configured to operate as a compute cluster. The communication mechanisms used by the compute cluster for synchronization and data exchange vary for each embodiment. For example, multiple instances of GPGPU700 communicate via host interface 702. In one embodiment, GPGPU700 includes an I / O hub 709 that couples GPGPU700 with a GPU link 710 that enables direct connection to other instances of the GPGPU. The GPU link 710 may be coupled to a dedicated inter - GPU bridge that enables communication and synchronization between multiple instances of GPGPU700. Optionally, the GPU link 710 is coupled to a high - speed interconnect to send and receive data to / from other GPGPUs or parallel processors. Multiple instances of GPGPU700 may be located in separate data - processing systems and communicate via a network device accessible by host interface 702. The GPU link 710 may be configured to enable connection to a host processor in addition to or instead of the host interface 702.

[0140] While the depicted configuration of the GPGPU 700 can be configured to train a neural network, an alternative configuration of the GPGPU 700 can be configured for deployment within a high-performance or low-power inference platform. In the inference configuration, the GPGPU 700 has fewer processing clusters 706A-706H included therein compared to the training configuration. Furthermore, the memory technology associated with memories 714A-714B may be different between the inference configuration and the training configuration. In one embodiment, the inference configuration of the GPGPU 700 can support inferring specific instructions. For example, the inference configuration can support one or more 8-bit integer inner product instructions commonly used during inference operations for a deployed neural network.

[0141] FIG. 8 depicts a multi-GPU computing system 800. The multi-GPU computing system 800 can include a processor 802 coupled to a plurality of GPGPUs 806A-806B via a host interface switch 804. The host interface switch 804 can be a PCI Express switch device that couples the processor 802 to a PCI Express bus. Via the PCI Express bus, the processor 802 can communicate with a set of GPGPUs 806A-806D. Each of the plurality of GPGPUs 806A-806D can be an instance of the GPGPU 700 of FIG. 7. The GPGPUs 806A-806D can be interconnected via a set of high-speed point-to-point GPU-to-GPU links 816. The high-speed GPU-to-GPU links can be connected to each of the GPGPUs 806A-806D via a dedicated GPU link such as the GPU link 710 of FIG. 7. The P2P GPU link 816 enables direct communication between each of the GPGPUs 806A-806D without requiring communication on the host interface bus to which the processor 802 is connected. According to the GPU-to-GPU traffic directed to the P2P GPU link, the host interface bus remains available for, e.g., system memory access or communication with other instances of the multi-GPU computing system 800 via one or more network devices. In FIG. 8, the GPGPUs 806A-806D are connected to the processor 802 via the host interface switch 804, while the processor 802 can alternatively include direct support for the P2P GPU link 816 and be directly connected to the GPGPUs 806-806D.

[0142] [Machine Learning Neural Network Implementation] The computing architecture described in the present application can be configured to perform a type of parallel processing that is particularly suitable for training and deploying neural networks for machine learning. A neural network can be generalized as a network of functions having a graph relationship. As is well known in the art, there are various types of neural network implementations used in machine learning. One exemplary type of neural network is, as described above, a feedforward network.

[0143] A second exemplary type of neural network is a convolutional neural network (CNN). A CNN is a specialized feedforward neural network for processing data having a known grid-like topology, such as image data. Thus, CNNs are widely used for computer vision and image recognition applications, although they may also be used for other types of pattern recognition, such as speech and language processing. The nodes in the CNN input layer are organized into sets of "filters" (feature detectors evoked by receptive fields in the retina), and the output of each set of filters is propagated to nodes in successive layers of the network. The computation of a CNN involves applying a convolutional mathematical operation to each filter to generate the output of that filter. Convolution is a specialized type of mathematical operation performed by two functions, which produces a third function that is a modified version of one of the two original functions. In the terminology of convolutional networks, the first function of convolution is called the input, while the second function may be called the convolution kernel. The output may be called a feature map. For example, the input to a convolutional layer can be a multi-dimensional array of data defining the various color components of an input image. The convolution kernel can be a multi-dimensional array of parameters, which are adapted by a training process for the neural network.

[0144] A recurrent neural network (RNN) is a type of feedforward neural network that includes feedback connections between layers. By sharing parameter data across different parts of the neural network, RNNs enable the modeling of sequential data. The architecture of an RNN contains cycles. A cycle means that at least a part of the output data from the RNN is used as feedback for processing subsequent inputs in order, representing the influence that the current value of a variable has on its own value at a future point in time. This feature makes RNNs particularly useful for language processing due to the variable nature in which language data can be structured.

[0145] The figures described later present exemplary feedforward, CNN, and RNN networks and describe the general processes for training and deploying each of these types of networks. These descriptions are examples and are non-limiting with respect to any specific embodiments described in this application, and the concepts presented can generally be applied to deep neural networks and machine learning techniques in general.

[0146] The above exemplary neural networks can be used to perform deep learning. Deep learning is machine learning that uses deep neural networks. The deep neural networks used in deep learning are artificial neural networks consisting of multiple hidden layers, in contrast to shallow neural networks that contain only a single hidden layer. Generally, deeper neural networks require more computational effort to train. However, the additional hidden layers of the network enable multi-step pattern recognition in which the output error is reduced compared to shallow machine learning techniques.

[0147] The deep neural network used in deep learning usually includes a front-end network for performing feature recognition coupled to a back-end network that represents a mathematical model capable of performing operations (e.g., object classification, speech recognition, etc.) based on the feature representation given to the model. Deep learning enables machine learning to be performed without the need for hand-crafted feature engineering to be performed on the model. Instead, the deep neural network can learn features based on the statistical structure or correlations within the input data. The learned features can be supplied to a mathematical model that can map the detected features to the output. The mathematical model used by the network is generally specialized for a particular task to be performed, and different models will be used to perform different tasks.

[0148] Once the neural network is structured, a learning model can be applied to the network to train the network to perform a specific task. The learning model describes how the weights within the model should be adjusted to reduce the output error of the network. Backpropagation of error is a common model used to train neural networks. An input vector is given to the network for processing. The output of the network is compared to the desired output using a loss function, and an error value is calculated for each neuron in the output layer. The error value is then propagated backward until each neuron has an associated error value that roughly represents its contribution to the original output. The network can then learn from those errors using an algorithm such as the stochastic gradient descent algorithm to update the weights of the neural network.

[0149] Figures 9A-9B represent an exemplary convolutional neural network. Figure 9A represents various layers within the CNN. As shown in Figure 9A, an exemplary CNN used to model image processing can receive an input 902 that describes the red, green, and blue (RGB) components of an input image. The input 902 can be processed by a plurality of convolutional layers (e.g., convolutional layer 904, convolutional layer 906). The output from the plurality of convolutional layers may optionally be processed by a set of fully connected layers 908. Neurons within the fully connected layers have a complete connection to all activations in the previous layer, as described above for feed-forward networks. The output from the fully connected layer 908 can be used to generate the output result from the network. The activations within the fully connected layer 908 can be calculated using matrix multiplication instead of convolution. Not all CNN implementations utilize the fully connected layer 908. For example, in some implementations, the convolutional layer 906 can generate the output of the CNN.

[0150] Convolutional layers are sparsely connected, which is different from the conventional neural network configuration seen in the fully connected layer 908. Conventional neural network layers are fully connected, whereby every output unit interacts with every input unit. However, convolutional layers are sparsely connected as the output of the convolution of the field is input into the nodes of the subsequent layer (instead of the respective state values of each of the nodes within the field). The kernel associated with the convolutional layer performs the convolution operation and its output is sent to the next layer. The dimensionality reduction performed by the convolutional layer is one aspect that enables the CNN to scale to process large images.

[0151] Figure 9B represents an exemplary computational stage within the convolutional layer of a CNN. The input 912 to the convolutional layer of the CNN can be processed in three stages of the convolutional layer 914. The three stages can include a convolution stage 916, a detector stage 918, and a pooling stage 920. The convolutional layer 914 can output data to a subsequent convolutional layer. The final convolutional layer of the network can generate the output feature map data or supply an input to a fully connected layer to generate, for example, classification values for the input to the CNN.

[0152] The convolution stage 916 performs several convolutions in parallel to generate a set of linear activations. The convolution stage 916 can include an affine transformation. An affine transformation can be any transformation that can be specified as a linear transformation plus a translation. Affine transformations include rotation, translation, scaling, and combinations of these transformations. The convolution stage calculates the output of a function (e.g., a neuron) connected to a specific region within the input that can be determined as a local region associated with the neuron. The neuron calculates the inner product between the weights of the neuron and the region within the local input to which the neuron is connected. The output from the convolution stage 916 defines a set of linear activations that are processed by subsequent stages of the convolutional layer 914.

[0153] The linear activations can be processed by the detector stage 918. In the detector stage 918, non-linear activations are processed by a non-linear activation function. The non-linear activation function enhances the non-linear characteristics of the network without affecting each field of the convolutional layer. Several types of non-linear activation functions may be used. One specific type is the rectified linear unit (ReLU), which uses an activation function defined as f(x) = max(0, x) such that the activation is thresholded at zero.

[0154] Pooling stage 920 uses a pooling function that replaces the output of convolutional layer 906 with a summary statistic of nearby outputs. The pooling function can be used to introduce translational invariance into the neural network so that small translations to the input do not change the pooled output. Invariance to local translations can be useful in scenarios where the presence of a feature in the input data is more important than the exact location of the feature. Various types of pooling functions, including max pooling, average pooling, and 12-norm pooling, can be used during pooling stage 920. Furthermore, some CNN implementations do not include a pooling stage. Instead, such implementations substitute additional convolutional stages with an increased stride with respect to the previous convolutional layer.

[0155] The output from convolutional layer 914 can then be processed by the next layer 922. The next layer 922 can be either an additional convolutional layer or one of the fully-connected layers 908. For example, the first convolutional layer 904 in FIG. 9A can output to the second convolutional layer 906, while the second convolutional layer can output to the first layer of the fully-connected layer 908.

[0156] Figure 10 represents an example recurrent neural network 100. In a recurrent neural network (RNN), the previous state of the network affects the output of the current state of the network. RNNs can be constructed in various ways using various functions. The use of RNNs generally centers around using a mathematical model to predict the future based on previous input sequences. For example, an RNN may be used to perform statistical language modeling to predict the coming words, assuming a previous word sequence. The depicted RNN 1000 can be described as having an input layer 1002 that receives an input vector, a hidden layer 1004 that implements a regression function, a feedback mechanism 1005 that enables'memory' of the previous state, and an output layer 1006 that outputs the result. The RNN 1000 operates based on time steps. The state of the RNN at a given time step is affected based on the previous time step via the feedback mechanism 1005. For a given time step, the state of the hidden layer 1004 is defined by the previous state and the input at the current time step. The first input (x1) at the first time step can be processed by the hidden layer 1004. The second input (x2) can be processed by the hidden layer 1004 using the state information determined during the processing of the first input (x1). A given state is s t =f(Ux t +Ws t-1 ) and can be calculated. At this time, U and W are parameter matrices. The function f is generally a non-linearity such as the hyperbolic tangent function (Tanh) or the rectified linear function f(x)=max(0,x). However, the specific mathematical function used in the hidden layer 1004 can vary depending on the specific implementation details of the RNN 1000.

[0157] In addition to the basic CNN and RNN networks described, variations to those networks may be possible. An example of a variation of the RNN is the long short term memory (LSTM) RNN. The LSTM RNN is capable of learning the long-term dependencies that may be necessary to process longer language sequences. An example of a variation of the CNN is the convolutional deep belief network that has a structure similar to the CNN and is trained in a manner similar to the deep belief network. The deep belief network (DBN) is a generative neural network consisting of multiple layers of probabilistic (random) variables. The DBN can be trained layer by layer using unsupervised greedy learning. The learned weights of the DBN can then be used to provide a pre-trained neural network by determining an optimal initial set of weights for the neural network.

[0158] Figure 11 illustrates the training and deployment of a deep neural network. When a given network is structured for a task, the neural network is trained using a training data set 1102. Various training frameworks 1104 have been developed to enable hardware acceleration of the training process. For example, the machine learning framework 604 of FIG. 6 may be configured as the training framework 1104. The training framework 1104 is connected to an untrained neural network 1106 and can enable the untrained neural network 1106 to be trained using the parallel processing resources described herein to generate a trained neural network 1108.

[0159] To initiate the training process, the initial weights may be selected randomly or by pre-training using a deep belief network. The training cycle is then executed either in a supervised or unsupervised manner.

[0160] Supervised learning is a learning method in which training is performed as a mediated operation, for example, when the training dataset 1112 includes an input paired with a desired output for the input, or when the training dataset includes an input with a known output and the output of the neural network is manually labeled. The network processes the input and compares the resulting output with the set of expected or desired outputs. Then, the error is backpropagated through the system. The training framework 1104 can adjust the weights that control the untrained neural network 1106. The training framework 1104 can provide a tool to monitor how well the untrained neural network 1106 converges towards a model suitable for generating the correct answer based on known input data. The training process is repeated such that the weights of the network are adjusted to refine the output generated by the neural network. The training process can continue until the neural network reaches a statistically desirable regime associated with the trained neural network 1108. The trained neural network 1108 can then be deployed to implement any number of machine learning operations to generate an inference result 1114 based on the input of new data 1102.

[0161] Unsupervised learning is a learning model in which the network attempts to train itself using unlabeled data. Thus, for unsupervised learning, the training dataset 1112 will contain inputs without any associated output data. The untrained neural network 1106 can learn the groupings within the unlabeled inputs and determine how individual inputs relate to the overall dataset. Unsupervised training can be used to generate a self-organizing map, a type of trained neural network 1108 that can perform operations useful for reducing the dimensionality of the data. Unsupervised training can also be used to perform anomaly detection. This enables the identification of data points within an input dataset that deviate from the normal pattern of the data.

[0162] Variations of supervised and unsupervised training may also be used. Semi-supervised learning is a technique in which the training dataset 1112 contains a mixture of labeled and unlabeled data from the same distribution. Incremental learning is a variation of supervised learning in which input data can be continuously used to further train the model. Incremental learning enables the trained neural network 1108 to adapt to new data 1112 without forgetting the knowledge implanted in the network during the initial training.

[0163] Regardless of whether it is supervised or unsupervised, the training process, especially for deep neural networks, can be computationally too expensive for a single computing node. Instead of using a single computing node, a distributed network of computing nodes can be used to accelerate the training process.

[0164] FIG. 12 is a block diagram representing distributed learning. Distributed learning is a training model that uses multiple distributed computing nodes to perform supervised or unsupervised training of a neural network. Each of the distributed computing nodes can include one or more host processors and one or more general-purpose processing nodes, such as the general-purpose graphics processing unit 700 as seen in FIG. 7. As represented, distributed learning can perform model parallelism 1202, data parallelism 1204, or a combination of model and data parallelism 1206.

[0165] In model parallelism 1202, different computing nodes within a distributed system can execute training computations for different parts of a single network. For example, each layer of a neural network can be trained by different processing nodes of a distributed system. An advantage of model parallelism is the ability to scale, particularly to very large models. Separating the computations associated with different layers of a neural network enables the training of very large neural networks where the weights of all layers do not fit into the memory of a single computing node. In some cases, model parallelism can be particularly useful for performing unsupervised training of large neural networks.

[0166] In data parallel processing 1204, different nodes of a distributed network have a complete instance of the model, and each node receives a different portion of the data. The results from the different nodes are then combined. While different approaches to data parallel processing are possible, all data parallel training approaches require techniques for combining the results and synchronizing the model parameters across the nodes. Examples of approaches to data combination include data parallel processing based on parameter averaging and updates. Parameter averaging trains each node with a subset of the training data and sets the global parameters (e.g., weights, biases) to the average of the parameters from each node. Parameter averaging uses a central parameter server that holds the parameter data. Data parallel processing based on updates is similar to parameter averaging, except that instead of transferring parameters from the nodes to the parameter server, updates to the model are transferred. Further, data parallel processing based on updates can be executed in a decentralized manner, and the updates are compressed and transferred between nodes.

[0167] Composite model and data parallel processing 1206 can be implemented, for example, in a distributed system where each computing node includes a plurality of GPUs. Each node has a complete instance of the model, and separate GPUs within each node are used to train different portions of the model.

[0168] Distributed learning has increased overhead compared to training on a single machine. However, the parallel processors and GPGPUs described in this application can implement various techniques to reduce the overhead of distributed training, including techniques that enable high bandwidth GPU-to-GPU data transfer and accelerated remote data synchronization.

[0169] [Example Machine Learning Applications] Machine learning can be applied to solve various technical problems including, but not limited to, computer vision, autonomous driving and navigation, speech recognition, and language processing. Computer vision has traditionally been one of the most active research areas for machine learning applications. Applications of computer vision range from reproducing human visual capabilities such as face recognition to creating new categories of visual capabilities. For example, a computer vision application can be configured to recognize sound waves from vibrations caused by objects visible in a video. Machine learning accelerated by parallel processors enables computer vision applications to be trained using significantly larger training data sets than previously achievable and enables inference systems to be deployed using lower power parallel processors.

[0170] Machine learning accelerated by parallel processors has applications in autonomous driving including lane and road sign recognition, obstacle avoidance, navigation, and driving control. The accelerated machine learning techniques can be used to train a driving model based on a data set that defines appropriate responses to specific training inputs. The parallel processors described in this application can enable the rapid training of increasingly complex neural networks used for autonomous driving solutions and enable the deployment of low power inference processors in mobile platforms suitable for integration into self-driving vehicles.

[0171] Deep neural networks accelerated by parallel processors have enabled a machine learning approach to automatic speech recognition (ASR). ASR involves the generation of a function that computes the most likely language sequence given an input acoustic sequence. Accelerated machine learning using deep neural networks has enabled the replacement of previously used hidden Markov models (HMMs) and Gaussian mixture models (GMMs) for ASR.

[0172] Machine learning accelerated by parallel processors can also be used to accelerate natural language processing. The automated learning procedure can utilize statistical inference algorithms to generate models that are robust to erroneous or unfamiliar inputs. Example natural language processor applications include automated machine translation between human languages.

[0173] The parallel processing platforms used for machine learning can be divided into a training platform and a deployment platform. The training platform is generally highly parallel and includes optimizations to accelerate multi-GPU single-node training and multi-node multi-GPU training. Examples of parallel processors suitable for training include the general-purpose graphics processing unit 700 of FIG. 7 and the multi-GPU computing system 800 of FIG. 8. In contrast, deployed machine learning platforms generally include lower-power parallel processors suitable for use in products such as cameras, autonomous robots, and autonomous vehicles.

[0174] FIG. 13 depicts an exemplary inference system - on - a - chip (SOC) 1300 suitable for performing inferences using a trained model. SOC 1300 can incorporate processing components including a media processor 1302, a vision processor 1304, a GPGPU 1306, and a multi - core processor 1308. GPGPU 1306 can be a GPGPU as described in this application, such as GPGPU 700, and multi - core processor 1308 can be a multi - core processor as described in this application, such as multi - core processors 405 - 406. SOC 1300 can further include an on - chip memory 1305 that enables a shared on - chip data pool accessible by each of the processing components. The processing components can be optimized for low - power operation to enable deployment to various machine - learning platforms including autonomous vehicles and autonomous robots. For example, one implementation of SOC 1300 can be used as part of a main control system for an autonomous vehicle. When SOC 1300 is configured for use in an autonomous vehicle, the SOC is designed and configured to comply with relevant functional safety standards for deployment authority.

[0175] During operation, media processor 1302 and vision processor 1304 can cooperate to accelerate computer vision operations. Media processor 1302 can enable low - latency decoding of multiple high - resolution (e.g., 4K, 8K) video streams. The decoded video streams can be written to a buffer within on - chip memory 1305. Vision processor 1304 can then parse the decoded video and perform pre - processing operations on the frames of the decoded video in preparation for processing the frames using a trained image recognition model. For example, vision processor 1304 can accelerate the convolution operations of a CNN used to perform image recognition on high - resolution video data, while the back - end model calculations are performed by GPGPU 1306.

[0176] The multi-core processor 1308 can include control logic that assists in ordering and synchronizing data transfer and shared memory operations executed by the media processor 1302 and the vision processor 1304. The multi-core processor 1308 can also operate as an application processor to execute software applications that can utilize the inference computing capabilities of the GPGPU 1306. For example, at least a portion of the navigation and drive logic can be implemented in software executed by the multi-core processor 1308. Such software can issue compute workloads directly to the GPGPU 1306, or the compute workloads can be issued to the multi-core processor 1308 that can offload at least a portion of these operations to the GPGPU 1306.

[0177] The GPGPU 1306 can include compute clusters such as a low-power configuration of the processing clusters 706A-706H within the general-purpose graphics processing unit 700. The compute clusters within the GPGPU 1306 can support instructions that are specifically optimized to perform inference computations on trained neural networks. For example, the GPGPU 1306 can support instructions for performing low-precision computations such as 8-bit and 4-bit integer vector operations.

[0178] [Further System Overview] FIG. 14 is a block diagram of a processing system 1400. Elements of FIG. 14 that have the same or similar names as elements in any other figure of this application are described for the same elements as seen in the other figures and can operate or function in the same way as described anywhere in this application, can have the same components, and can be linked to other entities, but are not so limited. System 1400 may be used in a single-processor desktop system, a multi-processor workstation system, or a server system with a number of processors 1402 or processor cores 1407. System 1400 may be a processing platform incorporated within a system-on-a-chip (SoC) integrated circuit used in portable, handheld, embedded devices such as within an Internet of Things (IoT) device with a wired or wireless connection to a local or wide area network.

[0179] System 1400 may be a processing system having components corresponding to the components of FIG. 1. For example, in different configurations, processor 1402 or processor core 1407 may correspond to processor 102 of FIG. 1. Graphics processor 1408 may correspond to parallel processor 112 of FIG. 1. External graphics processor 1418 may be one of the add-in devices 120 of FIG. 1. System 1400 can be included in, coupled to, or incorporated within a server-based game platform; a game console including a game and media console; a portable game console, a handheld gaming machine, or an online gaming machine. System 1400 may be part of a cellular phone, smartphone, tablet computer device, or mobile Internet-connected device, such as a laptop with low internal memory. Processing system 1400 can also be a wearable device such as a smartwatch; smart eyewear or clothing enhanced by augmented reality (AR) or virtual reality (VR) capabilities that supply visual, audio, or tactile output to complement real-world visual, audio, or tactile experiences, or supply text, audio, video, holographic images or video, or tactile feedback; or other virtual reality (VR) devices, can be included in, coupled to, or incorporated within it. Processing system 1400 may include or be part of a television receiver or set-top box device.

[0180] System 1400 can be included in, coupled to, or incorporated within an autonomous transportation means such as a bus, tractor-trailer, automobile, motorized or electric bicycle, airplane or glider (or any combination thereof). The autonomous transportation means may use system 1400 to process the environment sensed around the transportation means.

[0181] One or more processors 1402 may each include one or more processor cores 1407 that, when executing, process instructions for performing operations for system or user software. At least one of the one or more processor cores 1407 may be configured to process a particular instruction set 1409. The instruction set 1409 may facilitate computing by a plurality of instruction set computing (CISC), reduced instruction set computing (RISC), or very long instruction word (VLIW). The one or more processor cores 1407 may process different instruction sets 1409. The different instruction sets 1409 may include instructions that facilitate emulation of other instruction sets. The processor core 1407 may also include other processing devices such as a digital signal processor (DSP).

[0182] The processor 1402 may include a cache memory 1404. Depending on the architecture, the processor 1402 may have a single internal cache or multiple levels of internal caches. In some embodiments, the cache memory is shared among various components of the processor 1402. In some embodiments, the processor 1402 may also use an external cache (e.g., a level-3 (L3) cache or a last level cache (LLC)) (not shown). The external cache may be shared among the processor cores 1407 using known cache coherence techniques. The processor 1402 may further include a register file 1406. The register file 1406 may include different types of registers (e.g., integer registers, floating point registers, status registers, and instruction pointer registers) that store different types of data. Some of the registers may be general-purpose registers, while other registers may be specific to the design of the processor 1402.

[0183] One or more processors 1402 may be coupled to one or more interface buses 1410 to transmit communication signals, such as address, data, or control signals, between the processor 1402 and other components within the system 1400. The interface bus 1410 can be a processor bus, such as a variant of a Direct Media Interface (DMI) bus, in one of these embodiments. However, the processor bus is not limited to the DMI bus and may include one or more Peripheral Component Interconnect (e.g., PCI, PCI Express) buses, a memory bus, or other types of interface buses. For example, the processor 1402 may include an integrated memory controller 1416 and a platform controller hub 1430. The memory controller 1416 assists in communication between the memory devices and other components of the system 1400, while the platform controller hub (PCH) 1430 provides connections to I / O devices via a local I / O bus.

[0184] Memory device 1420 can be a dynamic random access memory (DRAM) device, a static random access memory (SRAM) device, a flash memory device, a phase change memory device, or other memory device with appropriate performance to function as process memory. Memory device 1420 can operate as system memory for system 1400, for example, to store data 1422 and instructions 1421 used when one or more processors 1402 execute an application or process. Memory controller 1416 is also coupled to any external graphics processor 1418. External graphics processor 1418 may communicate with one or more graphics processors 1408 within processor 1402 to perform graphics and media operations. In some embodiments, graphics, media, and / or computing operations may be assisted by accelerator 1412, which can be a coprocessor configured to execute a specialized set of graphics, media, or computing operations. For example, accelerator 1412 may be a matrix multiplication accelerator used to optimize machine learning or computing operations. Accelerator 1412 can be a ray tracing accelerator that can be used in cooperation with graphics processor 1408 to perform ray tracing operations. In one embodiment, external accelerator 1419 may be used instead of or in cooperation with accelerator 1412.

[0185] A display device 1411 may be provided, which can be connected to processor 1402. Display device 1411 can be one or more of a built-in display device such as seen in a mobile electronic device or a laptop device, or an external display device attached via a display interface (e.g., DisplayPort, etc.). Display device 1411 can be a head-mounted display (HMD) such as a stereoscopic display device used in virtual reality (VR) applications or augmented reality (AR) applications.

[0186] The platform controller hub 1430 may enable peripheral devices to be connected to the memory device 1420 and the processor 1402 via a high-speed I / O bus. The I / O peripherals include, but are not limited to, an audio controller 1446, a network device 1434, a firmware interface 1428, a wireless transceiver 1426, a touch sensor 1425, and a data storage device 1424 (e.g., non-volatile memory, volatile memory, hard disk drive, flash memory, NAND, 3D NAND, 3D XPoint / memory unit, etc.). The data storage device 1424 can be connected via a peripheral bus such as a Peripheral Component Interconnect (e.g., PCI, PCI Express) bus or via a storage interface (e.g., SATA). The touch sensor 1425 can include a touch screen sensor, a pressure sensor, or a fingerprint sensor. The wireless transceiver 1426 can be a Wi-Fi transceiver, a Bluetooth® transceiver, or a mobile network transceiver, e.g., a 3G, 4G, 5G, or Long-Term Evolution (LTE) transceiver. The firmware interface 1428 enables communication with system firmware and can be, for example, a unified extensible firmware interface (UEFI). The network controller 1434 can enable a network connection to a wired network. In some embodiments, a high-performance network controller (not shown) is coupled to the interface bus 1410. The audio controller 1446 can be a multi-channel high-definition audio controller. In some of these embodiments, the system 1400 includes any legacy I / O controller 1440 that couples legacy (e.g., Personal System 2 (PS / 2)) devices to the system.The platform controller hub 1430 can also be connected to one or more universal serial bus (USB) controllers 1442 that connect input devices such as a combination of a keyboard and mouse 1443, a camera 1444, or other USB input devices.

[0187] As is apparent, the illustrated system 1400 is an example and not a limitation, and other types of data processing systems configured differently may also be used. For example, instances of the memory controller 1416 and the platform controller hub 1430 may be incorporated into a separate external graphics processor such as the external graphics processor 1418. The platform controller hub 1430 and / or the memory controller 1416 may be outside of one or more processors 1402. For example, the system 1400 can include an external memory controller 1416 and a platform controller hub 1430, which may be configured as a memory controller hub and a peripheral controller hub within a system chipset that communicates with the processor 1402.

[0188] For example, a circuit board ("sleds") can be used, on which components such as a CPU, memory, and other components are placed and designed to improve thermal performance. Processing components such as processors may be placed on the upper surface of the sled, while nearby memory such as DIMMs is placed on the bottom surface of the sled. As a result of the enhanced airflow provided by this design, the components can operate at higher frequencies and power levels than in a normal system, improving performance. Further, the sled is configured to blindly couple with power and data communication cables within a rack, thereby enabling quick removal, upgrade, reinstallation, and / or replacement. Similarly, the individual components placed on the sled, such as processors, accelerators, memory, and data storage drives, are configured to be easily upgraded by increasing the spacing between them. In an exemplary embodiment, the components further include a hardware authentication function to prove their authenticity.

[0189] The data center can utilize a single network architecture ("fabric") that supports multiple other network architectures including Ethernet® and Omni-Path. The sled can be coupled to a switch via an optical fiber, which provides higher bandwidth and lower latency than typical twisted pair cables (Category 5, Category 5e, Category 6, etc.). The high-bandwidth, low-latency interconnect and network architecture enable the data center to pool resources such as memory, accelerators (e.g., GPUs, graphics accelerators, FPGAs, ASICs, neural networks and / or artificial intelligence accelerators, etc.), and data storage drives that are physically separated into components during use and provide them to computing resources (e.g., processors) as needed, allowing the computing resources to access the pooled resources as if they were local.

[0190] The power supply or power source can supply voltage and / or current to system 1400 or any of the components or systems described in this application. In one example, the power supply includes an AC-DC (alternating current to direct current) adapter to plug into a wall outlet. Such AC power can be from a renewable energy (e.g., solar energy) power source. In one example, the power source includes a DC power source such as an external AC-DC converter. The power source or power supply can also include wireless charging hardware that charges by proximity to a charging field. The power source can include a built-in battery, an alternating current power source, a motion-based power source, a solar power source, or a fuel cell.

[0191] Figures 15A - C depict a computing system and a graphics processor. Elements in Figures 15A - 15C that have the same or similar names as elements in any other figure of this application are described for the same elements seen in other figures, and can operate or function in the same way as described anywhere in this application, can have the same components, and can be linked to other entities, but are not so limited. Figure 15A may be a variation of one of the processors 1402 and is a block diagram of a processor 1500 that may be used in place of one of them. Thus, any feature combined with the processor 1500 in this application discloses a corresponding combination with the processor 1402, but is not so limited. The processor 1500 may include one or more processor cores 1502A - 1502N, an integrated memory controller 1514, and an integrated graphics processor 1508. When the integrated graphics processor 1508 is desired, a system including the processor will include a graphics processor device that is either within a system chipset or coupled via a system bus. The processor 1500 can include additional cores up to the additional core 1502N represented by the dashed box. Each of the processor cores 1502A - 1502N includes one or more internal cache units 1504A - 1504N. In some embodiments, each processor core 1502A - 1502N also has access to one or more shared cache units 1506.

[0192] The internal cache units 1504A - 1504N and the shared cache unit 1506 represent the cache memory hierarchy within the processor 1500. The cache memory hierarchy may include at least one level of instruction and data cache within each processor core and one or more levels of shared mid - level cache such as level 2 (L2), level 3 (L3), level 4 (L4), or other levels of cache. The highest level of cache before the external memory is classified as the LLC. In some embodiments, cache coherence logic maintains coherence among the various cache units 1506 and 1504A - 1504N.

[0193] The processor 1500 may also include a set of one or more bus controller units 1516 and a system agent core 1510. The one or more bus controller units 1516 manage a set of peripheral buses such as one or more PCI or PCI Express buses. The system agent core 1510 provides management functionality for various processor components. The system agent core 1510 may include one or more integrated memory controllers 1514 that manage access to various external memory devices (not shown).

[0194] One or more of the processor cores 1502A - 1502N may include support for simultaneous multi - threading. The system agent core 1510 includes components for coordinating and operating the cores 1502A - 1502N during multi - threaded processing. The system agent core 1510 may further include a power control unit (PCU) that includes logic and components for adjusting the power states of the processor cores 1502A - 1502N and the graphics processor 1508.

[0195] Processor 1500 may further include a graphics processor 1508 that performs graphics processing operations. In some of these embodiments, the graphics processor 1508 is coupled to a system agent core 1510 that includes a set of shared cache units 1506 and one or more integrated memory controllers 1514. The system agent core 1510 may also include a display controller 1511 that drives the graphics processor output to one or more coupled displays. The display controller 1511 may also be a separate module coupled to the graphics processor via at least one interconnect, or may be incorporated within the graphics processor 1508.

[0196] A ring-based interconnect unit 1512 may be used to couple the internal components of the processor 1500. However, alternative interconnect units may be used, for example, point-to-point interconnects, switched interconnects, or other techniques including techniques well known in the art. In some of these embodiments, according to the ring-based interconnect unit 1512, the graphics processor 1508 is coupled to the ring-based interconnect 1512 via an I / O link 1513.

[0197] An exemplary I / O link 1513 represents at least one of a variety of I / O interconnects including a package I / O interconnect that facilitates communication between a high-performance embedded memory module 1518, such as an eDRAM module, and various processor components. Optionally, each of the processor cores 1502A - 1502N and the graphics processor 1508 can use the embedded memory module 1518 as a shared last-level cache.

[0198] Processor cores 1502A - 1502N may be, for example, homogeneous cores that execute the same instruction set architecture. Alternatively, processor cores 1502A - 1502N may be heterogeneous with respect to instruction set architecture (ISA), with one or more of processor cores 1502A - 1502N executing a first instruction set, while at least one of the remaining cores executes a subset of the first instruction set or a different instruction set. Processor cores 1502A - 1502N may be heterogeneous with respect to microarchitecture, and one or more cores having a relatively higher power consumption are coupled with one or more power cores having a lower power consumption. As another example, processor cores 1502A - 1502N are heterogeneous with respect to computing power. Further, processor 1500 may be implemented on one or more chips, or as a SoC integrated circuit comprising the components shown, in addition to other components.

[0199] FIG. 15B is a block diagram of the hardware logic of a graphics processor core 1519 according to some embodiments described herein. Graphics processor core 1519, sometimes referred to as a core slice, can be one or more graphics cores within a modular graphics processor. Graphics processor core 1519 is an example of one graphics core slice, and the graphics processors described herein may include multiple graphics core slices based on a target power and performance envelope. Each graphics processor core 1519 can include a fixed - function block 1530 coupled with a plurality of sub - cores 1521A - 1521F, also referred to as sub - slices, which include module blocks of general - purpose and fixed - function logic.

[0200] The fixed function block 1530 may include, for example, a geometry / fixed function pipeline 1531 that can be shared by all sub-cores within the graphics processor core 1519 in a lower performance and / or lower power graphics processor implementation. The geometry / fixed function pipeline 1531 may include a 3D fixed function pipeline (e.g., the 3D pipeline 1612 seen in FIG. 16A described later), a video front-end unit, a thread spawner and dispatcher, and an integrated return buffer manager that manages an integrated return buffer (e.g., the integrated return buffer 1718 of FIG. 17 described later).

[0201] The fixed function block 1530 may also include a graphics SoC interface 1532, a graphics microcontroller 1533, and a media pipeline 1534. The graphics SoC interface 1532 provides an interface between the graphics processor core 1519 and other processors within the SoC integrated circuit. The graphics microcontroller 1533 is a programmable sub-processor that can be configured to manage various functions of the graphics processor core 1519, including thread dispatching, scheduling, and preemption. The media pipeline 1534 (e.g., the media pipeline 1616 of FIGS. 16A and 17) includes logic to assist in the decoding, encoding, pre-processing, and / or post-processing of multimedia data, including image and video data. The media pipeline 1534 implements media operations in response to requests to the computation or sampling logic within sub-cores 1521A - 1521F.

[0202] The SoC interface 1532 enables the graphics processor core 1519 to communicate with other components within the SoC, such as a general-purpose application processor core (e.g., a CPU), and / or memory hierarchy elements such as a shared last-level cache memory, system RAM, and / or embedded on-chip or on-package DRAM. The SoC interface 1532 can also enable communication with fixed-function devices within the SoC, such as a camera imaging pipeline, and enable and / or implement the use of global memory atomics that can be shared between the graphics processor core 1519 and the CPU within the SoC. The SoC interface 1532 also implements power management control for the graphics processor core 1519 and can enable an interface between the clock domain of the graphics processor core 1519 and other clock domains within the SoC. Optionally, the SoC interface 1532 enables receipt of command buffers from a command streamer and a global thread dispatcher configured to supply commands and instructions to respective ones of one or more graphics cores within the graphics processor. The commands and instructions can be dispatched to the media pipeline 1534 when media operations are to be performed, or to the geometry and fixed-function pipelines (e.g., geometry and fixed-function pipeline 1531, geometry and fixed-function pipeline 1537) when graphics processing operations are to be performed.

[0203] The graphics microcontroller 1533 can be configured to perform various scheduling and management tasks for the graphics processor core 1519. In one configuration, the graphics microcontroller 1533 can perform graphics and / or compute workload scaling, for example, for the execution unit (EU) arrays 1522A - 1522F, 1524A - 1524F within the sub-cores 1521A - 1521F and various graphics parallel engines. In this workload scheduling, host software executed on the CPU core of the SoC including the graphics processor core 1519 can issue a workload to one of a plurality of graphics processor doorbells that call a scaling operation in an appropriate graphics engine. The scheduling operation then includes determining which workloads should be executed, issuing the workloads to the command streamer, pre-empting existing workloads being executed in the engine, managing the progress of the workloads, and notifying the host software when the workloads are complete. Optionally, the graphics microcontroller 1533 can also facilitate a low power or idle state of the graphics processor core 1519, providing the graphics processor core 1519 with the ability to save and restore registers within the graphics processor core 1519 across low power state transitions independently of the operating system and / or graphics driver software on the system.

[0204] The graphics processor core 1519 may have up to N module sub-cores, more or fewer than the represented sub-cores 1521A - 1521F. For each set of N sub-cores, the graphics processor core 1519 may also include shared function logic 1535, shared and / or cache memory 1536, geometry / fixed function pipeline 1537, and additional fixed function logic 1538 to accelerate various graphics and compute processing operations. The shared function logic 1535 may include logic units (e.g., sampler, mas, and / or inter-thread communication logic) related to the shared function logic 1720 of FIG. 17 that can be shared by each of the N sub-cores within the graphics processor core 1519. The shared and / or cache memory 1536 can be a last-level cache for a set of N sub-cores 1521A - 1521F within the graphics processor core 1519 and can also act as shared memory accessible by multiple sub-cores. The geometry / fixed function pipeline 1537 may be included instead of the geometry / fixed function pipeline 1531 within the fixed function block 1530 and may include the same or similar logic units.

[0205] Graphics processor core 1519 may include additional fixed function logic 1538 that can include various fixed function acceleration logic used by graphics processor core 1519. Optionally, the additional fixed function logic 1538 includes an additional geometry pipeline used in position only shading. In position only shading, there are two geometry pipelines, a full geometry pipeline in the geometry / fixed function pipelines 1538, 1531 and a cull pipeline which is an additional geometry pipeline that may be included within the additional fixed function logic 1538. For example, the cull pipeline may be a reduced version of the full geometry pipeline. The full pipeline and the cull pipeline can execute different instances of the same application, each instance having a separate context. Position only shading can hide long cull runs of discarded triangles and in some instances enables shading to be completed more quickly. For example, the cull pipeline logic within the additional fixed function logic 1538 can execute a position shader in parallel with the main application and generally produces a critical result faster than the full pipeline. The cull pipeline fetches and shades only the vertex position attributes without performing rasterization and rendering to a pixel frame buffer. The cull pipeline can use the generated critical results to compute visibility information for all triangles regardless of whether those triangles are culled. The full pipeline (which may be referred to as the replay pipeline in this instance) can consume the visibility information to skip culled triangles in order to shade only the visible triangles that are ultimately passed to the rasterization phase.

[0206] Optionally, the additional fixed function logic 1538 can also include machine learning acceleration logic, such as fixed function matrix multiplication, for implementations that include optimizations for machine learning training or inference.

[0207] Within each of the graphics sub-cores 1521A to 1521F, there is a set of execution resources that can be used to execute graphics, media, and computing operations in response to requests by a graphics pipeline, a media pipeline, or a shader program. The graphics sub-cores 1521A to 1521F include a plurality of EU arrays 1522A to 1522F, 1524A to 1524F, thread dispatch and inter-thread communication (TD / IC) logic 1523A to 1523F, 3D (e.g., texture) samplers 1525A to 1525F, media samplers 1506A to 1506F, shader processors 1527A to 1527F, and shared local memory (SLM) 1528A to 1528F. The EU arrays 1522A to 1522F, 1524A to 1524F each include a plurality of execution units that are general-purpose graphics processing units capable of performing floating-point and integer / fixed-point logic operations in the service of graphics, media, or computing operations including graphics, media, and compute shader programs. The TD / IC logic 1523A to 1523F performs local thread dispatch and thread control operations for the execution units within the sub-core and facilitates communication between the threads executed by the execution units of the sub-core. The 3D samplers 1525A to 1525F can read texture or other 3D graphics-related data into memory. The 3D samplers can read texture data differently based on the texture format and configured sample state associated with a given texture. The media samplers 1506A to 1506F can perform similar read operations based on the type and format associated with media data. For example, each of the graphics sub-cores 1521A to 1521F can alternatively include an integrated 3D and media sampler. Threads executed by the execution units within each of the sub-cores 1521A to 1521F can utilize the shared local memory 1528A to 1528F within each sub-core such that threads executing within a thread group can execute using a common pool of on-chip memory.

[0208] FIG. 15C is a block diagram of a graphics processor, such as graphics processor 1508, and / or a general-purpose graphics processing unit (GPGPU) 1570 that can be configured as a computing accelerator, according to an embodiment described herein. The GPGPU 1570 can be interconnected with a host processor (e.g., one or more CPUs 1546) and memories 1571, 1572 via one or more system and / or memory buses. Memory 1571 may be a system memory that can be shared with one or more CPUs 1546, while memory 1572 is a device memory dedicated to the GPGPU 1570. For example, components within the GPGPU 1570 and the device memory 1572 may be mapped to memory addresses accessible by one or more CPUs 1546. Access to memories 1571 and 1572 may be facilitated via a memory controller 1568. The memory controller 1568 may include an internal direct memory access (DMA) controller 1569, or alternatively, can include logic that performs operations that would otherwise be performed by a DMA controller.

[0209] The GPGPU 1570 includes a plurality of cache memories including an L2 cache 1553, an L1 cache 1554, an instruction cache 1555, and a shared memory 1556, and at least a part of them may be partitioned as cache memories. The GPGPU 1570 also includes a plurality of computing units 1560A to 1560N. Each of the computing units 1560A to 1560N includes a set of a vector register 1561, a scalar register 1562, a vector logic unit 1563, and a scalar logic unit 1564. The computing units 1560A to 1560N can also include a local shared memory 1565 and a program counter 1566. The computing units 1560A to 1560N can be coupled to a constant cache 1567 that can be used to store constant data that is data that does not change during the execution of a kernel or shader program executed on the GPGPU 1570. The constant cache 1567 can be a scalar data cache, and the cached data can be fetched directly into the scalar register 1562.

[0210] During operation, one or more CPUs 1546 can write commands to the registers or memories of the GPGPU 1570 that are mapped to an accessible address space. The command processor 1557 can read commands from the registers or memories and determine how those commands are to be processed within the GPGPU 1570. Then, a thread dispatcher 1558 can be used to dispatch threads to the computing units 1560A to 1560N that execute those commands. Each of the computing units 1560A to 1560N is independently configurable for conditional computing and can conditionally output the results of the computing to memory. The command processor 1557 can interrupt one or more CPUs 1546 when the issued commands are completed.

[0211] Figures 16A - 16C represent block diagrams of further graphics processors and compute accelerator architectures provided, for example, in accordance with Figures 15A - 15C, according to embodiments described in this application. Elements in Figures 16A - 16C having the same or similar names as elements in any other figure of this application describe the same elements as seen in other figures and can operate or function in the same way as described anywhere in this application, can have the same components, and can be linked to other entities, but are not so limited.

[0212] Figure 16A is a block diagram of a graphics processor 1600, which may be a separate graphics processing unit or may be a graphics processor integrated with other semiconductor devices including, but not limited to, multiple processing cores, or memory devices or network interfaces. Graphics processor 1600 may be a variation of graphics processor 1508 and may be used in place of graphics processor 1508. Accordingly, any features combined with processor 1508 in this application disclose corresponding combinations with processor 1600, but are not so limited. The graphics processor may communicate with registers on the graphics processor and commands placed in processor memory via a memory - mapped I / O interface. Graphics processor 1600 may include a memory interface 1614 for accessing memory. Memory interface 1614 can be an interface to local memory, one or more internal caches, one or more shared external caches, and / or system memory.

[0213] Optionally, the graphics processor 1600 also includes a display controller 1602 that drives display output data to the display device 1618. The display controller 1602 includes hardware for one or more overlay planes for the display and composition of multiple layers of video or user interface elements. The display device 1618 can be an internal or external display device. In one embodiment, the display device 1618 is a head-mounted display device such as a virtual reality (VR) display device or an augmented reality (AR) display device. The graphics processor 1600 may include a video codec engine 1606 that encodes, decodes, or transcodes media to, from, or between one or more media encoding formats including, but not limited to, MPEG (Moving Picture Experts Group) formats such as MPEG-2, AVC (Advanced Video Coding) formats such as H.264 / MPEG-4 AVC, H.265 / HEVC, AOMedia (Alliance for Open Media) VP8, VP9, and SMPTE (Society of Motion Picture & Television Engineers) 421M / VC-1, and JPEG (Joint Photographic Experts Group) formats such as JPEG and MJPEG (Motion JPEG).

[0214] The graphics processor 1600 may include, for example, a block image transfer (BLIT) engine 1604 that performs two-dimensional (2D) rasterizer operations including, for example, bit-boundary block transfer. However, alternatively, 2D graphics operations may be performed using one or more components of the graphics processing engine (GPE) 1610. In some embodiments, the GPE 1610 is a computational engine that performs graphics operations including three-dimensional (3D) graphics operations and media operations.

[0215] The GPE 1610 may include a 3D pipeline 1612 that performs 3D operations such as rendering 3D images and scenes using processing functions that operate according to 3D primitive shapes (e.g., rectangles, triangles, etc.). The 3D pipeline 1612 includes programmable fixed functions that perform various tasks within the element and / or generate execution threads for the 3D / media subsystem 1615. While the 3D pipeline 1612 can be used to perform media operations, embodiments of the GPE 1610 also include a media pipeline 1616 that is specifically used to perform media operations such as video post-processing and image enhancement.

[0216] The media pipeline 1616 may include a fixed function or programmable logic unit that performs one or more specialized media operations such as video decoding acceleration, video deinterlacing, and video encoding acceleration, instead of or on behalf of the video codec engine 1606. The media pipeline 1616 may further include a thread generation unit that generates threads to be executed in the 3D / media subsystem 1615. The generated threads execute calculations for media operations on one or more graphics execution units included in the 3D / media subsystem 1615.

[0217] The 3D / media subsystem 1615 may include logic for executing threads generated by the 3D pipeline 1612 and the media pipeline 1616. These pipelines may send thread execution requests to the 3D / media subsystem 1615. The 3D / media subsystem 1615 includes thread dispatch logic that arbitrates various requests and dispatches them to available thread execution resources. The execution resources include an array of graphics execution units that process 3D and media threads. The 3D / media subsystem 1615 may include one or more internal caches for thread instructions and data. Additionally, the 3D / media subsystem 1615 may also include a shared memory that includes registers and addressable memory for sharing data between threads and storing output data.

[0218] FIG. 16B depicts a graphics processor 1620, which is a variation of the graphics processor 1600, and may be used in place of the graphics processor 1600, and vice versa. Thus, any features combined with the processor 1600 in the present application disclose corresponding combinations with the processor 1602, but are not so limited. The graphics processor 1620 has an architecture on a tile according to the embodiments described in the present application. The graphics processor 1620 may include a graphics processing engine cluster 1622 having a plurality of instances of the graphics processing engine 1610 of FIG. 16A within the graphics engine tiles 1610A-1610D. Each of the graphics engine tiles 1610A-1610D may be interconnected via a set of interconnects 1623A-1623F. Each of the graphics engine tiles 1610A-1610D may also be connected to a memory module or memory device 1626A-1626D via a memory interconnect 1625A-1625D. The memory devices 1626A-1626D may use any graphics memory technology. For example, the memory devices 1626A-1626D may be graphics double data rate (GDDR) memories. The memory devices 1626A-1626D may be high bandwidth memory (HBM) modules that can be on-die with their respective graphics engine tiles 1610A-1610D. The memory devices 1626A-1626D may also be stacked memory devices that can be stacked on top of their respective graphics engine tiles 1610A-1610D. Each of the graphics engine tiles 1610A-1610D and the associated memories 1626A-1626D may be present on separate chiplets that are affixed to a base die or a base substrate, as will be described in more detail in FIGS. 24B-24D.

[0219] The graphics processor 1620 may be configured in non-uniform memory access (NUMA) where memory devices 1626A - 1626D are coupled to graphics engine tiles 1610A - 1610D. A given memory device may be accessed by graphics engine tiles other than the tile to which it is directly connected. However, the access latency to memory devices 1626A - 1626D is lowest when accessing the local tile. In one embodiment, a cache coherent NUMA (ccNUMA) system may be enabled, which uses tile interconnects 1623A - 1623F to enable communication between cache controllers within graphics engine tiles 1610A - 1610D to maintain a consistent memory image when more than one cache stores the same memory location.

[0220] The graphics processing engine cluster 1622 can be connected to an on-chip or on-package fabric interconnect 1624. The fabric interconnect 1624 can enable communication between the graphics engine tiles 1610A - 1610D and components such as video codecs 1606 and one or more copy engines 1604. The copy engine 1604 can be used for moving data to, from, and between memory devices 1626A - 1626D and memory outside the graphics processor 1620 (e.g., system memory). The fabric interconnect 1624 can also be used to interconnect the graphics engine tiles 1610A - 1610D. The graphics processor 1620 may optionally include a display controller 1602 that enables connection to an external display device 1618. The graphics processor 1620 may also be configured as a graphics or compute accelerator. In the accelerator configuration, the display controller 1602 and the display device 1618 may be omitted.

[0221] The graphics processor 1620 can be connected to a host system via a host interface 1628. The host interface 1628 can enable communication between the graphics processor 1620, the system memory, and / or other system components. The host interface 1628 can be, for example, a PCI Express bus or other type of host system interface.

[0222] FIG. 16C depicts a compute accelerator 1630 according to an embodiment described herein. The compute accelerator 1630 can include architectural similarities to the graphics processing engine 1620 of FIG. 16B and is optimized for compute acceleration. The compute engine cluster 1632 can include a set of compute engine tiles 1640A-1640D that include execution units optimized for parallel or vector-based general-purpose compute operations. The compute engine tiles 1640A-1640D may not include fixed-function graphics processing logic, but in some embodiments, one or more of the compute engine tiles 1640A-1640D can include logic for performing media acceleration. The compute engine tiles 1640A-1640D can be connected to memories 1626A-1626D via memory interconnects 1625A-1625D. The memories 1626A-1626D and the memory interconnects 1625A-1625D can be the same technology as seen in the graphics processor 1620, or can be different. The graphics compute engine tiles 1640A-1640D can also be interconnected via a set of tile interconnects 1623A-1623F and can be connected to and / or interconnected by the fabric interconnect 1624. In one embodiment, the compute accelerator 1630 can include a large L3 cache 1636 that can be configured as a device-wide cache. The compute accelerator 1630 can also be connected to the host processor and memory via a host interface 1628 in the same manner as the graphics processor 1620 of FIG. 16B.

[0223] [Graphics Processing Engine] FIG. 17 is a block diagram of a graphics processing engine 1710 of a graphics processor according to some embodiments. The graphics processing engine (GPE) 1710 may be a variation of the GPE 1610 shown in FIG. 16A and may represent the graphics engine tiles 1610A-1610D of FIG. 16B. Elements of FIG. 17 that have the same or similar names as elements in some figure of this application are described for the same elements as seen in other figures and may operate or function in the same manner as described anywhere in this application, may have the same components, and may be linked to other entities, but are not so limited. For example, the 3D pipeline 1612 and the media pipeline 1616 of FIG. 16A are also represented in FIG. 17. The media pipeline 1616 is optional in some embodiments of the GPE 1710 and may not be explicitly included within the GPE 1710. For example, in at least one embodiment, a separate media and / or image processor is coupled to the GPE 1710.

[0224] The GPE1710 may be coupled to, or may include, a command streamer 1703 that supplies a command stream to the 3D pipeline 1612 and / or the media pipeline 1616. Alternatively, or additionally, the command stream 1703 may be directly coupled to an integrated return buffer 1718, which may be communicatively coupled to a graphics core array 1714. Optionally, the command streamer 1703 may be coupled to a memory that can be system memory, or one or more of an internal cache memory and a shared cache memory. The command streamer 1703 may receive commands from the memory and send the commands to the 3D pipeline 1612 and / or the media pipeline 1616. The commands are instructions fetched from a ring buffer that stores commands for the 3D pipeline 1612 and the media pipeline 1616. The ring buffer may further include a batch command buffer that stores batches of multiple commands. Commands for the 3D pipeline 1612 may also include references to data stored in memory, such as, but not limited to, vertex and geometry data for the 3D pipeline 1612, and / or image data and memory objects for the media pipeline 1616. The 3D pipeline 1612 and the media pipeline 1616 process commands and data by executing operations by logic within each pipeline, or by dispatching one or more execution threads to the graphics core array 1714. The graphics core array 1714 may include one or more blocks of graphics cores (e.g., graphics core 1715A, graphics core 1715B), and each block may include one or more graphics cores. Each graphics core includes a set of graphics execution resources that includes general-purpose and graphics-specific execution logic for performing graphics and computational operations, and fixed-function texture processing and / or machine learning and artificial intelligence acceleration logic.

[0225] In various embodiments, the 3D pipeline 1612 can include fixed function and programmable logic for processing one or more shader programs, such as vertex shaders, geometry shaders, pixel shaders, fragment shaders, compute shaders, or other shader programs, by processing instructions and dispatching execution threads to the graphics core array 1714. The graphics core array 1714 provides an integrated block of execution resources used when processing these shader programs. The general-purpose execution logic (e.g., execution units) within the graphics cores 1715A - 1715B of the graphics core array 1714 includes support for various 3D API shader languages and can execute multiple concurrent execution threads associated with multiple shaders.

[0226] The graphics core array 1714 may include execution logic for performing media functions such as video and / or image processing. The execution units may include general-purpose logic that is programmable to perform parallel general-purpose computing operations in addition to graphics processing operations. The general-purpose logic can perform processing operations in parallel with or in conjunction with the general-purpose logic within the processor core 1407 of FIG. 14 or cores 1502A - 1502N of FIG. 15A.

[0227] The output data generated by thread execution on the graphics core array 1714 can output data to memory within the unified return buffer (URB) 1718. The URB 1718 can store data for multiple threads. The URB 1718 may be used to send data between different threads executed on the graphics core array 1714. The URB 1718 may further be used for synchronization between threads in the fixed function logic within the graphics core array 1714 and the shared function logic 1720.

[0228] Optionally, the graphics core array 1714 may be scalable such that the array includes a variable number of graphics cores each having a variable number of execution units based on the target power and performance levels of the GPE 1710. The execution resources may be dynamically scalable such that the execution resources can be enabled or disabled as needed.

[0229] The graphics core array 1714 is coupled to shared function logic 1720 that includes a plurality of resources shared among the graphics cores within the graphics core array. The shared functions within the shared function logic 1720 are hardware logic units that provide specialized auxiliary functions to the graphics core array 1714. In various embodiments, the shared function logic 1720 includes, but is not limited to, sampler 1721, math 1722, and inter-thread communication (ITC) 1723 logic. Additionally, one or more caches 1725 within the shared function logic 1720 may be implemented.

[0230] The shared functions are implemented at least when the demand for a given specialized function is insufficient to be included within the graphics core array 1714. Instead, a single instantiation of this specialized function is implemented as a stand-alone entity in the shared function logic 1720 and shared among the execution resources within the graphics core array 1714. The exact set of functions shared among the graphics core arrays 1714 and included within the graphics core arrays 1714 varies for each embodiment. Certain shared functions within the shared function logic 1720 that are widely used by the graphics core arrays 1714 may be included within the shared function logic 1716 within the graphics core arrays 1714. Optionally, the shared function logic 1716 within the graphics core arrays 1714 can include some or all of the logic within the shared function logic 1720. All logic elements within the shared function logic 1720 may be replicated within the shared function logic 1716 of the graphics core arrays 1714. Alternatively, the shared function logic 1720 supports and is excluded from the shared function logic 1716 within the graphics core arrays 1714.

[0231] [Execution Unit] Figures 18A - 18B depict a thread execution logic 1800 that includes an array of processing elements used in a graphics processor core, in accordance with embodiments described herein. Elements in Figures 18A - 18B that have the same or similar names as elements in any other figure of this application describe the same elements as seen in the other figures and can operate or function in the same manner as described anywhere in this application, can have the same components, and can be linked to other entities, but are not so limited. Figures 18A - 18B represent an overview of the thread execution logic 1800 that can represent the hardware logic represented by each sub - core 1521A - 1521F in Figure 15B. Figure 18A represents the execution units within a general - purpose graphics processor, while Figure 18B represents the execution units that can be used within a compute accelerator.

[0232] As shown in FIG. 18A, the thread execution logic 1800 may include a shader processor 1802, a thread dispatcher 1804, an instruction cache 1806, a scalable execution unit array including a plurality of execution units 1808A to 1808N, a sampler 1810, a shared local memory 1811, a data cache 1812, and a data port 1814. Optionally, the scalable execution unit array may be dynamically scalable by enabling or disabling one or more execution units (e.g., any of execution units 1808A, 1808B, 1808C, 1808D, up to 1808N-1 and 1808N) based on the computational requirements of the workload. The components included may be interconnected via an interconnect fabric that links to each of the components. The thread execution logic 1800 may include one or more connections to memory such as system memory or cache memory through the instruction cache 1806, the data port 1814, the sampler 1810, and one or more of the execution units 1808A to 1808N. Each execution unit (e.g., 1808A) may be a stand-alone programmable general-purpose computing unit capable of executing a plurality of simultaneous hardware threads while processing a plurality of data elements in parallel for each thread. In various embodiments, the array of execution units 1808A to 1808N may be scalable to include any number of individual execution units.

[0233] Execution units 1808A to 1808N may be mainly used to execute shader programs. The shader processor 1802 can process various shader programs and dispatch execution threads related to the shader programs by the thread dispatcher 1804. The thread dispatcher may include logic that arbitrates thread start requests from the graphics and media pipeline and instantiates the requested threads in one or more of the execution units 1808A to 1808N. For example, the geometry pipeline can dispatch vertex, tessellation, or geometry shaders to the thread execution logic for processing. Optionally, the thread dispatcher 1804 can also process runtime thread generation requests from the executing shader program.

[0234] Execution units 1808A to 1808N may support an instruction set that includes native support for many standard 3D graphics shader instructions so that shader programs from a graphics library (e.g., Direct 3D and OpenGL) can be executed with minimal conversion. The execution units support vertex and geometry processing (e.g., vertex programs, geometry programs, vertex shaders), pixel processing (e.g., pixel shaders, fragment shaders), and general-purpose processing (e.g., compute and media shaders). Each of execution units 1808A to 1808N is capable of multi-issue SIMD (single instruction multiple data) execution, and multithreaded operation enables an efficient execution environment in the face of higher-latency memory accesses. Each hardware thread within each execution unit has a dedicated high-bandwidth register file and associated independent thread state. Execution multi-issues clock-by-clock into a pipeline capable of integer, single-precision and double-precision floating-point arithmetic, SIMD branching capabilities, logical operations, transcendental operations, and various other miscellaneous operations. While waiting for data from one of the memory or shared functions, the dependency logic within execution units 1808A to 1808N puts the waiting thread to sleep until the requested data is returned. While the waiting thread is sleeping, the hardware resources may be applied to processing other threads. For example, during the latency associated with vertex shader operations, the execution unit can execute operations of other types of shader programs, such as pixel shaders, fragment shaders, or another vertex shader such as vertex shader 2107 shown in FIG. 21. Various embodiments are applicable to use execution by means of SIMT (single instruction multiple thread) instead of, or in addition to, the use of SIMD. References to SIMD cores or operations can also apply to SIMT, or to SIMD combined with SIMT.

[0235] Each execution unit among execution units 1808A to 1808N acts on an array of data elements. The number of data elements is the "execution size" for the instruction or the number of execution channels, and the execution channels are the logic units for data element access, masking, and flow control within the instruction. The number of channels can be independent of the number of physical arithmetic logic units (ALUs), floating-point units (FPUs), or other logic units (e.g., tensor cores, ray tracing cores, etc.) for a particular graphics processor. Furthermore, execution units 1808A to 1808N may support integer and floating-point data types.

[0236] The execution unit instruction set includes SIMD instructions. Various data elements can be stored in registers as packed data types, and the execution units will process various elements based on the data size of the elements. For example, when acting on a 256-bit wide vector, the 256 bits of the vector are stored in a register, and the execution unit acts on the vector as 4 separate 64-bit packed data elements (Quad-Word, QW size data elements), 8 separate 32-bit packed data elements (Double Word, DW size data elements), 16 separate 16-bit packed data elements (Word, W size data elements), or 32 separate 8-bit data elements (Byte, B size data elements). However, different vector widths and register sizes are possible.

[0237] Optionally, one or more execution units can be grouped into fused execution units 1809A - 1809N that have thread control logic (1807A - 1807N) common to the fused EUs. Multiple EUs can be fused into EU groups. Each EU within a fused EU group can be configured to execute separate SIMD hardware threads. The number of EUs within a fused EU group can vary according to embodiments. Further, various SIMD widths are executable per EU, including but not limited to SIMD8, SIMD16, and SIMD32. Each of the fused graphics execution units 1809A - 1809N includes at least two execution units. For example, the fused execution unit 1809A includes a first EU 1808A, a second EU 1808B, and a third control logic 1807A common to the first EU 1808A and the second EU 1808B. The thread control logic 1807A controls the threads executed in the fused graphics execution unit 1809A, enabling each EU within the fused execution units 1809A - 1809N to execute using a common instruction pointer register.

[0238] One or more internal instruction caches (e.g., 1806) are included in the thread execution logic 1800 to cache thread instructions for the execution units. One or more data caches (e.g., 1812) may be included in the thread execution logic 1800 to cache thread data during thread execution. Threads executed by the execution logic 1800 can also store explicitly managed data in the shared local memory 1811. A sampler 1810 may be included to provide texture sampling for 3D operations and a media sampler for media operations. The sampler 1810 includes a specialized texture or media sampling function that processes texture or media data during the sampling process before supplying the sampled data to the execution units.

[0239] During execution, the graphics and media pipeline sends thread start requests to the thread execution logic 1800 via thread generation and dispatch logic. When a group of geometry objects is processed and rasterized into pixel data, the pixel processor logic (e.g., pixel shader logic, fragment shader logic, etc.) within the shader processor 1802 further calculates output information and is called to write the results to an output surface (e.g., color buffer, depth buffer, stencil buffer, etc.). The pixel shader or fragment shader may calculate the values of various vertex attributes to be interpolated across the rasterized objects. The pixel processor logic within the shader processor 1802 may then execute a pixel or fragment shader program supplied by an application programming interface (API). To execute the shader program, the shader processor 1802 dispatches threads to execution units (e.g., 1808A) via the thread dispatcher 1804. The shader processor 1802 may use the texture sampling logic within the sampler 1810 to access texture data in a texture map stored in memory. Arithmetic operations on the texture data and input geometry data either calculate pixel color data for each geometry fragment or discard one or more pixels from further processing.

[0240] Furthermore, the data port 1814 may provide a memory access mechanism for the thread execution logic 1800 to output processed data to memory for further processing in the graphics processor output pipeline. The data port 1814 may include or be coupled to one or more cache memories (e.g., data cache 1812) that cache data for memory access by the data port.

[0241] Optionally, the execution logic 1800 can also include a ray tracer 1805 that can provide a ray tracing acceleration function. The ray tracer 1805 can support a ray tracing instruction set that includes instructions / functions for ray generation. The ray tracing instruction set may be the same as or different from the ray tracing instruction set supported by the ray tracing core 372 of FIG. 3C.

[0242] FIG. 18B shows exemplary internal details of the execution unit 1808. The graphics execution unit 1808 can include a combination of an instruction fetch unit 1837, a general-purpose register file array (GFR) 1824, an architecture register file array (ARF) 1826, a thread arbiter 1822, a dispatch unit 1830, a branch unit 1832, a SIMD floating-point unit (FPU) 1834, and optionally, a dedicated integer SIMD ALU 1835. The GRF 1824 and the ARF 1826 include a set of general-purpose register files and architecture register files associated with each simultaneous hardware thread that can be active in the graphics execution unit 1808. The per-thread architecture state may be held in the ARF 1826, while the data used during thread execution is stored in the GRF 1824. The execution state of each thread, including the instruction pointer of each thread, may be held in the thread-specific registers within the ARF 1826.

[0243] The graphics execution unit 1808 may have an architecture that is a combination of simultaneous multi-threading (SMT) and fine-grained interleaved multi-threading (IMT). The architecture may have a modular configuration that can be fine-tuned at design time based on the target number of simultaneous threads and the number of registers per execution unit, and the execution unit resources are divided across the logic used to execute multiple simultaneous threads. The number of logical threads that may be executed by the graphics execution unit 1808 is not limited to the number of hardware threads, and multiple logical threads may be assigned to each hardware thread.

[0244] Optionally, the graphics execution unit 1808 can co-issue a plurality of instructions, each of which may be a different instruction. The thread arbiter 1822 of the graphics execution unit 1808 can dispatch instructions for execution to one of the send unit 1830, the branch unit 1832, or the SIMD FPU 1834. Each execution thread can access 128 general-purpose registers in the GRF 1824, and each register can store 32 bytes that can be accessed as a SIMD8-element vector of 32-bit data elements. Each execution unit thread may have access to 4K bytes in the GRF 1824, but embodiments are not so limited, and more or fewer register resources may be provided in other embodiments. The graphics execution unit 1808 may be partitioned into seven hardware threads that can execute computational operations independently, but the number of threads per execution unit can also vary according to embodiments. For example, up to 16 hardware threads may be supported. In an exemplary embodiment where seven threads can access 4K bytes, the GRF 1824 can store a total of 28K bytes. In another exemplary embodiment where 16 threads can access 4K bytes, the GRF 1824 can store a total of 64K bytes. However, the number of threads per execution unit is not limited to these examples and may be more or less than the given number. The flexible addressing mode can enable registers to be addressed together to effectively form a wider register or to represent a strided rectangular block data structure.

[0245] Additionally, or alternatively, memory operations, sampler operations, and other system communications with longer latencies may be dispatched by "send" instructions executed by the message passing send unit 1830. Branch instructions may be dispatched to a dedicated branch unit 1832 to facilitate SIMD divergence and the resulting convergence.

[0246] The graphics execution unit 1808 may include one or more SIMD floating point units (FPUs) 1834 to perform floating point operations. The FPU 1834 may also support integer calculations. In some instances, the FPU 1834 can perform up to M 32-bit floating point (or integer) operations in SIMD, or up to 2M 16-bit integer or 16-bit floating point operations in SIMD. Optionally, at least one of the FPUs provides enhanced mathematical capabilities to support high throughput transcendental functions and double precision 184-bit floating point. A set of 8-bit integer SIMD ALUs 1835 may also be present and may be particularly optimized to perform operations related to machine learning calculations.

[0247] Optionally, an array of multiple instances of the graphics execution unit 1808 can be instantiated in a graphics subcore grouping (e.g., sub-slice). For scalability, the product innovator can select the exact number of execution units for each subcore grouping. The execution unit 1808 may execute instructions across multiple execution channels. Furthermore, each thread executed on the graphics execution unit 1808 may be executed on a different channel.

[0248] FIG. 19 depicts a further exemplary execution unit 1900. Elements of FIG. 19 that have the same or similar names as elements in any other figure of this application are described for the same elements seen in other figures and can operate or function in the same way as described anywhere in this application, can have the same components, and can be linked to other entities, but are not so limited. The execution unit 1900 can be, for example, a computer-optimized execution unit used in the compute engine tiles 1640A - 1640D seen in FIG. 16C, but is not so limited. The execution unit 1900 can also be used in the graphics engine tiles 1610A - 1610D seen in FIG. 16B. The execution unit 1900 can include a thread control unit 1901, a thread state unit 1902, an instruction fetch / prefetch unit 1903, and an instruction decoding unit 1904. The execution unit 1900 can further include a register file 1906 that stores registers that can be assigned to hardware threads within the execution unit. The execution unit 1900 can further include a send unit 1907 and a branch unit 1908. The send unit 1907 and the branch unit 1908 can operate in the same way as the send unit 1830 and the branch unit 1832 of the graphics execution unit 1808 in FIG. 18B.

[0249] The execution unit 1900 can also include a computing unit 1910 that includes a variety of types of functional units. The computing unit 1910 may also include an ALU unit 1911 that includes an array of arithmetic logic units. The ALU unit 1911 can be configured to perform 64-bit, 32-bit, and 16-bit integer and floating-point operations. The integer and floating-point operations may be performed simultaneously. The computing unit 1910 can also include a systolic array 1912 and a MAC unit 1913. The systolic array 1912 includes a network of data processing units of width W and depth D that can be used to perform vector or other data parallel operations in a systolic fashion. The systolic array 1912 can be configured to perform matrix operations such as matrix inner product operations. The systolic array 1912 can support 16-bit floating-point operations in addition to 8-bit and 4-bit integer operations. The systolic array 1912 can be configured to accelerate machine learning operations. The systolic array 1912 can be configured with the support of bfloat16, which is a 16-bit floating-point format. The MAC unit 1913 can be included to perform a particular subset of mathematical operations in a more efficient and lower-power way than the ALU unit 1911. The MAC unit 1913 can include MAC logic such as the MAC logic 1722 of the shared function logic 1720 of FIG. 17, which is found in the shared function logic of the graphics processing engine provided by other described embodiments. The MAC unit 1913 can be configured to perform 32-bit and 64-bit floating-point operations.

[0250] The thread control unit 1901 includes logic for controlling the execution of threads within the execution unit. The thread control unit 1901 can include thread arbitration logic for starting, stopping, and preempting the execution of threads within the execution unit 1900. The thread state unit 1902 can be used to store the thread state of the threads assigned to be executed in the execution unit 1900. Storing the thread state within the execution unit 1900 enables rapid preemption of threads when those threads are in a blocked or idle state. The instruction fetch / prefetch unit 1903 can fetch instructions from the instruction cache of higher-level execution logic (e.g., instruction cache 1806 as seen in FIG. 18A). The instruction fetch / prefetch unit 1903 can also issue a prefetch request for instructions to be loaded into the instruction cache based on the analysis of the currently executing thread. The instruction decoding unit 1904 can be used to decode the instructions to be executed by the computing unit. The instruction decoding unit 1904 can be used as a secondary decoder to decode complex instructions into the constituent micro-operations.

[0251] The execution unit 1900 further includes a register file 1906 that can be used by the hardware threads executed in the execution unit 1900. The registers within the register file 1906 can be partitioned across the logic used to execute multiple simultaneous threads within the computing unit 1910 of the execution unit 1900. The number of logical threads that can be executed by the graphics execution unit 1900 is not limited to the number of hardware threads, and multiple logical threads can be assigned to each hardware thread. The size of the register file 1906 can vary for each embodiment based on the number of supported hardware threads. Register renaming may be used to dynamically assign registers to hardware threads.

[0252] FIG. 20 is a block diagram representing a graphics processor instruction format 2000. The graphics processor execution unit supports an instruction set having instructions in multiple formats. The solid boxes represent components generally included in the execution unit instructions, while the dashed lines include components that are optional or are included in only a subset of the instructions. The instruction formats 2000 described and illustrated are macro instructions in that they are instructions supplied to the execution unit as opposed to micro-operations that result from instruction decoding when the instructions are processed.

[0253] The graphics processor execution unit described in the present application originally supports instructions in a 128-bit instruction format 2010. A 64-bit compact instruction format 2030 is available for some instructions based on the selected instructions, instruction options, and number of operands. The original 128-bit instruction format 2010 provides access to all instruction options, while some options and operations are restricted in the 64-bit format 2030. The original instructions available in the 64-bit format 2030 vary by embodiment. The instructions are partially compressed using a set of index values in an index field 2013. The execution unit hardware references a set of compression tables based on the index values and uses the compression table output to reconstruct the original instructions in the 128-bit instruction format 2010. Other sizes and formats of instructions are available.

[0254] For each format, the instruction opcode 2012 defines the operation to be performed by the execution unit. The execution unit executes each instruction in parallel across multiple data elements of each operand. For example, in response to an add instruction, the execution unit performs simultaneous addition operations across each color channel representing a texture element or a picture element. By default, the execution unit executes each instruction across all data channels of the operand. The instruction control field 2014 may enable control for specific execution options such as channel selection (e.g., predication) and data channel order (swizzle). For instructions in the 128-bit instruction format 2010, the execution size field 2016 limits the number of data channels executed in parallel. The execution size field 2016 may not be used in the 64-bit compact instruction format 2030.

[0255] Some execution unit instructions have a maximum of three operands, including two source operands src0 2020 and src1 2022, and one destination 2018. The execution unit may support double destination instructions where one destination is implied. Data manipulation instructions can have a third source operand (e.g., SRC2 2024), and the instruction opcode 2012 determines the number of source operands. The last source operand of the instruction can be an immediate (e.g., hard-coded) value passed with the instruction.

[0256] The 128-bit instruction format 2010 may include an access / address mode field 2026 that specifies whether, for example, direct register addressing mode or indirect register addressing mode is used. When direct register addressing mode is used, the register addresses of one or more operands are supplied directly by bits within the instruction.

[0257] The 128-bit instruction format 2010 may also include an access / address mode field 2026 that specifies the instruction's address mode and / or access mode. The access mode may be used to define data access alignment for the instruction. Access modes may be supported that include a 16-byte alignment access mode and a 1-byte alignment access mode, and the byte alignment of the access mode determines the access alignment of the instruction operand. For example, in a first mode, the instruction may use byte-aligned addressing for source and destination operands, and in a second mode, the instruction may use 16-byte-aligned addressing for all source and destination operands.

[0258] The address mode portion of the access / address mode field 2026 may determine whether the instruction should use direct or indirect addressing. When the direct register addressing mode is used, bits within the instruction directly supply the register addresses of one or more operands. When the indirect register addressing mode is used, the register addresses of one or more operands may be calculated based on an address intermediate field and an address register value within the instruction.

[0259] Instructions may be grouped based on the bit fields of opcode 2012 to simplify opcode 2040. For 8-bit opcodes, bits 4, 5, and 6 enable the execution unit to determine the type of opcode. The exact opcode groupings shown are for example only. The move and logical opcode group 2042 may include data movement and logical instructions (e.g., move (mov), compare (cmp)). The move and logical group 2042 may share the five most significant bits (MSBs), with the move (mov) instruction taking the form 0000xxxxb and the logical instruction taking the form 0001xxxxb. The flow control instructions 2044 (e.g., call, jump (jmp)) include instructions that take the form 0010xxxxb (e.g., 0x20). The other instruction group 2046 includes a mix of instructions including synchronization instructions (e.g., wait, send) that take the form 0011xxxxb (e.g., 0x30). The parallel arithmetic instruction group 2048 includes per-component arithmetic instructions (e.g., add, multiply (mul)) that take the form 0100xxxxb (e.g., 0x40). The parallel arithmetic group 2048 executes arithmetic operations in parallel across data channels. The vector arithmetic group 2050 includes arithmetic instructions (e.g., dp4) that take the form 0101xxxxb (e.g., 0x50). The vector arithmetic group executes arithmetic operations such as inner product calculations on vector operands. The opcode decoding 2040 shown may be used in one embodiment to determine which part of the execution unit is to be used to execute the decoded instruction. For example, some instructions may be designated as systolic instructions to be executed by a systolic array. Other instructions, such as ray tracing instructions (not shown), may be routed to a ray tracing core or ray tracing logic within a slice or partition of the execution logic.

[0260] [Graphics Pipeline] FIG. 21 is a block diagram of a graphics processor 2100 according to another embodiment. Elements of FIG. 21 having the same or similar names as elements in any other figure of the present application are described for the same elements seen in the other figures and can operate or function in the same manner as described anywhere in the present application, can have the same components, and can be linked to other entities, but are not so limited.

[0261] The graphics processor 2100 may include various types of graphics pipeline pipelines such as a geometry pipeline 2120, a media pipeline 2130, a display engine 2140, thread execution logic 2150, and a render output pipeline 2170. The graphics processor 2100 may be a graphics processor within a multi-core processing system that includes one or more general-purpose processing cores. The graphics processor may be controlled by commands issued to the graphics processor 2100 via the ring interconnect 2102 or by register writes to one or more control registers (not shown). The ring interconnect 2102 may couple the graphics processor 2100 to other processing components such as other graphics processors or general-purpose processors. Commands from the ring interconnect 2102 are interpreted by a command streamer 2103 that supplies instructions to individual components of the geometry pipeline 2120 or the media pipeline 2130.

[0262] The command streamer 2103 may direct the operation of a vertex fetcher 2105 that reads vertex data from memory and executes vertex processing commands supplied by the command streamer 2103. The vertex fetcher 2105 may supply the vertex data to a vertex shader 2107, and the vertex shader 2107 performs coordinate space transformation and shading operations on each vertex. The vertex fetcher 2105 and the vertex shader 2107 may execute vertex processing instructions by dispatching execution threads to execution units 2152A - 2152B via a thread dispatcher 2131.

[0263] The execution units 2152A - 2152B may be an array of vector processors having instruction sets for performing graphics and media operations. The execution units 2152A - 2152B may have attached L1 caches 2151 that are specific to each array or shared among the arrays. The cache may be configured as a data cache, an instruction cache, or a single cache that is partitioned to contain data and instructions in different partitions.

[0264] The geometry pipeline 2120 may include a tessellation component that performs hardware - accelerated tessellation of 3D objects. The programmable hull shader 2111 may configure the tessellation operations. The programmable domain shader 2117 may provide back - end evaluation of the tessellation output. The tessellator 2113 may operate under the instructions of the hull shader 2111 and may include dedicated logic for generating a set of detailed geometry objects based on a coarse geometry model supplied as an input to the geometry pipeline 2120. Further, if tessellation is not used, the tessellation components (e.g., the hull shader 2111, the tessellator 2113, and the domain shader 2117) may be bypassed.

[0265] A complete geometry object may be processed by a geometry shader 2119 via one or more threads dispatched to execution units 2152A-2152B, or may proceed directly to clipper 2129. The geometry shader 2119 may operate on the entire geometry object rather than on vertices or patches of vertices as seen in the previous stage of the graphics pipeline. When tessellation is disabled, the geometry shader 2119 receives input from the vertex shader 2107. The geometry shader 2119 may be programmable by a geometry shader program to perform geometry tessellation when the tessellation unit is disabled.

[0266] Before rasterization, clipper 2129 processes vertex data. The clipper 2129 may be a fixed-function clipper with clipping and geometry shader functionality or a programmable clipper. The rasterizer and depth test components 2173 within the render output pipeline 2170 may dispatch a pixel shader to convert the geometry object into a per-pixel representation. The pixel shader logic may be included in the thread execution logic 2150. Optionally, an application may bypass the rasterizer and depth test components 2173 and access the non-rasterized vertex data via the stream output unit 2123.

[0267] The graphics processor 2100 has an interconnect bus, an interconnect fabric, or other interconnect mechanism that enables data and message passing among the main components of the processor. In some embodiments, the execution units 2152A-2152B and associated logic units (e.g., L1 cache 2151, sampler 2154, texture cache 2158, etc.) are interconnected via data ports 2156 to execute memory accesses and communicate with the processor's render output pipeline components. The sampler 2154, caches 2151, 2158, and execution units 2152A-2152B may each have separate memory access paths. Optionally, the texture cache 2158 may also be configured as a sampler cache.

[0268] The render output pipeline 2170 may include a rasterizer and depth test component 2173 that converts vertex-based objects to pixel-based representations. The rasterizer logic may include a windower / masker unit that performs fixed-function triangle and line rasterization. Associated render cache 2178 and depth cache 2179 are also available in some embodiments. The pixel operation component 2177 performs pixel-based operations on data, although in some instances, pixel operations related to 2D operations (e.g., bit block image transfer with blending) are performed by the 2D engine 2141 or replaced at display time by the display controller 2143 using an overlay display surface. The shared L3 cache 2175 may be available to all graphics components and enables data sharing without the use of main system memory.

[0269] The graphics processor media pipeline 2130 may include a media engine 2137 and a video front end 2134. The video front end 2134 may receive pipeline commands from a command streamer 2103. The media pipeline 2130 may include a separate command streamer. The video front end 2134 may process media commands before sending the commands to the media engine 2137. The media engine 2137 may include a thread generation function to generate threads for dispatch to thread execution logic 2150 by a thread dispatcher 2131.

[0270] The graphics processor 2100 may include a display engine 2140. The display engine 2140 may be external to the processor 2100 and may be coupled to the graphics processor via a ring interconnect 2102 or other interconnect bus or fabric. The display engine 2140 may include a 2D engine 2141 and a display controller 2143. The display engine 2140 may include dedicated logic capable of operating independently from the 3D pipeline. The display controller 2143 may be coupled to a display device (not shown), which may be an in-system display device or an external display device attached via a display device connector, as seen in a laptop computer.

[0271] The geometry pipeline 2120 and the media pipeline 2130 may be configured to execute operations based on multiple graphics and media programming interfaces and are not specific to any one application programming interface (API). Driver software for the graphics processor may convert API calls that are specific to a particular graphics or media library into commands that can be processed by the graphics processor. Support may be provided for all of the Open Graphics Library (OpenGL), Open Computing Language (OpenCL), and / or Vulkan graphics and compute APIs from the Khronos Group. Support may also be provided for the Direct3D library from Microsoft Corporation. Combinations of these libraries may be supported. Support may also be provided for the Open Source Computer Vision Library (OpenCV). Future APIs with compatible 3D pipelines will also be supported if mapping can be done from the pipeline of the future API to the pipeline of the graphics processor.

[0272] [Graphics Pipeline Programming] FIG. 22A is a block diagram showing a graphics processor command format 2200 used, for example, to program a graphics processing pipeline such as the pipeline described in this application together with FIGS. 16A, 17, and 21. FIG. 22B is a block diagram showing a graphics processor command sequence 2210 according to an embodiment. The solid box in FIG. 22A represents components generally included in a graphics command, and the dashed line includes components that are optional or included only in a subset of the graphics commands. The exemplary graphics processor command format 2200 in FIG. 22A includes data fields that identify a client 2202, a command operation code (opcode) 2204, and data 2206 for the command. A sub-opcode 2205 and a command size 2208 are also included in some commands.

[0273] The client 2202 may specify a client unit of a graphics device that processes command data. The graphics processor command parser may examine the client field of each command to condition further processing of the command and send the command data to the appropriate client unit. The graphics processor client units may include a memory interface unit, a render unit, a 2D unit, a 3D unit, and a media unit. Each client unit may have a corresponding processing pipeline for processing commands. When a command is received by a client unit, the client unit reads the opcode 2204 and, if present, the sub-opcode 2205 to determine the operation to be performed. The client unit executes the command using the information in the data field 2206. For some commands, an explicit command size 2208 is expected to specify the size of the command. The command parser may automatically determine at least a partial size of the command based on the command opcode. The commands may be aligned by multiples of a double word. Other command formats may also be used.

[0274] The flowchart of FIG. 22B represents an exemplary graphics processor command sequence 2210. The software or firmware of a data processing system that characterizes an exemplary graphics processor may use a variation of the command sequence shown to set up, execute, and terminate a set of graphics operations. The sample command sequence is illustrated and described for illustrative purposes only and is not limited to these specific commands or to this command sequence. Further, the commands may be issued as a batch of commands within the command sequence such that the graphics processor processes the sequence of commands at least partially simultaneously.

[0275] The graphics processor command sequence 2210 may begin with a pipeline flush command 2212 to complete the currently pending commands in the pipeline of any active graphics pipeline. Optionally, the 3D pipeline 2222 and the media pipeline 2224 may not operate simultaneously. The pipeline flush is executed to complete any pending commands in the active graphics pipeline. In response to the pipeline flush, the command parser of the graphics processor pauses command processing until the active rendering engine has completed the pending operations and the associated read cache has been invalidated. Optionally, any data in the render cache marked as "dirty" may be flushed to memory. The pipeline flush command 2212 may be used before placing the graphics processor in a low power state or for pipeline synchronization.

[0276] The pipeline selection command 2213 may be used when the command sequence requests an explicit switch between pipelines to the graphics processor. The pipeline selection command 2213 may only be required once within the execution context before issuing pipeline commands, unless the context is to issue commands for both pipelines. The pipeline flush command 2212 may be required immediately before a pipeline switch by the pipeline selection command 2213.

[0277] The pipeline control command 2214 may configure the graphics pipeline for operation and may be used to program the 3D pipeline 2222 and the media pipeline 2224. The pipeline control command 2214 may set the pipeline state for the active pipeline. The pipeline control command 2214 may be used for pipeline synchronization and to clear data from one or more cache memories within the active pipeline before processing a batch of commands.

[0278] The return buffer state command 2216 may be used to set a set of return buffers for each pipeline to write data. Some pipeline operations require the assignment, selection, or setting of one or more return buffers that write intermediate data during operation. The graphics processor may also use one or more return buffers to store output data and to perform cross-thread communication. The return buffer state 2216 may include selecting the size and number of return buffers to use for a set of pipeline operations.

[0279] The remaining commands within the command sequence differ based on the active pipeline for the operation. Based on the pipeline determination 2220, the command sequence is tailored to either the 3D pipeline 2222 starting from the 3D pipeline state 2230 or the media pipeline 2224 beginning from the media pipeline state 2240.

[0280] Commands that set the 3D pipeline state 2230 include 3D state setting commands for vertex buffer state, vertex element state, constant color state, depth buffer state, and other state variables that should be set before 3D primitive commands are processed. The values of these commands are determined at least in part based on the specific 3D API in use. The 3D pipeline state 2230 commands may also be able to selectively invalidate or bypass certain pipeline elements if those elements are not used.

[0281] The 3D primitive 2232 commands may be used to submit 3D primitives to be processed by the 3D pipeline. The commands and associated parameters passed to the graphics processor by the 3D primitive 2232 commands are transferred to the vertex fetch function within the graphics pipeline. The vertex fetch function uses the 3D primitive 2232 command data to generate a vertex data structure. The vertex data structure is stored in one or more return buffers. The 3D primitive 2232 commands may be used to perform vertex operations on the 3D primitives by the vertex shader. To process the vertex shader, the 3D pipeline 2222 dispatches shader execution threads to the graphics processor execution units.

[0282] The 3D pipeline 2222 may be triggered by an execution 2234 command or event. A register may write to trigger command execution. The execution may be triggered by a "go" or "kick" command within a command sequence. Command execution may be triggered using a pipeline synchronization command to flush the command sequence through the graphics pipeline. The 3D pipeline executes geometry processing on 3D primitives. When the operation is complete, the resulting geometry object is rasterized and the pixel engine colors the resulting pixels. Additional commands for controlling pixel shading and pixel backend operations may also be included for these operations.

[0283] The graphics processor command sequence 2210 may traverse the media pipeline 2224 path when performing media operations. In general, the specific use and method of programming for the media pipeline 2224 depends on the media or compute operation to be performed. Specific media decoding operations may be offloaded to the media pipeline during media decoding. The media pipeline may also be bypassed and media decoding may be performed in whole or in part using resources provided by one or more general-purpose processing cores. The media pipeline may also include elements for general-purpose graphics processor unit (GPGPU) operations, and the graphics processor is used to perform SIMD vector operations using a compute shader program that is not explicitly related to the rendering of graphics primitives.

[0284] The media pipeline 2224 may be configured in the same way as the 3D pipeline 2222. A set of commands for setting the media pipeline state 2240 is dispatched or inserted into the command queue before the media object commands 2242. The commands for the media pipeline state 2240 may include data that configures the media pipeline elements used to process the media object. This may include data that configures the video decoding and video encoding logic within the media pipeline, such as an encoding or decoding format. The commands for the media pipeline state 2240 may also support the use of one or more pointers to "indirect" state elements that include a batch of state settings.

[0285] The media object commands 2242 may supply a pointer to the media object for processing by the media pipeline. The media object includes a memory buffer that contains the video data to be processed. Optionally, all media pipeline states must be valid before issuing the media object commands 2242. When the pipeline state is set and the media object commands 2242 are queued, the media pipeline 2224 is triggered by an execute command 2244 or an equivalent execution event (e.g., a register write). The output from the media pipeline 2224 may then be post-processed by operations supplied by the 3D pipeline 2222 or the media pipeline 2224. GPGPU operations may be configured and executed in the same way as media operations.

[0286] [Graphics Software Architecture] Figure 23 represents an exemplary graphics software architecture for a data processing system 2300. Such a software architecture may include a 3D graphics application 2310, an operating system 2320, and at least one processor 2330. The processor 2330 may include a graphics processor 2332 and one or more general-purpose processor cores 2334. The processor 2330 may be the processor 1402 or any other variation of the processors described in this application. The processor 2330 may be used in place of the processor 1402 or any other of the processors described in this application. Thus, the disclosure of any features combined with the processor 1420 or any other of the processors described in this application also discloses the corresponding combination with the graphics processor 2330, but is not so limited. Further, elements of Figure 23 having the same or similar names as elements in any other figure of this application are described for the same elements seen in other figures and may operate or function in the same manner as described anywhere in this application, may have the same components, and may be linked to other entities, but are not so limited. The graphics application 2310 and the operating system 2320 are each executed in the system memory 2350 of the data processing system.

[0287] The 3D graphics application 2310 may include one or more shader programs including shader instructions 2312. The shader language instructions may be in a high-level shader language such as the High-Level Shader Language (HLSL) of Direct3D, the OpenGL Shader Language (GLSL), etc. The application may also include executable instructions 2314 in machine language suitable for execution by the general-purpose processor cores 2334. The application may also include a graphics object 2316 defined by vertex data.

[0288] The operating system 2320 may be a Microsoft® Windows® operating system from Microsoft Corporation, a Unix-like operating system with proprietary specifications, or an open-source Unix®-like operating system using a variant of the Linux® kernel. The operating system 2320 can support a graphics API 2322 such as the Direct3D API, the OpenGL API, or the Vulkan API. When the Direct3D API is in use, the operating system 2320 uses a front-end shader compiler 2324 to compile any shader instructions 2312 in HLSL into a lower-level shader language. The compilation may be just-in-time (JIT) compilation, or alternatively, the application can perform shader pre-compilation. The higher-level shader may be compiled into a lower-level shader during the compilation of the 3D graphics application 2310. The shader instructions 2312 may be supplied in an intermediate form such as a variant of SPIR (Standard Portable Intermediate Representation) used by the Vulkan API.

[0289] The user-mode graphics driver 2326 may include a back-end shader compiler 2327 to convert the shader instructions 2312 into a hardware-specific representation. When the OpengGL API is in use, the shader instructions 2312 in the GLSL higher-level language are passed to the user-mode graphics driver 2326 for compilation. The user-mode graphics driver 2326 may use the operating system kernel-mode function 2328 to communicate with the kernel-mode graphics driver 2329. The kernel-mode graphics driver 2329 may communicate with the graphics processor 2332 to dispatch commands and instructions.

[0290] [IP Core Implementation] One or more aspects may be implemented by representative code stored on a machine-readable medium that represents and / or defines logic within an integrated circuit such as a processor. For example, the machine-readable medium may include instructions that represent various logic within the processor. When read by a machine, the instructions may cause the machine to assemble logic to perform the techniques described herein. Such representations, known as “IP cores,” are reusable units of integrated circuit logic that may be stored on a tangible machine-readable medium as a hardware model that describes the structure of the integrated circuit. The hardware model may be provided to various customers or manufacturing facilities that load the hardware model into an assembly machine that manufactures the integrated circuit. The integrated circuit may be assembled such that the circuit performs the operations described in connection with any of the embodiments herein.

[0291] FIG. 24A is a block diagram depicting an IP core development system 2400 that may be used to manufacture an integrated circuit to perform operations in accordance with an embodiment. The IP core development system 2400 may be used to generate a modular, reusable design that can be incorporated into a larger design or, alternatively, to construct an entire integrated circuit (e.g., a system-on-a-chip integrated circuit). A design agency 2430 may be able to generate a software simulation 2410 of the IP core design in a high-level programming language (e.g., C / C++). The software simulation 2410 may be used to design, test, and verify the behavior of the IP core using a simulation model 2412. The simulation model 2412 may include functional, behavioral, and / or timing simulations. A register transfer level (RTL) design 2415 is an abstraction of the behavior of an integrated circuit that models the flow of digital signals between hardware registers, including associated logic that is executed using the modeled digital signals. In addition to the RTL design 2415, lower-level designs at the logic level or transistor level may also be generated, designed, or synthesized. Thus, the specific details of the initial design and simulation may vary.

[0292] The RTL design 2415 or equivalent may be further synthesized by a design organization into a hardware model, which may be in a hardware description language (HDL) or other representation of physical design data. The HDL may be further simulated or tested to verify the IP core design. The IP core design may be stored for delivery to a third-party assembly facility 2465 using a non-volatile memory 2440 (e.g., hard disk, flash memory, or any non-volatile storage medium). Alternatively, the IP core design may be transmitted via a wired connection 2450 or a wireless connection 2460 (e.g., via the Internet). The assembly facility 2465 may then assemble an integrated circuit based at least in part on the IP core design. The assembled integrated circuit may be configured to perform operations according to at least one embodiment described herein.

[0293] FIG. 24B shows a side cross-sectional view of integrated circuit package assembly 2470. Integrated circuit package assembly 2470 represents the implementation of one or more of the processors or accelerator devices described in the present application. Package assembly 2470 includes a plurality of units 2672, 2674 of hardware logic connected to substrate 2480. Logic 2672, 2674 may be at least partially implemented in configurable logic or fixed-function logic hardware and may include one or more portions of a processor core, a graphics processor, or any of the other accelerator devices described in the present application. Each unit of logic 2672, 2674 is implemented within a semiconductor die and may be coupled to substrate 2480 via interconnect structure 2473. Interconnect structure 2473 may be configured to convey electrical signals between logic 2672, 2674 and substrate 2480 and may include interconnects such as, but not limited to, bumps or pillars. Interconnect structure 2473 may be configured to convey electrical signals such as, for example, input / output (I / O) signals and / or power or ground signals related to the operation of logic 2672, 2674. Optionally, substrate 2480 may be an epoxy-based laminated substrate. Substrate 2480 may include other suitable types of substrates. Package assembly 2470 may be connected to other electrical devices via package interconnect 2483. Package interconnect 2483 may be coupled to the surface of substrate 2480 to convey electrical signals to other electrical devices such as a motherboard, other chip sets, or multi-chip modules.

[0294] The units of logic 2672, 2674 may be electrically coupled to a bridge 2482 configured to convey electrical signals between logic 2672, 2674. Bridge 2482 may be a dense interconnect structure providing a route for electrical signals. Bridge 2482 may include a bridge substrate made of glass or a suitable semiconductor material. An electrical routing structure may be formed on the bridge substrate to provide an inter-chip connection between logic 2672, 2674.

[0295] Two units 2672, 2674 of logic and bridge 2482 are shown, but embodiments described in the present application may include more or fewer logic units on one or more dies. One or more dies may be connected by zero or more bridges in that, when the logic is included on a single die, bridge 2482 may be excluded. Alternatively, multiple dies or units of logic may be connected by one or more bridges. Further, multiple logic units, dies, and bridges may be connected in other possible configurations including three-dimensional configurations.

[0296] FIG. 24C depicts a package assembly 2490 that includes multiple units of a hardware logic chiplet connected to a substrate 2480 (e.g., a base die). The graphics processing units, parallel processors, and compute accelerators described in the present application may be composed of a variety of silicon chiplets that are manufactured separately. In this regard, a chiplet is at least a partially packaged integrated circuit that includes different logic units that can be assembled together with other chiplets into a larger device. A variety of combinations of chiplets with different IP core logic can be assembled within a single device. Further, a chiplet can be integrated into a base die or base chiplet using active interposer technology. The concepts described in the present application enable different forms of IP-aware interconnection and communication within a GPU. IP cores can be manufactured using different process technologies and assembled during manufacturing. This, in particular, in a large-scale SoC with several flavors of IP, avoids the complexity of converging multiple IPs into the same process. Enabling the use of multiple process technologies improves time to market and provides a cost-effective way to create multiple product SKUs. Further, the decomposed IP is more easily independently power-gated, and the power of components not being used for a given workload can be turned off to reduce overall power consumption.

[0297] The hardware logic chiplet can include a dedicated hardware logic chiplet 2472, a logic or I / O chiplet 2474, and / or a memory chiplet 2475. The hardware logic chiplet 2472 and the logic or I / O chiplet 2474 may be at least partially implemented in configurable logic or fixed-function logic hardware, and can include one or more portions of any of the processor cores, graphics processors, parallel processors, or other accelerator devices described in this application. The memory chiplet 2475 can be DRAM (e.g., GDDR, HBM) memory or cache (SRAM) memory.

[0298] Each chiplet can be manufactured as a separate semiconductor die and coupled to the substrate 2480 via an interconnect structure 2473. The interconnect structure 2473 may be configured to transmit electrical signals between various chiplets and logics within the substrate 2480. The interconnect structure 2473 can include interconnects such as, but not limited to, bumps or pillars. In some embodiments, the interconnect structure 2473 may be configured to transmit electrical signals such as input / output (I / O) signals and / or power or ground signals related to the operation of the logic, I / O, and memory chiplets.

[0299] The substrate 2480 may be an epoxy-based laminated substrate, but is not limited thereto, and the substrate 2480 may also include other suitable types of substrates. The package assembly 2490 can be connected to other electrical devices via package interconnects 2483. The package interconnects 2483 may be coupled to the surface of the substrate 2480 to transmit electrical signals to other electrical devices such as a motherboard, other chip sets, or multi-chip modules.

[0300] The logic or I / O chiplet 2474 and the memory chiplet 2475 may be electrically coupled via a bridge 2487 configured to convey electrical signals between the logic or I / O chiplet 2474 and the memory chiplet 2475. The bridge 2487 may be a dense interconnect structure that provides a route for electrical signals. The bridge 2487 may include a bridge structure made of glass or a suitable semiconductor material. The electrical routing function may be formed in the bridge structure to effect an inter-chip connection between the logic or I / O chiplet 2474 and the memory chiplet 2475. The bridge 2487 may also be referred to as a silicon bridge or an interconnect bridge. For example, the bridge 2487 is an Embedded Multi-die Interconnect Bridge (EMIB). Alternatively, the bridge 2487 may simply be a direct connection from one chiplet to another chiplet.

[0301] The substrate 2480 may include hardware components for I / O 2491, cache memory 2492, and other hardware logic 2493. The fabric 2485 may be embedded in the substrate 2480 to enable communication between various logic chiplets and the logic 2491, 2493 within the substrate 2480. Optionally, the I / O 2491, fabric 2485, cache, bridge, and other hardware logic 2493 may be integrated into a base die stacked on the substrate 2480.

[0302] Furthermore, the package assembly 2490 can also include fewer or more components and die interconnected by the fabric 2485 or one or more bridges 2487. The die within the package assembly 2490 may be arranged in a 3D or 2.5D configuration. Generally, the bridge structure 2487 may be used, for example, to facilitate point-to-point interconnects between logic or I / O die and memory die. The fabric 2485 can be used to interconnect various logic and / or I / O die (e.g., die 2472, 2474, 2491, 2493) with other logic and / or I / O die. The cache memory 2492 within the substrate can operate as a global cache for the package assembly 2490, or as part of a distributed global cache, or as a dedicated cache for the fabric 2485.

[0303] Figure 24D depicts a package assembly 2494 that includes a replaceable die 2495, according to an embodiment. The replaceable die 2495 can be assembled within standardized slots on one or more base die 2496, 2498. The base die 2496, 2498 can be similar to other bridge interconnects described herein and can be coupled via a bridge interconnect 2497, which can be, for example, an EMBI. Memory die can also be connected to logic or I / O die via a bridge interconnect. The I / O and logic die can communicate via an interconnect fabric. Each of the base die can support one or more slots in a standardized format for one of logic or I / O or memory / cache.

[0304] The SRAM and power distribution circuitry may be fabricated within one or more of base chiplets 2496, 2498 that may be manufactured using different process technologies for the replaceable chiplet 2495 that is stacked on the base chiplet. For example, base chiplets 2496, 2498 may be manufactured using a larger process technology while the replaceable chiplet may be manufactured using a smaller process technology. One or more of the replaceable chiplets 2495 may be memory (e.g., DRAM) chiplets. Different memory densities may be selected for the package assembly 2494 based on the power and / or performance targeted for the product using the package assembly 2494. Further, logic chiplets with different numbers of types of functional units may be selected at assembly based on the power and / or performance targeted for the product. Further, chiplets including different types of IP logic cores may be insertable within replaceable chiplet slots, enabling a hybrid processor design that can mix and match IP blocks of different technologies.

[0305] [Exemplary System-on-a-Chip Integrated Circuit] Figures 25-26 depict an exemplary integrated circuit and associated graphics processor that may be assembled using one or more IP cores. In addition to those shown, other logic and circuitry may be included that includes additional graphics processors / cores, processors / cores, peripheral interface controllers, or general purpose processor cores. Elements in Figures 25-26 that have the same or similar names as elements in any other figure of this application describe the same elements as seen in the other figures and can operate or function in the same manner as described anywhere in this application, can have the same components, and can be linked to other entities, but are not so limited.

[0306] FIG. 25 is a block diagram representing an exemplary system - on - a - chip integrated circuit 2500 that can be assembled using one or more IP cores. The exemplary integrated circuit 2500 can be a variant of the graphics processors 1408, 1508, 2510, or any of the graphics processors described herein, and can be used in place of any of the described graphics processors. It includes one or more application processors 2505 (e.g., CPUs) and at least one graphics processor 2510. Thus, any disclosure of features in combination with a graphics processor herein also discloses the corresponding combination with the graphics processor 2510, but is not so limited. The integrated circuit 2500 may further include an image processor 2515 and / or a video processor 2520, either of which may be module IP cores from the same or multiple different design facilities. The integrated circuit 2500 may include peripheral or bus logic including a USB controller 2525, a UART controller 2530, an SPI / SDIO controller 2535, and an I 2 S / I 2 2C controller 2540. Further, the integrated circuit 2500 can include a display device 2545 coupled to one or more of a high - definition multimedia interface (HDMI (registered trademark)) controller 2550 and a mobile industry processor interface (MIPI) display interface 2555. Storage may be provided by a flash memory subsystem 2560 including flash memory and a flash memory controller. The memory interface may be provided via a memory controller 2565 for access to SCRAM or SRAM memory devices. Some integrated circuits further include an embedded security engine 2570.

[0307] Figures 26A-26B are block diagrams representing exemplary graphics processors used within an SoC, in accordance with embodiments described herein. The graphics processor may be a variation of graphics processors 1408, 1508, 2510, or any of the graphics processors described herein. The graphics processor may be used in place of graphics processors 1408, 1508, 2510, or any of the graphics processors described herein. Accordingly, any disclosure of features combined with graphics processors 1408, 1508, 2510, or any of the graphics processors described herein also discloses corresponding combinations with the graphics processors of FIGS. 26A-26B, but is not so limited. FIG. 26A represents an exemplary graphics processor 2610 of an SoC integrated circuit that may be assembled using one or more IP cores, in accordance with an embodiment. FIG. 26B represents a further exemplary graphics processor 2640 of an SoC integrated circuit that may be assembled using one or more IP cores, in accordance with an embodiment. The graphics processor 2610 of FIG. 26A is an example of a low-power graphics processor core. The graphics processor 2640 of FIG. 26B is an example of a higher-performance graphics processor core. For example, each of the graphics processors 2610, 2640 may be a variation of the graphics processor 2510 of FIG. 25, as described at the beginning of this paragraph.

[0308] As shown in FIG. 26A, the graphics processor 2610 includes a vertex processor 2605 and one or more fragment processors 2615A - 2615N (e.g., 2615A, 2615B, 2615C, 2615d, up to 2615N - 1, and 2615N). The graphics processor 2610 can execute different shader programs via separate logic such that the vertex processor 2605 is optimized to execute the operations of a vertex shader program, while the one or more fragment processors 2615A - 2615N execute fragment (e.g., pixel) shading operations of a fragment or pixel shader program. The vertex processor 2605 executes the vertex processing stage of the 3D graphics pipeline and generates primitives and vertex data. The fragment processors 2615A - 2615N use the primitives and vertex data generated by the vertex processor 2605 to generate a frame buffer to be displayed on a display device. The fragment processors 2615A - 2615N may be optimized to execute a fragment shader program as provided by the OpenGL API, which can be used to perform operations similar to a pixel shader program as provided by the Direct3D API.

[0309] The graphics processor 2610 further includes one or more memory management units (MMUs) 2620A - 2620B, caches 2625A - 2625B, and circuit interconnects 2630A - 2630B. The one or more MMUs 2620A - 2620B provide virtual - physical address mapping for the graphics processor 2610, which includes a vertex processor 2605 and / or fragment processors 2615A - 2615N that can reference vertex or image / texture data stored in memory in addition to vertex or image / texture data stored in the one or more caches 2625A - 2625B. The one or more MMUs 2620A - 2620B may be synchronized with one or more other MMUs in the system, including one or more MMUs associated with the one or more application processors 2505, image processors 2515, and / or video processors 2520 of FIG. 25, such that each processor 2505 - 2520 can participate in a shared or integrated virtual memory system. Components of the graphics processor 2610 may correspond to components of other graphics processors described in this application. The one or more MMUs 2620A - 2620B may correspond to the MMU 245 of FIG. 2C. The vertex processor 2605 and fragment processors 2615A - 2615N may correspond to the graphics multiprocessor 234. The one or more circuit interconnects 2630A - 2630B enable the graphics processor 2610 to interface with other IP cores within the SoC, according to an embodiment, via the internal bus of the SoC or via a direct connection. The one or more circuit interconnects 2630A - 2630B may correspond to the data crossbar 240 of FIG. 2C. Further correspondences may be found between similar components of the graphics processor 2610 and various graphics processor architectures described in this application.

[0310] As shown in FIG. 26B, the graphics processor 2640 includes one or more MMUs 2620A-2620B, caches 2625A-2625B, and circuit interconnects 2630A-2630B of the graphics processor 2610 of FIG. 26A. The graphics processor 2640 provides an integrated shader core architecture capable of executing all types of programmable shader code, including shader program code in which a single core or type or core implements a vertex shader, a fragment shader, and / or a compute shader, with one or more shader cores 2655A-2655N (e.g., 2655A, 2655B, 2655C, 2655D, 2655E, 2655F, up to 2655N-1, and 2655N). The exact number of shader cores can vary depending on the embodiment and implementation. Further, the graphics processor 2640 includes an inter-core task manager 2645 that operates as a thread dispatcher to dispatch execution threads to one or more shader cores 2655A-2655N and a tiling unit 2658 to accelerate tiling operations for tile-based rendering in which rendering operations for a scene are subdivided in image space, for example, to utilize local spatial coherence within the scene or to optimize the use of internal caches. The shader cores 2655A-2655N may correspond, for example, to the graphics multiprocessor 234 seen in FIG. 2D, or the graphics multiprocessors 325, 350 of FIGS. 3A and 3B, respectively, or the multi-core group 365A of FIG. 3C.

[0311] The embodiments described in this application include software, firmware, and hardware logic that provide techniques for performing arithmetic on sparse data by a systolic processing unit. The embodiments described in this application provide techniques for skipping computational operations on zero-filled matrices and submatrices. The embodiments further provide techniques for maintaining data compression up to the processing unit. The embodiments further provide an architecture for a sparse-aware logic unit.

[0312] [GPGPU with Tensor Acceleration Logic and Integrated Memory] FIG. 27 is a block diagram of a data processing system 2700 according to an embodiment. The data processing system 2700 is a heterogeneous processing system including a processor 2702, an integrated memory 2710, and a GPGPU 2720 including machine learning acceleration logic. The processor 2702 and the GPGPU 2720 can be any of the processors and GPGPUs / parallel processors described in the present application. The processor 2702 can execute instructions for a compiler 2715 stored in a system memory 2712. The compiler 2715 is executed by the processor 2702 to compile source code 2714A into compiled code 2714B. The compiled code 2714B can include instructions executable by the processor 2702 and / or instructions executable by the GPGPU 2720. During compilation, the compiler 2715 can perform an operation of inserting metadata including hints regarding the level of data parallelism present in the compiled code 2714B and / or hints regarding data locality related to threads to be dispatched based on the compiled code 2714B. The compiler 2715 can include information necessary to perform such an operation, or the operation can be performed with the assistance of a runtime library 2716. The runtime library 2716 can also assist the compiler 2715 in the compilation of the source code 2714A and can include instructions linked with the compiled code 2714B at runtime to assist in the execution of the compiled code 2714B on the GPGPU 2720.

[0313] The integrated memory 2710 represents an integrated address space that can be accessed by the processor 2702 and the GPGPU 2720. The integrated memory can include the system memory 2172 together with the GPGPU memory 2718. The GPGPU memory 2718 is memory within the address space of the GPGPU 2720 and can include some or all of the system memory 2712. In one embodiment, the GPGPU memory 2718 can also include at least a portion of any memory dedicated for use solely by the GPGPU 2720. In one embodiment, the compiled code 2714B stored in the system memory 2712 can be mapped to the GPGPU memory 2718 for access by the GPGPU 2720.

[0314] The GPGPU 2720 includes a plurality of compute blocks 2724A - 2724N that can include one or more of the various processing resources described herein. The processing resources can be, or can include, a variety of computational resources such as, for example, execution units, compute units, streaming multiprocessors, graphics multiprocessors, or multi - core groups. In one embodiment, the GPGPU 2720 can further include a tensor (e.g., matrix) accelerator 2723 that can include one or more special function compute units designed to accelerate a subset of matrix operations (e.g., inner products, etc.). The tensor accelerator 2723 can also be referred to as a tensor accelerator or a tensor core. In one embodiment, the logic components within the tensor accelerator 2723 can be distributed across the processing resources of the plurality of compute blocks 2724A - 2724N.

[0315] GPGPU 2720 may also include a set of resources that can be shared by the compute blocks 2724A - 2724N and the tensor accelerator 2723, including but not limited to a set of registers 2725, a power and performance module 2726, and a cache 2727. In one embodiment, the registers 2725 include registers that are directly and indirectly accessible, and the indirectly accessible registers are optimized for use by the tensor accelerator 2723. The power and performance module 2726 may be configured to adjust the power distribution and clock frequency for the compute blocks 2724A - 2724N to power - gate idle components within the compute blocks 2724A - 2724N. In various embodiments, the cache 2727 can include an instruction cache and / or a low - power data cache.

[0316] GPGPU 2720 may further include an L3 data cache 2730 that can be used to cache data accessed from the integrated memory 2710 by the compute elements within the tensor accelerator 2723 and / or the compute blocks 2724A - 2724N. In one embodiment, the L3 data cache 2730 includes a shared local memory 2732 that can be shared by the compute elements within the compute blocks 2724A - 2724N and the tensor accelerator 2723.

[0317] In one embodiment, the GPGPU 2720 includes instruction handling logic such as a fetch and decode unit 2721 and a scheduler controller 2722. The fetch and decode unit 2721 includes a fetch unit and a decode unit that fetches and decodes instructions for execution by one or more of the compute blocks 2724A - 2724N or the tensor accelerator 2723. The instructions can be scheduled by the scheduler controller 2722 to appropriate functional units within the compute blocks 2724A - 2724N or the tensor accelerator 2723. In one embodiment, the scheduler controller 2722 is an ASIC configured to perform advanced scheduling operations. In one embodiment, the scheduler controller 2722 is a microcontroller or a low EPI (energy - per - instruction) processing core that can execute scheduler instructions loaded from a firmware module.

[0318] In one embodiment, some of the functions executed by compute blocks 2724A - 2724N may be directly scheduled or offloaded to tensor accelerator 2723. In various embodiments, tensor accelerator 2723 includes processing element logic configured to efficiently execute matrix - computing operations such as multiplication and addition operations and inner - product operations used by 3D graphics or compute shader programs. In one embodiment, tensor accelerator 2723 may be configured to accelerate operations used by a machine - learning framework. In one embodiment, tensor accelerator 2723 is an application - specific integrated circuit explicitly configured to execute a particular set of parallel matrix multiplication and / or addition operations. In one embodiment, tensor accelerator 2723 is a field - programmable gate array (FPGA) that provides fixed - function logic that can be updated between workloads. The set of matrix operations that can be executed by tensor accelerator 2723 may be limited with respect to the operations that can be executed by compute blocks 2724A - 2724N. However, tensor accelerator 2723 can execute these operations with a higher throughput compared to compute blocks 2724A - 2724N.

[0319] FIG. 28 represents matrix operation 2805 executed by instruction pipeline 2800 according to an embodiment. Instruction pipeline 2800 may be configured to execute a set of matrix operations, such as, but not limited to, inner - product operations. The inner product of two vectors is a scalar value equal to the sum of the products of the corresponding components of the vectors. The inner product can be calculated as shown in Equation (1) below.

Equation

[0320] The inner product can be used in the convolution operation of a convolutional neural network (CNN). FIG. 28 represents a two-dimensional (2D) convolution that uses a matrix operation 2805 including an inner product operation. Although a 2D convolution is shown, an N-dimensional convolution can be performed on an N-dimensional volume using an N-dimensional filter. The receptive field tile 2802 highlights a portion of the input volume in the input volume buffer 2804. The input volume buffer 2804 can be stored in the memory 2830. The inner product matrix operation 2805 can be performed between the data in the receptive field tile 2802 and the convolution filter to generate data points in the output buffer 2806. The output buffer 2806 can also be stored in the memory 2830. The memory 2830 can be any of the memories described in the present application, including the system memory 2712, GPGPU memory 2718, or one or more caches 2727, 2730 as seen in FIG. 27.

[0321] The combination of data points in the output buffer 2806 represents an activation map generated by the convolution operation. Each point in the activation map is generated by sliding the receptive field tile 2802 across the input volume buffer 2804. The activation map can be input to an activation function to determine the output activation value. In one embodiment, the convolution of the input volume buffer 2804 can be defined in the framework as a high-level matrix operation 2805. The high-level matrix operation 2805 can be executed by primitive operations such as BLAS (basic linear algebra subprogram) operations. The primitive operations can be accelerated by hardware instructions executed by the instruction pipeline 2800.

[0322] The instruction pipeline 2800 used to accelerate hardware instructions can include an instruction fetch and decoding unit 2721 capable of fetching and decoding hardware instructions, and a scheduler controller 2722 capable of scheduling the decoded instructions to one or more processing resources within the calculation blocks 2724A to 2724N and / or the tensor accelerator 2723. In one embodiment, the hardware instructions can be scheduled to the calculation blocks 2724A to 2724N and offloaded to the tensor accelerator 2723. One or more hardware instructions and related data for performing the matrix operation 2805 can be stored in the memory 2830. The output of the hardware instructions can also be stored in the memory 2830.

[0323] In one embodiment, the tensor accelerator 2723 can execute one or more hardware instructions to perform the matrix operation 2805 using the built-in systolic array 2808 (DP logic). The systolic array 2808 includes a combination of programmable hardware and fixed-function hardware configured to perform inner product operations. The functional units within the calculation blocks 2724A to 2724N can also be configured to perform inner product operations, but the systolic array 2808 can be configured to perform a limited subset of inner product operations with a much higher throughput compared to the calculation blocks 2724A to 2724N.

[0324] Figures 29A - 29B illustrate the details of a hardware - based systolic array 2808 according to some embodiments. Figure 29A depicts a grid of multiple functional units configured to execute multiple inner - product operations within a single clock cycle. Figure 29B represents a single, exemplary functional unit. During systolic matrix calculations, as shown in Figure 29A, in one embodiment, the systolic array 2808 is configured to execute sets of parallel inner - product operations using various functional units. The inner product can be executed in a'systolic' fashion where SIMD data is pumped across multiple layers of functional units. The systolic array 2808 is a collection of functional units arranged in a grid. The grid of functional units operates in lockstep and is optimized to perform multiply - accumulate operations. The matrices acted upon by the systolic array 2808 are divided into sub - matrices that are pumped across the grid of functional units.

[0325] In one embodiment, the systolic array 2808 can process a configurable number of SIMD channels of data using a configurable systolic depth. For a given instruction, the SIMD width and systolic depth can be selected to process a set of source data. The systolic depth defines the number of systolic layers of hardware logic that will be used to process the instruction. A systolic layer is a group of multiplier and adder logic units with a variable SIMD width. A systolic layer can receive an initial accumulator value as input and generate an inner - product value that is output to a subsequent systolic layer or to an output register.

[0326] In some embodiments, three sources may be processed. At this time, each source can be a vector register or an immediate. In one embodiment, source 2900 (SRC0) can be one or more initial accumulator values and can be a single value or a vector of accumulator values. The initial accumulator values will be added to the first set of inner products calculated by each functional unit within the first systolic layer. The inner products calculated by the functional units can be supplied to the next systolic layer for a given SIMD channel. The inner products can be calculated based on source 2901 (SRC1) and source 2902 (SRC2), which are vector registers that can include one or more channels of packet data. Each channel includes a 4-element vector. In one embodiment, each channel is 32 bits wide and supplies four 8-bit vector elements. Some embodiments are configured to calculate inner products from input vectors having 8-bit elements, 4-bit elements, and / or 2-bit elements. In one embodiment, mixed-precision operations can be performed using any combination of supported element sizes (e.g., 8-bit × 2-bit, 8-bit × 4-bit, 4-bit × 4-bit, etc.). In one embodiment, systolic array 2808 is configured for integer calculations, but automatic fixed-point arithmetic is configurable in some embodiments. The instructions described in this application are for 4-element inner products, but in some embodiments, systolic array 2808 may also be configured to support floating-point inner product calculations with different numbers of elements per vector.

[0327] In one embodiment, the multiple channels of the 4-element vector can be packed into a single vector register of various widths (e.g., 64 bits, 128 bits, 256 bits, 512 bits, etc.). A systolic array 2808 can compute the simultaneous inner products for the multiple channels of vector elements supplied by sources 2901 and 2902. The number of channels of vector elements to be processed can be set based on the selected execution size and systolic depth for the inner product calculation. In one embodiment, a source vector wider than the specified execution size and / or systolic depth may be computed using multiple cycles of the systolic array 2808.

[0328] The number of calculations that can be performed within a given clock cycle can vary based on the number of SIMD lanes and systolic layers. As represented, the systolic array 2808 can execute 16 inner products per SIMD lane of throughput using a systolic depth of 4. When configured for 8 SIMD lanes, the logic can execute 128 8-bit integer (INT8) inner products within a given cycle. When configured for 8 SIMD lanes and 8 systolic depths, each lane can execute a total of 32 8-bit integer (INT8) inner products and 256 inner products. These specific numbers of operations are examples of one embodiment, and other embodiments will differ in throughput. Furthermore, when the data types are different, the number of operations will increase or decrease based on the different data types.

[0329] In each functional unit, the inner product is computed by multiplier and adder logic, and the inner product is added to the accumulator value. The resulting data can be output to a destination register or supplied to the accumulator of the next systolic layer. Details of the functional unit 2912 are shown in FIG. 29B.

[0330] As shown in FIG. 29B, the functional unit 2912 can include input data buffers 2904, 2906 and an accumulator 2922, each of which can receive input data. In one embodiment, the data buffer 2906 can receive a source 2902 (SRC2) that can be a packed vector of input data. The input data buffer 2904 can receive a source 2901 (SRC1) that can also be a packed vector of input data. The accumulator 2922 can receive a source 2900 (SRC0) that supplies an initial accumulator value for the functional unit 2912. The initial accumulator value is added to the inner product calculated from the elements of the sources 2901 and 2902. The inner product is calculated by element-wise multiplication of the elements of the source vectors using a set of multipliers 2923A - 2923D and an adder 2924. The multipliers 2923A - 2923D are used to calculate a set of products. The sum of the set of products is calculated by the adder 2924. The sum can be integrated (e.g., added) with any initial value supplied by the source 2900. In one embodiment, this integrated value can be supplied as an input value 2626 to the next accumulator that can be present in a subsequent systolic layer. In one embodiment, the source 2901 can include a plurality of channels of input data. Further channels of the source 2901 can be relayed as SRC1 input to further SIMD lanes 2928. In one embodiment, the source 2902 can include a plurality of channels of input data. Further channels of the source 2902 can be used as SIC2 input data to logic units within a further systolic depth. In one embodiment, the source 2900 can optionally include a plurality of channels, along with further channels supplied as input to accumulators within further functional units. In one embodiment, the source 2900 can be a single value added to each accumulator within each functional unit of the first systolic layer.

[0331] [Skipping Calculation Operations for Zero-Filled Matrices and Submatrices] One embodiment provides a technique for optimizing training and inference for systolic arrays when using sparse data. When a matrix or submatrix to be processed by systolic tensor array 2808 or systolic array 1912 is entirely zero, the dimension values of the matrix or submatrix can be set to zero, and systolic tensor array 2808 may bypass one or more calculation phases associated with the submatrix according to the operations to be executed. During preprocessing of matrix data, zero submatrices can be identified, and a submatrix map of the matrix can be generated to indicate which submatrices contain only zero values. In one embodiment, at least some operations may be bypassed for matrices or submatrices that include rows or columns consisting entirely of zero values. In one embodiment, submatrices that contain only one non-zero value can also be bypassed.

[0332] Figures 30A - 30B illustrate a system for bypassing zero-valued submatrices according to an embodiment. As shown in Figure 30B, matrices 3002 and 3004 are matrices in which one or more submatrices contain only zero values. Processing logic can generate a submatrix map 3012 of matrix 3002 and submatrix 3014 of matrix 3004 to indicate whether the submatrices contain only zero values. The submap can be generated using various techniques including performing a bitwise comparison with zero for each submatrix. The submatrix map can be generated by a framework or driver executed by general-purpose processing logic (e.g., a CPU), or by dedicated hardware logic within the processing resources. In one embodiment, when the number of non-zero values contained in submatrix 3005 is limited, the systolic operations for that submatrix may also be bypassed. Instead, the result of the matrix operation on submatrix 3005 can be calculated using an ALU instead of systolic tensor array 2808.

[0333] As shown in FIG. 30B, the memory 3020 can store the matrix 3002 and the matrix 3004. The systolic tensor array 2808 can include a matrix A load unit 3026, a matrix B load unit 3022, a matrix A feed unit 3028, and a matrix B feed unit 3024. The matrix 3002 can be loaded and supplied as the matrix B, while the matrix 3004 can be loaded and supplied as the matrix A. Sub-matrices of the matrix A and the matrix B can be loaded and supplied through the functional unit 3030 that operates as a processing element of the systolic tensor array 2808.

[0334] In one embodiment, the load B filter 3021 and the load A filter 3027 can include buffers that store a sub-matrix map 3012 for the matrix 3002 and a sub-matrix map 3014 for the matrix 3004. The load B filter 3021 can bypass the loading of zero-value sub-matrices by the matrix B load unit 3022. The load A filter 3027 can bypass the loading of zero-value sub-matrices by the matrix A load unit 3026. Sub-matrices that are not bypassed can be processed by the functional unit 3030. Depending on the operation executed by the systolic tensor array 2808, if one of the sub-matrices is zero, the entire operation can be bypassed. When a sub-matrix includes a single non-zero value matrix, the sub-matrix related to the operation to be executed can bypass the systolic tensor array 2808, and the operation can be executed by the ALU through the system 3110 as shown in FIG. 31B below.

[0335] Figures 31A-31B depict a method 3100 and a system 3100 for performing matrix multiplication operations on sparse data. FIG. 31A depicts a method 3100 for skipping matrix multiplication operations on a matrix filled with zeros and sub-matrices. FIG. 31B depicts a system 3110 in which operations on near-sparse matrices and sub-matrices can be bypassed. Method 3100 may be executed by a processing resource including a systolic array or a systolic tensor array described in the present application, and the processing resource further includes compression or encoding logic including a zero detection and / or zero skip circuit. Exemplary zero detection and / or zero skip circuits include, for example, the load B filter 3021 and the load A filter 3027 seen in FIG. 30. System 3110 includes components of the computing units described in the present application, such as, but not limited to, the computing unit 1910.

[0336] As shown in FIG. 31A, method 3100 includes a graphics processing unit, a graphics multiprocessor, or a graphics processor having a zero detection and / or zero skip circuit tracking zero values for operands of a matrix to be input to processing elements of a systolic array (3102). When the circuit detects a zero-filled matrix or zero-filled sub-matrix to be supplied as input (3104, YES), the circuit can bypass operations on the entire matrix or sub-matrix (3109). A zero-filled sub-matrix can include a block filled with zeros of an input matrix, or a row or column filled with zeros of the matrix. When the matrix or sub-matrix to be supplied as input is not filled with zeros (3104, NO), the circuit can proceed to perform operations on the matrix or sub-matrix, such as matrix multiplication operations or one or more vectorized calculations that would have a zero result based on zero value inputs (3106). The processing circuit may then bypass the calculation operations for each operand as necessary (3107). For example, even if the entire matrix or sub-matrix cannot be bypassed, the total number of operations can be reduced by skipping operations based on individual zero value operands. Further, nearly sparse matrices or sub-matrices may bypass the systolic array, and the limited number of required operations may be performed by a conventional logic unit without processing resources, as shown in FIG. 31B.

[0337] FIG. 31B depicts a system 3110 that bypasses a systolic tensor array for operations on a submatrix that contains a limited number of non-zero values. The system 3110 includes a memory 3020 and a computing unit 1910 as seen in FIG. 19. Although the computing unit 1910 is depicted, the system 3110 may alternatively or additionally include other types of computing units or processing resources described in this application. The computing unit 1910 includes a register file 1906, an ALU 1911, a systolic array 1912, and a mass unit 1913 optimized to execute a particular subset of mathematical operations. The techniques described with respect to the systolic array 1912 may also be applicable to the systolic tensor array 2808 of FIG. 28.

[0338] The elements of matrix 3002 and matrix 3004 stored in memory 3120 can be loaded into the register file 1906 of the computing unit 1910. For example, if a submatrix of matrix 3004 contains a limited number of non-zero values (e.g., submatrix 3005 of FIG. 30A), the logic within the systolic array 1912 can transmit a submatrix bypass message 3112 to the ALU 1911 that identifies the registers storing the operands of the bypassed operation and the bypassed operation to be executed. The ALU 1911 can then read the bypassed submatrix data 3114 from the register file 1906 and use vector processing logic to execute the bypassed operation. The processing of the bypassed operation can be executed in parallel with the non-bypassed operations executed by the systolic array 1912.

[0339] Using the above techniques, one of ordinary skill in the art may implement, for example, a graphics processor having processing resources including a tensor accelerator, where the tensor accelerator includes a load filter that bypasses the loading of sparse submatrices of an input tensor. The tensor accelerator includes a systolic tensor array that processes one or more submatrices of a set of input matrix data. The processing resources may further include a logic unit that executes operations on the bypassed submatrices of the input tensor, and the bypassed submatrices of the input tensor contain only zero values. In one embodiment, the bypassed submatrices contain a limited number of non-zero values.

[0340] Using the above techniques, one of ordinary skill in the art may also implement an apparatus having processing resources including a tensor accelerator and a decoder. The tensor accelerator includes a load filter that bypasses the loading of sparse submatrices of an input tensor. The decoder is configured to decode an encoded set of data related to the input tensor to generate a decoded set of the decoder. The decoder can decode the encoded set of data based on metadata related to the encoded set of data, and the load filter bypasses the loading of the sparse submatrices based on metadata related to the encoded set of data. In one embodiment, the decoder supplies metadata related to the encoded set of data to the load filter. The metadata related to the encoded set of data includes a validity map, and the validity map indicates zero or non-zero values for a bitstream of the decoded set of the data. In one embodiment, the decoder is included in the tensor accelerator, and the tensor accelerator reads one or more encoded submatrices as inputs.

[0341] [Transmission of Compressed Data to the EU] The GPU can include a data compression pipeline that enables compression of pixel data used by 3D and media applications. Data used by 3D and media applications is considered typed data from the perspective of GPU data compression. However, GPGPU compute data, such as data for machine learning and deep learning operations, is considered untyped data, and conventional GPGPU compression pipelines cannot compress such data using typed data compression techniques. For example, typed data is generally not sequentially stored in memory and may instead be stored using tiles, planes, or other data formats that are specific to the type of data to be compressed. In contrast, untyped compute data is sequentially stored in memory.

[0342] Embodiments described herein enable 3D compression blocks within a GPGPU to be adapted to enable compression of untyped compute data. In some embodiments, current graphics assets are utilized for compression of machine learning (ML) data using the current pipeline for 3D data.

[0343] For 3D data, there is state metadata related to the data that specifies how the data is laid out in memory. In some embodiments, in order to reuse 3D compression hardware for the compression of untyped data, an apparatus, system, or process is configured to detect the memory allocation of untyped data by software and generate state metadata about the allocation. The graphics driver associated with the GPGPU can be used to determine whether the data should be compressed, and the compressed status may be configured within the state metadata related to the allocation. The state metadata is used to indicate to the system that compression should be performed on the memory allocation related to untyped compute data. The state metadata can include one or more data structures that can be used to specify the parameters used for data compression. The state metadata may also include a pointer to the data, and may include elements such as a pointer to an auxiliary buffer that stores the compression metadata used when compressing the buffer, compression type, and untyped data.

[0344] In some embodiments, the data port enables the transmission of DL / ML data without format conversion. For the 3D data format, there is a conversion to the shader format for pixel data. However, different processes are used for machine learning and deep learning data, and there is no data format conversion. In some embodiments, the data port is modified to enable transmission without format conversion. In some embodiments, software will instead address DL / ML.

[0345] FIG. 32 is a block diagram of a graphics processing system 3200 that includes a GPGPU data compression pipeline. The data compression pipeline of the graphics processing system 3200 is configured to enable compression of 3D graphics and media data in addition to untyped compute data. The graphics processing system 3200 includes processing resources 3205 that include a set of shader cores 3210, 3212, 3214, a data port 3220, and a surface state cache 3222. The shader cores 3210, 3212, 3214 send memory read / write messages to the data port 3220 to access the memory subsystem 3234 of the graphics processing system 3200. Access to the memory subsystem 3234 is cached by a cache hierarchy 3230. Cached data within the cache hierarchy may be compressed by a GPGPU codec unit 3232 before the data is written to the memory subsystem 3234. The GPGPU codec unit 3232 can also decompress data read from the memory subsystem 3234 before the data is written to the cache hierarchy 3230.

[0346] The shader memory surface is related to the surface state that can be stored in the surface state metadata 3222. The surface state has information about the surface, such as per-pixel bits, tiling mode, clear pixel status, compression status, etc. This information is used by the GPGPU codec unit 3232 to compress data before sending it to the memory subsystem. For GPGPU programs executed on the processing resources 3205, such as deep learning and / or machine learning workloads, the data is usually "untyped" (i.e., no hardware data format is used). And the data is not tiled (i.e., linearly laid out in memory). During software memory allocation, the GPU driver applies heuristics to determine whether data compression is enabled for a buffer. If data compression is enabled, the driver allocates an auxiliary buffer to store the compression metadata and also allocates the surface state. In some embodiments, the compiler should ensure that all accesses from the kernel to the buffer are made with the appropriate surface state pointer. In some embodiments, for GPGPU applications, the surface state indicates the buffer memory layout (not tiled or structured in 2D / 3D). The data format for these surfaces depends on the data type of the surface. For example, for deep learning inference, the data type may be an 8-bit integer data type (e.g., INT8). For deep learning training, the format may be a 16-bit floating-point format (e.g., FP16, bfloat16).

[0347] In some embodiments, even if the compute data surface has a specified format, the data port 3220 does not perform any format conversion during access, because this is not required by those applications. Instead, the format information is only used to drive the compression / decompression algorithm heuristics. A compressed data "block" is a block of cache lines (usually two or four cache lines) that are compressed together by the compression unit. In one embodiment, for GPGPU compression, the cache line addresses within a block are always sequential. Deep learning programs typically have excellent sequential locality in access, which results in excellent performance while keeping the hardware simple.

[0348] Figures 33A-33B depict, in accordance with an embodiment, computing architectures 3300, 3320 configured to enable compressed transmission of neural network data to processing resources in a parallel compute processor or a general purpose graphics processing unit. Figure 33A depicts a computing architecture 3300 in which data decompression is performed within a compute block 3302. The compute block 3302 shown may represent one of the compute blocks 2724A-2724N seen in Figure 27. Figure 33B depicts a computing architecture 3320 in which data compression and decompression are performed within the processing resources.

[0349] As shown in FIG. 33A, the computing architecture 3300 includes a computing block 3302 and a hardware scratch buffer 3304 coupled to a memory 3308 via a DMA controller 3306. The memory 3308 can be the main memory or system memory of the data processing system described in the present application. The computing block 3302 includes a set of processing resources described in the present application and can be similar to any of the computing blocks 2724A through 2724N seen in FIG. 27. The scratch buffer 3304 can be a high-speed on-chip memory such as on-chip static random access memory (SRAM). In one embodiment, the scratch buffer 3304 is optimized to store feature block units or kernel block units for neural network operations executed by the computing block 3302.

[0350] In one embodiment, decoder 3312 can be hardware decoder logic incorporated into compute block 3302 to enable compressed transmission of neural network data across a computing architecture. For example, when processing a CNN, compute block 3302 can generate an output feature map (OFM) in an uncompressed format in scratch buffer 3304. Encoder 3316 is incorporated into DMA controller 3306 to enable writing of output feature map data to memory 3308 in a compressed format. When the OFM of one layer becomes the input feature map (IFM) of the next layer, those IFMs are read from memory 3306 as compressed data 3314 and stored in scratch buffer 3304. Decoder 3312 can enable compute block 3302 to read with the compressed data 3314 without the data needing to be decoded. Alternatively, a codec unit with both encoding and decoding logic can be incorporated into controller 3306 to enable the compressed data to be transmitted and read by DMA controller 3306. The feature map data can then be decompressed by DMA controller 3306 and written to scratch buffer 3304 in an uncompressed format read by compute block 3302.

[0351] In the embodiments described in the present application, the specific encoding format for the kernel and the feature data can be changed based on the statistical values of the data to be encoded. Analysis of neural network feature maps indicates that many feature maps can be highly sparse. Analysis of neural network kernel data indicates that while the kernel data is not as sparse as the feature maps, much of the data within the kernel data is repeated. The dynamic range of the kernel data is relatively low, indicating that raw data allocates more bits than necessary to store the coefficients. Using various encoding techniques, the feature maps and the kernel data can be compressed by up to about 80% in a reversible manner using the selection of various encoding techniques.

[0352] Neural network related data can be encoded (or compressed) using various encoding techniques such as, but not limited to, unique absolute value (UAV) table encoding, significance mapping (SM) encoding, table encoding (TE), unique value coordinate (UVC) encoding, and mean encoding (ME). The metadata of the encoded data indicates the type of encoding format used for the data. In one embodiment, a specific encoding format can be selected for a specific type of data such as kernel data or feature data. In one embodiment, statistical analysis is performed on the data prior to encoding to enable an appropriate encoder to be selected for each block of data.

[0353] In one embodiment, the data generated during SM encoding can be used to assist with submatrix bypass within the systolic tensor array. In the SM encoding mode, only the non-zero values within the block are encoded. The number of non-zero values within the sample block is indicated in the header, followed by a significance mapping indicating the map of non-zero values within the block. The non-zero values are then encoded in the order of their appearance within the stream.

[0354] As shown in FIG. 33B, the computing architecture 3320 can include logic for performing data compression and decompression within the processing resources. The computing architecture 3320 includes the memory 3308 and the DMA controller 3306 as seen in the computing architecture 3300 of FIG. 33A. However, the DMA controller 3306 may exclude the encoder 3316. Instead, the compressed data read from the memory 3308 via the DMA controller 3306 can be stored in the cache hierarchy (e.g., L3 cache 3326, L1 cache 3327) in the compressed format. The computing architecture 3320 includes a computing block 3322 having a plurality of processing resources 3328A to 3328N, and each processing resource can decode the compressed data from the cache hierarchy via a load operation and encode the compressed data written to the cache hierarchy via a store operation. The data may be decompressed by the codecs 3334A to 3334N before being supplied to the systolic arrays 3332A to 3332N within the processing resources 3328A to 3328N. The results generated by the systolic arrays 3332A to 3332N may be compressed by the codecs 3334A to 3334N before the data is written to the cache hierarchy and / or the memory 3308.

[0355] FIG. 34 represents a processing resource 3328 that includes zero detection logic for output data. The represented processing resource 3328 may be one of the processing resources 3328A-3328N of FIG. 33B. The processing resource 3328 can include an ALU 3434 that executes a processing operation based on source operands (e.g., SRC1 3432, SRC2 3433) stored in a register file of the processing resource 3328. The output of the operation executed by the ALU 3434 can be written to a temporary destination register 3436 within the register file of the processing resource 3328. The processing resource 3328 further includes zero detection logic 3438 that can execute a zero detection operation on the data in the temporary destination register 3436 while the data is flowing from the temporary destination register 3436 to a destination register 3440 within the register file. The zero detection logic 3438 can write zero detection metadata 3439 that indicates whether the value written to the destination register 3440 is a zero value. The zero detection logic 3438 may be a hardware circuit within the processing resource, or may be higher-level logic such as a shader program executed by a shader processor of the processing resource 3328. When the register is a vector register that includes a plurality of elements of packed data, the zero detection metadata 3439 may be a bit field that includes one bit for each vector element.

[0356] In one embodiment, the zero detection metadata 3439 may be used by a codec 3334 within the processing resource 3328 to compress destination data when data compression is enabled. The zero detection metadata 3439 may be used by the codec 3334 to generate metadata (e.g., surface state data stored in the surface state cache 3222 seen in FIG. 32, etc.) that is used to manage the compressed data when the compressed data is stored in the cache hierarchy and memory.

[0357] The various ALUs 3434 may be any one of the various processing elements described in the present application and may be configured to perform integer and / or floating-point operations including parallel integer and floating-point operations. The ALU 3434 may also represent the systolic arrays 3332A - 3332N seen in FIG. 33B.

[0358] FIG. 35 depicts a method 3500 for compressing and decompressing data for matrix operations within a processing resource. The method 3500 may be performed by a processing resource including compression and decompression logic, such as the processing resource 3328 seen in FIG. 34 or the processing resources 3328A - 3328N seen in FIG. 33B.

[0359] The method 3500 includes the processing resource reading compressed data including elements of a plurality of matrices from the cache hierarchy of the GPGPU (3502). The processing resource can decompress the compressed data and store the elements of the plurality of matrices in the register file of the processing resource (3504). The processing resource can then load the elements of the plurality of matrices into a systolic array within the processing resource (3506). For example, the processing resource may send a message to the systolic array along with the operations to be performed (e.g., opcodes) and their associated source and destination registers. The processing resource may then execute a specified matrix operation (e.g., the operation specified by the opcode) by the systolic array, and the systolic array may store the output of the matrix operation in the register file (3508).

[0360] For some operations, the processing resource may perform one or more additional operations on the output of the matrix operation by additional functional units of the processing resource (e.g., ALU, FPU) (3510). For example, some post-processing operations may be performed on the output generated by the systolic array, including performing a set of operations bypassed by the systolic array on a near-sparse matrix or sub-matrix.

[0361] The processing resource can then compress the output of the matrix operation or the output of one or more additional operations for output from the processing resource (3512). For example, the processing resource can output the compressed data and compression metadata to the cache hierarchy of the GPGPU.

[0362] FIG. 36 depicts a method 3600 of performing a zero detection operation on the output generated by a processing resource. Method 3600 may be performed by a processing resource that includes zero detection logic, such as processing resource 3328 seen in FIG. 34.

[0363] Method 3600 includes the processing resource reading data elements from the cache hierarchy of the GPGPU (3602). The data elements may be vector elements, packed data elements, or matrix elements for SIMD or SIMT operations to be performed by the processing resource. The processing resource can then perform one or more operations on the data elements by one or more functional units of the processing resource (3604). The one or more functional units may be integer or floating point functional units, mixed precision capable units, or matrix acceleration units. The matrix acceleration unit may be, for example, one of systolic array 1912, systolic tensor array 2808, or systolic arrays 3332A - 3332N described herein.

[0364] The processing resource can then perform a zero detection operation on the output of the one or more operations (3606). The zero detection operation is performed by zero detection logic within the processing resource. The zero detection logic may be a dedicated hardware circuit within the processing resource. The zero detection logic may also be shader logic executed by the processing resource. The zero detection operation may be performed while data is read from a temporary destination register and written to a destination register specified as an operand for the operation.

[0365] The processing resource may then write the output of one or more operations and zero detection metadata to the cache hierarchy of the GPGPU (3608). In one embodiment, the zero detection metadata may be used by a codec to compress data before writing the data to the cache hierarchy. The data may remain compressed within the cache hierarchy and memory of the GPGPU. If the compressed data needs to be read by the CPU, decompression logic may decompress the data during writing to CPU visible memory. In one embodiment, the decompression logic is used to write data to CPU visible memory and may be present within a DMA controller.

[0366] Using the above techniques, one of ordinary skill in the art may implement, for example, a general purpose graphics processor having a processing resource that includes a tensor accelerator that executes one or more matrix multiplication operations, and codec hardware that decompresses compressed data received by a load operation to be executed by the processing resource, where the compressed data includes elements of an input matrix identified by an operand of the one or more matrix multiplication operations. The general purpose graphics processor includes a register file, and the codec hardware can write elements of the input matrix to registers within the register file. The tensor accelerator can read elements of the input matrix from the register file. The processing resource can include the codec hardware and the register file.

[0367] Using the above techniques, one skilled in the art may implement, for example, in a general-purpose graphics processing unit (GPGPU), a method having: reading data elements from the GPGPU's cache hierarchy with the GPGPU's processing resources; performing one or more operations on the data elements by one or more functional units of the processing resources; performing a zero-detection operation on the output of the one or more operations by zero-detection logic within the processing resources; and writing the output of the one or more operations and zero-detection metadata to the GPGPU's cache hierarchy. The method may further have compressing the output of the one or more operations using the zero-detection metadata by a compressor of a codec unit, where the codec unit is included within the processing resources in the GPGPU.

[0368] [Additional example computing device] FIG. 37 is a block diagram of a computing device 3700 that includes a graphics processor 3704 according to an embodiment. The computing device 3700 can be a computing device that includes the functionality of each of the above-described embodiments. The computing device 3700 can be a communication device such as a set-top box (e.g., an Internet-based cable television set-top box, etc.), a global positioning system (GPS)-based device, etc., or can be included in such a device. The computing device 3700 can also be a mobile computing device such as a cellular phone, a smartphone, a personal digital assistant (PDA), a tablet computer, a laptop computer, an electronic reader, a smart television, a television platform, a wearable device (e.g., glasses, a watch, a bracelet, a smart card, a jewel, a clothing item, etc.), a media player, etc., or can be included in such a device. For example, in one embodiment, the computing device 3700 includes a mobile computing device that uses an integrated circuit (IC) such as a system-on-a-chip (SoC or SOC) that incorporates various hardware and / or software components of the computing device 3700 on a single chip.

[0369] The computing device 3700 includes a graphics processor 3704. The graphics processor 3704 represents any graphics processor described in the present application. The graphics processor includes one or more graphics engines, graphics processor cores, and other graphics execution resources described in the present application. Such graphics execution resources may be presented in the form of, but not limited to, execution units, shader engines, fragment processors, vertex processors, streaming multiprocessors, graphics processor clusters, or any set of computing resources suitable for processing graphics resources or image resources or performing general-purpose computing operations on heterogeneous processors.

[0370] In one embodiment, the graphics processor 3704 includes a cache 3714 that is either a single cache including, but not limited to, any number of L1, L2, L3, or L4 caches, render caches, depth caches, sampler caches, and / or shader unit caches, or a cache that can be divided into multiple segments of cache memory. The cache 3714 may have regions throughout the areas described in the present application. The cache 3714 may also include dynamic hash logic that supports dynamic reconfiguration of the memory bank hash algorithm. In some embodiments, the graphics processor 3704 includes a GPGPU engine 3744 that includes shared local memory (SLM 3734) in addition to a register file 3724 that includes registers used by the GPGPU engine 3744. The register file 3724 can include general-purpose registers, architecture registers, configuration registers, and other types of registers. The general-purpose register file (GRF) and the architecture register file (ARF) can also be present among the processing resources within one or more blocks of the computing units (e.g., compute 3750, compute 3755) within the GPGPU engine 3744. A shared fabric 3742 may also be present to enable rapid communication between the various components of the GPGPU engine 3744.

[0371] As shown, in one embodiment, in addition to the graphics processor 3704, the computing device 3700 may further include any number and type of hardware components and / or software components, including but not limited to an application processor 3706, a memory 3708, and an input / output (I / O) source 3710. The application processor 3706 can interact with a hardware graphics pipeline to share graphics pipeline functionality. The processed data is stored in a buffer within the hardware graphics pipeline, and the state information is stored in the memory 3708. The resulting data can be transferred to a display controller for output by a display device, which may be of various types such as a display device, a cathode ray tube (CRT), a thin film transistor (TFT), a liquid crystal display (LCD), an organic light emitting diode (OLED) array, etc., and may be configured to display information to a user via a graphical user interface.

[0372] The application processor 3706 can include one or more processors and may be a central processing unit (CPU) that is at least partially used to execute an operating system (OS) 3702 for the computing device 3700. The OS 3702 can serve as an interface between the hardware and / or physical resourc...

Claims

1. having processing resources including a matrix accelerator, a decoder, and hardware logic, wherein the matrix accelerator includes a plurality of processing elements arranged as a systolic array and configured to process one or more sub-matrices of a set of input matrix data, the matrix accelerator includes a load filter for bypassing the loading of a sparse sub-matrix of an input matrix, the decoder decodes an encoded set of data related to the input matrix to generate a decoded set of data, the decoder decodes the encoded set of data based on metadata related to the encoded set of data, the load filter bypasses the loading of the sparse sub-matrix based on the metadata related to the encoded set of data, the load filter is configured to bypass the loading of a nearly sparse sub-matrix having a limited number of non-zero values, the matrix accelerator is configured to send a message indicating the bypass of the nearly sparse sub-matrix to the hardware logic, the hardware logic is configured to execute operations on the bypassed nearly sparse sub-matrix of the input matrix, a general-purpose graphics processor.

2. the decoder supplies the metadata related to the encoded set of data to the load filter, the general-purpose graphics processor according to Claim 1.

3. the metadata related to the encoded set of data includes a validity map that indicates zero or non-zero values for a bitstream of the decoded set of data, the general-purpose graphics processor according to Claim 2.

4. the decoder is included in the matrix accelerator, and the matrix accelerator reads one or more encoded sub-matrices as inputs, the general-purpose graphics processor according to Claim 3.

5. A general-purpose graphics processing unit including a matrix accelerator including a plurality of processing elements arranged as a systolic array, the plurality of processing elements being configured to process one or more sub-matrices of a set of input matrix data, monitoring, by a zero detection circuit, zero-value operands of a matrix to be input to the matrix accelerator, Responding to the detection of a zero-value operand by bypassing the matrix multiplication operation in the matrix accelerator, where the zero-value operand is associated with a first sub-matrix of the input matrix, and the first sub-matrix is a zero-value sub-matrix, Responding to the detection of a second sub-matrix of the input matrix by bypassing the matrix multiplication operation in the matrix accelerator, where the second sub-matrix has a limited number of non-zero values, Sending a message indicating the bypass of the second sub-matrix to a processing resource outside the matrix accelerator so that the bypassed operation for the second sub-matrix is executed by the processing resource A method having.

6. The input matrix is a zero-value matrix, All matrix multiplication operations in the matrix accelerator are bypassed for the zero-value matrix, The method according to claim 5.

7. Receiving the message at the processing resource, Configuring the processing resource to execute the bypassed operation for the second sub-matrix The method according to claim 5, further comprising.

8. Further comprising executing, by the processing resource, the bypassed operation for the second sub-matrix based on the message, The method according to claim 7.

9. The message identifies the bypassed operation and the operand of the bypassed operation, The method according to claim 8.

10. Monitoring, by the zero detection circuit, the zero-value operands of the matrix to be input to the matrix accelerator includes generating a sub-matrix map of the first input matrix, The sub-matrix map identifies the zero-value sub-matrix of the first input matrix, The method according to claim 5.

11. A memory device, A general-purpose graphics processor coupled to the memory device Having, The general-purpose graphics processor has a processing resource including a matrix accelerator, a decoder, and hardware logic, The matrix accelerator includes a plurality of processing elements, which are arranged as a systolic array and configured to process one or more sub-matrices of a set of input matrix data, The matrix accelerator includes a load filter that bypasses the loading of sparse sub-matrices of the input matrix, The decoder decodes an encoded set of data related to the input matrix to generate a decoded set of data. The decoder decodes the encoded set of data based on metadata related to the encoded set of data. The load filter bypasses the load of the sparse submatrix based on the metadata related to the encoded set of data. The load filter bypasses the load of a nearly sparse submatrix having a limited number of non-zero values. The matrix accelerator sends a message indicating the bypass of the nearly sparse submatrix to the hardware logic. The hardware logic executes an operation on the bypassed nearly sparse submatrix of the input matrix. Data processing system.

12. The decoder supplies the metadata related to the encoded set of data to the load filter. The data processing system according to claim 11.

13. The metadata related to the encoded set of data includes a validity mapping that indicates zero or non-zero values for the bitstream of the decoded set of data. The data processing system according to claim 12.

14. The decoder is included in the matrix accelerator, and the matrix accelerator reads one or more encoded submatrices as inputs. The data processing system according to claim 13.

Images