Systolic separation within matrix accelerator architecture

Abstract

The embodiments described herein include software, firmware, and hardware logic that provide techniques for performing arithmetic on sparse data through a systolic processing unit. One embodiment provides techniques for optimizing training and inference on a systolic array when using sparse data. One embodiment provides techniques for using decompression information when performing sparse computing operations. One embodiment enables the separation of a special function computation array through a shared reg file. One embodiment enables packed data compression and extension operations on a GPGPU. One embodiment provides techniques for utilizing block sparsity within the cache hierarchy of a GPGPU.

Description

Technology Field

[0001] Cross-reference regarding related applications

[0002] This application relates to U.S. Provisional Application No. 62 / 819,337 (Attorney Case No. AC0271-Z), filed by Abhishek Appu et al. on March 15, 2019, under the title "GRAPHICS PROCESSING" pursuant to 35 USC 119(e); No. 62 / 819,435 (Attorney Case No. AC0285-Z), filed by Lakshminarayanan Striramassarma et al. on March 15, 2019, under the title "GRAPHICS DATA PROCESSING"; and No. 62 / 819,361 (Attorney Docket Number), filed by Subramaniam Maiyuran et al. on March 15, 2019, under the title "SYSTEMS AND METHODS FOR PARTITIONING CACHE TO REDUCE CACHE ACCESS LATENCY". AC0286-Z), related to No. 62 / 935,670 (Agent Case No. AC5197-Z) filed by Abhishek Appu et al. on March 15, 2019, under the title SYSTOLIC ARITHMETIC ON SPARSE DATA, claiming priority thereof, the full text of which is incorporated herein by reference.

[0003] The present disclosure generally relates to data processing, and more specifically to facilitating matrix operations through a general-purpose graphics processing unit. Background Technology

[0004] Current parallel graphics data processing includes systems and methods developed to perform specific operations on graphics data, such as linear interpolation, tessellation, rasterization, texture mapping, and depth testing. Typically, graphics processors processed graphics data using fixed-function computing units, but more recently, some graphics processors have become programmable, enabling them to support a wide range of operations for processing vertex and fragment data.

[0005] Graphics processors implement processing techniques, such as pipelining, which typically attempt to process as much graphics data as possible in parallel across various parts of the graphics pipeline to further enhance performance. Parallel graphics processors with a Single Instruction, Multi-Thread (SIMT) architecture are designed to maximize the amount of parallel processing in the graphics pipeline. In a SIMT architecture, groups of parallel threads attempt to execute program instructions synchronously together as often as possible to increase processing efficiency. A general overview of software and hardware regarding SIMT architecture can be found in Shane Cook, CUDA Programming Chapter 3 (2013), pp. 37–51. For example, prior art in this field is European Patent Application Publication EP 3407183. Brief explanation of the drawing

[0006] Accordingly, in order to enable a detailed understanding of the aforementioned features of the embodiments, a more specific description of the embodiments briefly summarized above may be provided with reference to the embodiments, some of which are illustrated in the accompanying drawings. However, note that the accompanying drawings illustrate only typical embodiments and are therefore not to be construed as limiting the scope thereof. FIG. 1 is a block diagram illustrating a computer system configured to implement one or more aspects of the embodiments described in this specification. FIGS. 2a to 2d illustrate parallel processor components. FIGS. 3a to 3c are block diagrams of a graphics multiprocessor and a multiprocessor-based GPU. FIGS. 4a through 4f illustrate an exemplary architecture in which a plurality of GPUs are connected to a plurality of multi-core processors for communication. Figure 5 illustrates a graphics processing pipeline. Figure 6 illustrates a machine learning software stack. Figure 7 illustrates a general-purpose graphics processing unit. Figure 8 illustrates a multi-GPU computing system. FIGS. 9a and 9b illustrate exemplary layers of a deep neural network. Figure 10 illustrates an exemplary recurrent neural network. Figure 11 illustrates the training and deployment of a deep neural network. Figure 12 is a block diagram illustrating distributed learning. Figure 13 illustrates an exemplary inference SOC (system on a chip) suitable for performing inference using a trained model. Figure 14 is a block diagram of a processing system. FIGS. 15a to 15c illustrate a computing system and a graphics processor. FIGS. 16a to 16c illustrate block diagrams of additional graphics processor and computing accelerator architectures. Figure 17 is a block diagram of the graphics processing engine of a graphics processor. FIGS. 18a and FIGS. 18b illustrate thread execution logic including an array of processing elements used in a graphics processor core. Fig. 19 illustrates an additional execution unit. Figure 20 is a block diagram illustrating a graphics processor instruction format. Figure 21 is a block diagram of an additional graphics processor architecture. FIGS. 22a and FIGS. 22b illustrate a graphics processor command format and a command sequence. FIG. 23 illustrates an exemplary graphics software architecture for a processing system. FIG. 24a is a block diagram illustrating an IP core development system. FIG. 24b shows a side cross-sectional view of an integrated circuit package assembly. FIG. 24c illustrates a package assembly comprising a plurality of units of hardware logic chiplets connected to a substrate. FIG. 24d illustrates a package assembly including interchangeable chiplets. FIG. 25 is a block diagram illustrating an exemplary system-on-chip integrated circuit. FIGS. 26a and FIGS. 26b are block diagrams showing an exemplary graphics processor for use within an SoC. FIG. 27 illustrates an additional execution unit according to one embodiment. FIG. 28 illustrates a matrix operation performed by an instruction pipeline according to one embodiment. FIGS. 29a and 29b illustrate details of a hardware-based systolic array according to some embodiments. FIG. 30 illustrates a systolic array including circuits that facilitate partial sum loopback and sparse matrix multiplication. FIGS. 31a to 31c illustrate techniques that facilitate sparse matrix multiplication by omitting operations for zero input values. FIG. 32 illustrates a processing resource including a separated systolic array. FIG. 33 illustrates data types for packed bytes, packed words, and packed double words (dword) for operands with a width of 128 bits. FIG. 34 illustrates a processing system including logic for executing packed data compression and expansion operations for a GPGPU. FIGS. 35a and FIGS. 35b illustrate GPGPU packed data compression and extension operations. Figures 36a and 36b illustrate a comparison between unstructured sparsity and block sparsity within training data for a neural network. FIG. 37 illustrates a processing system including a cache in which sparse data access can be bypassed. FIG. 38 is a block diagram of a graphics processing system including a GPGPU data compression pipeline. FIG. 39 is a block diagram of a computing device including a graphics processor according to one embodiment. Specific details for implementing the invention

[0007] A Graphics Processing Unit (GPU) is communicably connected to a host / processor core to facilitate, for example, graphics operations, machine learning operations, pattern analysis operations, and / or various general-purpose GPU (GPGPU) functions. The GPU may be communicably connected to the host processor / core via a bus or other interconnects (e.g., high-speed interconnects such as PCIe or NVLink). Alternatively, the GPU may be integrated on the same package or chip as the core, or may be communicably connected to the core via an internal processor bus / interconnect (i.e., inside the package or chip). Regardless of how the GPU is connected, the processor core may assign tasks to the GPU in the form of a sequence of commands / instructions contained in a work descriptor. The GPU then uses dedicated circuitry / logic to efficiently process these commands / instructions.

[0008] The embodiments described herein include software, firmware, and hardware logic that provide techniques for performing arithmetic on sparse data through a systolic processing unit. One embodiment provides techniques for optimizing training and inference on a systolic array when using sparse data. One embodiment provides techniques for using decompression information when performing sparse computing operations. One embodiment enables the disaggregation of special function computation arrays through a shared reg file. One embodiment enables packed data compression and extension operations on a GPGPU. One embodiment provides techniques for utilizing block sparsity within the cache hierarchy of a GPGPU.

[0009] In the following description, a number of specific details are presented for a more complete understanding. However, it will be apparent to those skilled in the art that the embodiments described herein may be practiced without one or more of these specific details. In other cases, well-known features have not been described to avoid obscuring the details of the embodiments.

[0010] System Overview

[0011] FIG. 1 is a block diagram illustrating a computer system (100) configured to implement one or more aspects of the embodiments described herein. The computing system (100) includes a processing subsystem (101) having a system memory (104) and one or more processor(s) (102) that communicate through an interconnection path that may include a memory hub (105). The memory hub (105) may be a separate component within a chipset component or may be integrated within one or more processor(s) (102). The memory hub (105) is connected to an I / O subsystem (111) through a communication link (106). The I / O subsystem (111) includes an I / O hub (107) that enables the computing system (100) to receive input from one or more input device(s) (108). Additionally, the I / O hub (107) may enable a display controller, which may be included in one or more processor(s) (102), to provide output to one or more display devices(s, 110A). In one embodiment, one or more display devices(s) (110A) connected to the I / O hub (107) may include local, internal, or built-in display devices.

[0012] A processing subsystem (101) includes one or more parallel processor(s) (112) connected to a memory hub (105) via, for example, a bus or other communication link (113). The communication link (113) may be any number of standard-based communication link technologies or protocols, such as (but not limited to) PCI Express, or may be a vendor-specific communication interface or communication fabric. One or more parallel processor(s) (112) may form a computationally focused parallel or vector processing system that may include multiple processing cores and / or processing clusters, such as many integrated core (MIC) processors. For example, one or more parallel processor(s) (112) may form a graphics processing subsystem capable of outputting pixels to one or more display device(s) (110A) connected via an I / O hub (107). One or more parallel processor(s) (112) may also include a display controller and a display interface (not shown) that enable direct connection to one or more display device(s) (110B).

[0013] Within the I / O subsystem (111), a system storage unit (114) may connect to an I / O hub (107) to provide a storage mechanism for the computing system (100). An I / O switch (116) may be used to provide an interface mechanism that enables connection between the I / O hub (107), other components such as a network adapter (118) and / or a wireless network adapter (119) that can be integrated within the platform, and various other devices that can be added via one or more add-in device(s) (120). The add-in device(s) (120) may also include, for example, one or more external graphics processor devices and / or computing accelerators. The network adapter (118) may be an Ethernet adapter or another wired network adapter. The wireless network adapter (119) may include one or more of Wi-Fi, Bluetooth, Near Field Communication (NFC), or other network devices including one or more wireless devices.

[0014] The computing system (100) may include other components not explicitly shown, such as USB or other port connections, optical storage drives, video capture devices, etc., and may also be connected to an I / O hub (107). A communication path interconnecting the various components of FIG. 1 may be implemented using any suitable protocol, such as a PCI (Peripheral Component Interconnect) based protocol (e.g., PCI-Express), any other bus or point-to-point communication interface and / or protocol(s) such as NVLink high-speed interconnect, or any interconnection protocol known in the art.

[0015] One or more parallel processor(s) (112) may include circuits optimized for graphics and video processing, for example, including video output circuits, and constitute a graphics processing unit (GPU). In addition to or instead of this, one or more parallel processor(s) (112) may include circuits optimized for general-purpose processing while maintaining the basic computing architecture described in more detail herein. Components of the computing system (100) may be integrated with one or more other system elements on a single integrated circuit. For example, one or more parallel processors (112), a memory hub (105), a processor (102), and an I / O hub (107) may be integrated into a system-on-chip (SoC) integrated circuit. Alternatively, components of the computing system (100) may be integrated into a single package to form a system-in-package (SIP) configuration. In one embodiment, at least some of the components of the computing system (100) may be integrated into a multi-chip module (MCM) that can be interconnected with other multi-chip modules to become a modular computing system.

[0016] It will be understood that the computing system (100) described herein is exemplary and subject to variation and modification. The connection topology, including the number and arrangement of bridges, the number of processor(s) (102), and the number of parallel processor(s) (112), can be modified as desired. For example, system memory (104) may be connected directly to the processor(s) (102) without passing through bridges, while other devices communicate with the system memory (104) through the memory hub (105) and the processor(s) (102). In other topologies, the parallel processor(s) (112) are connected to the I / O hub (107) or directly to one or more processors (102) rather than the memory hub (105). In other embodiments, the I / O hub (107) and the memory hub (105) may be integrated into a single chip. Additionally, a set of two or more processor(s) (102) can be attached through multiple sockets, and these sockets can be connected to two or more instances of parallel processor(s) (112).

[0017] Some of the specific components described in this specification are optional and may not be included in all implementations of the computing system (100). For example, any number of add-in cards or peripherals may be supported, and some components may be omitted. Additionally, some architectures may use different terminology for components similar to those exemplified in FIG. 1. For example, in some architectures, the memory hub (105) may be referred to as the Northbridge, while the I / O hub (107) may be referred to as the Southbridge.

[0018] FIG. 2a illustrates a parallel processor (200). The parallel processor (200) may be a GPU, GPGPU, etc., as described herein. Various components of the parallel processor (200) may be implemented using one or more integrated circuit devices, such as a programmable processor, an application-specific integrated circuit (ASIC), or a field-programmable gate array (FPGA). The illustrated parallel processor (200) may be one of the parallel processor(s) (112) illustrated in FIG. 1.

[0019] A parallel processor (200) includes a parallel processing unit (202). The parallel processing unit includes an I / O unit (204) that enables communication with other devices, including other instances of the parallel processing unit (202). The I / O unit (204) may be directly connected to other devices. For example, the I / O unit (204) connects to other devices through the use of a hub or switch interface, such as a memory hub (105). The connection between the memory hub (105) and the I / O unit (204) forms a communication link (113). Within the parallel processing unit (202), the I / O unit (204) connects to a host interface (206) and a memory crossbar (216), wherein the host interface (206) receives a command instructed to perform a processing operation, and the memory crossbar (216) receives a command instructed to perform a memory operation.

[0020] When the host interface (206) receives a command buffer through the I / O unit (204), the host interface (206) may instruct the front end (208) to perform a task operation to execute these commands. In one embodiment, the front end (208) is connected to a scheduler (210) configured to distribute commands or other task items to a processing cluster array (212). The scheduler (210) ensures that the cluster array (212) is properly configured and in an active state before processing tasks are distributed to the processing clusters of the processing cluster array (212). The scheduler (210) may be implemented through firmware logic running on a microcontroller. The scheduler (210) implemented on the microcontroller may be configured to perform complex scheduling and task distribution operations at coarse and fine granularity, enabling rapid preemption and context switching of threads running on the processing array (212). Preferably, the host software can inspect the workload to be scheduled on the processing array (212) through one of the graphics processing doorbells. Then, the workloads can be automatically distributed across the processing array (212) by the scheduler (210) logic within the scheduler microcontroller.

[0021] The processing cluster array (212) may include up to "N" processing clusters (e.g., cluster (214A), cluster (214B) through cluster (214N)). Each cluster (214A-214N) of the processing cluster array (212) may execute multiple concurrent threads. The scheduler (210) may assign tasks to the clusters (214A-214N) of the processing cluster array (212) using various scheduling and / or task distribution algorithms, which may vary depending on the workload occurring for each type of program or computation. Scheduling may be handled dynamically by the scheduler (210) or may be partially supported by compiler logic during the compilation of program logic configured for execution by the processing cluster array (212). Optionally, different clusters (214A-214N) of the processing cluster array (212) may be assigned to process different types of programs or perform different types of computations.

[0022] The processing cluster array (212) may be configured to perform various types of parallel processing operations. For example, the cluster array (212) may be configured to perform general-purpose parallel computing operations. For example, the processing cluster array (212) may include logic for executing processing tasks, including filtering video and / or audio data, performing modeling operations including physical operations, and performing data conversion.

[0023] The processing cluster array (212) is configured to perform parallel graphics processing operations. In this embodiment, where the parallel processor (200) is configured to perform graphics processing operations, the processing cluster array (212) may include additional logic to support the execution of such graphics processing operations, including but not limited to texture sampling logic for performing texture operations, as well as tessellation logic and other vertex processing logic. Additionally, the processing cluster array (212) may be configured to execute graphics processing-related shader programs, such as vertex shaders, tessellation shaders, geometry shaders, and pixel shaders, but not limited to. The parallel processing unit (202) may transfer data from system memory through an I / O device (204) for processing. During processing, the transferred data may be stored in on-chip memory (e.g., parallel processor memory (222)) and then written back to system memory.

[0024] In an embodiment where a parallel processing unit (202) is used to perform graphics processing, the scheduler (210) may be configured to divide the processing workload into tasks of approximately equal size to better distribute graphics processing operations to a number of clusters (214A-214N) of the processing cluster array (212). In some of these embodiments, parts of the processing cluster array (212) may be configured to perform different types of processing. For example, a first part may be configured to perform vertex shading and topology generation, a second part may be configured to perform tessellation and geometry shading, and a third part may be configured to perform pixel shading or other screen space operations to generate an image rendered for display. Intermediate data generated by one or more of the clusters (214A-214N) may be stored in a buffer so that the intermediate data can be transferred between the clusters (214A-214N) for further processing.

[0025] During operation, the processing cluster array (212) may receive a processing task to be executed via a scheduler (210) that receives a command defining the processing task from the front end (208). For a graphics processing operation, the processing task may include an index of the data to be processed, e.g., surface (patch) data, primitive data, vertex data, and / or pixel data, as well as state parameters and commands defining how the data will be processed (e.g., which program will be executed). The scheduler (210) may be configured to fetch an index corresponding to the task, or may receive an index from the front end (208). The front end (208) may be configured to ensure that the processing cluster array (212) is configured to be in an active state before a workload specified by an incoming command buffer (e.g., batch buffer, push buffer, etc.) is initiated.

[0026] Each of one or more instances of a parallel processing unit (202) may be connected to a parallel processor memory (222). The parallel processor memory (222) may be accessed via a memory crossbar (216) capable of receiving memory requests from the processing cluster array (212) as well as the I / O unit (204). The memory crossbar (216) may access the parallel processor memory (222) via a memory interface (218). The memory interface (218) may include a plurality of partition units (e.g., partition unit (220A) and partition unit (220B) to partition unit (220N)) which may be connected to each part (e.g., memory unit) of the parallel processor memory (222). The number of partition units (220A-220N) may be configured to be equal to the number of memory units, so that the first partition unit (220A) has a corresponding first memory unit (224A), the second partition unit (220B) has a corresponding memory unit (224B), and the Nth partition unit (220N) has a corresponding Nth memory unit (224N). In other embodiments, the number of partition units (220A-220N) may not be equal to the number of memory devices.

[0027] The memory units (224A-224N) may include various types of memory devices, such as dynamic random access memory (DRAM) or graphics random access memory (SGRAM), including graphics double data rate (GDDR) memory. Optionally, the memory units (224A-224N) may also include 3D stacked memory, including but not limited to high bandwidth memory (HBM). Those skilled in the art will understand that the specific implementation of the memory units (224A-224N) may vary and may be selected from one of various conventional designs. Rendering targets, such as frame buffers or texture maps, may be stored across the memory units (224A-224N), allowing the partition units (220A-220N) to write portions of each render target in parallel to efficiently use the available bandwidth of the parallel processor memory (222). In some embodiments, a local instance of parallel processor memory (222) may be excluded for an integrated memory design that utilizes system memory together with local cache memory.

[0028] Optionally, any of the clusters (214A-214N) of the processing cluster array (212) has the ability to process data to be written to any of the memory units (224A-224N) within the parallel processor memory (222). The memory crossbar (216) may be configured to transfer the output of each cluster (214A-214N) to any partition unit (220A-220N) or to another cluster (214A-214N) capable of performing additional processing operations on the output. Each cluster (214A-214N) may communicate with the memory interface (218) through the memory crossbar (216) to read from or write to various external memory devices. In one of the embodiments having a memory crossbar (216), the memory crossbar (216) has a connection to a memory interface (218) for communicating with an I / O unit (204), as well as a connection to a local instance of parallel processor memory (222), thereby enabling a processing unit within a different processing cluster (214A-214N) to communicate with a memory other than the system memory or parallel processing unit (202). Generally, the memory crossbar (216) may use a virtual channel to separate traffic streams between, for example, clusters (214A-214N) and partition units (220A-220N).

[0029] Although a single instance of a parallel processing unit (202) is depicted within the parallel processor (200), any number of instances of the parallel processing unit (202) may be included. For example, multiple instances of the parallel processing unit (202) may be provided on a single add-in card, or multiple add-in cards may be interconnected. Different instances of the parallel processing unit (202) may be configured to interact with each other, even if these different instances have different numbers of processing cores, different amounts of local parallel processor memory, and / or other configuration differences. Optionally, some instances of the parallel processing unit (202) may include higher precision floating-point units than other instances. A system comprising one or more instances of the parallel processing unit (202) or the parallel processor (200) may be implemented in various configurations and form factors, including but not limited to desktops, laptops, or handheld personal computers, servers, workstations, game consoles, and / or embedded systems.

[0030] FIG. 2b is a block diagram of a partition unit (220). The partition unit (220) may be an instance of one of the partition units (220A-220N) of FIG. 2a. As illustrated, the partition unit (220) includes an L2 cache (221), a frame buffer interface (225), and a raster operations unit (ROP) (226). The L2 cache (221) is a read / write cache configured to perform load and save operations received from the memory crossbar (216) and the ROP (226). Read misses and urgent write-back requests are output from the L2 cache (221) to the frame buffer interface (225) for processing. Updates may also be transmitted to the frame buffer via the frame buffer interface (225) for processing. In one embodiment, the frame buffer interface (225) interfaces with one of the memory units of the parallel processor memory (e.g., within the parallel processor memory (222)) such as the memory units (224A-224N) of FIG. 2a. In addition to, or instead of, the partition unit (220) may interface with one of the memory units within the parallel processor memory through a memory controller (not shown).

[0031] In a graphics application, the ROP (226) is a processing unit that performs raster operations such as stenciling, z-testing, and blending. The ROP (226) then outputs the processed graphics data, which is stored in graphics memory. In some embodiments, the ROP (226) includes compression logic that compresses depth or color data written to memory and decompresses depth or color data read from memory. The compression logic may be lossless compression logic using one or more multiple compression algorithms. The type of compression performed by the ROP (226) may vary based on the statistical characteristics of the data to be compressed. For example, in one embodiment, delta color compression is performed on depth and color data on a tile basis.

[0032] The ROP (226) may be included in each processing cluster (e.g., clusters (214A-214N) of FIG. 2a) instead of the partition unit (220). In this embodiment, read and write requests for pixel data are transmitted via the memory crossbar (216) instead of the pixel fragment data. The processed graphic data may be displayed on a display device such as one of the one or more display device(s) (110) of FIG. 1, routed for further processing by the processor(s) (102), or routed for further processing by one of the processing entities within the parallel processor (200) of FIG. 2a.

[0033] FIG. 2c is a block diagram of a processing cluster (214) within a parallel processing unit. For example, the processing cluster is an instance of one of the processing clusters (214A-214N) of FIG. 2a. The processing cluster (214) may be configured to execute multiple threads in parallel, where the term "thread" refers to an instance of a specific program executed on a specific set of input data. Optionally, SIMD (single-instruction, multiple-data) instruction issuance technology may be used to support the parallel execution of multiple threads without providing multiple independent instruction units. Alternatively, SIMT (single-instruction, multiple-thread) technology may be used to support the parallel execution of multiple generally synchronized threads by using a common instruction unit configured to issue instructions to a set of processing engines within each cluster of the processing cluster. Unlike SIMD execution systems where all processing engines typically execute the same instructions, SIMT execution allows different threads to more easily follow different execution paths through a given thread program. Those skilled in the art will understand that a SIMD processing system represents a functional subset of a SIMT processing system.

[0034] The operation of the processing cluster (214) can be controlled through a pipeline manager (232) that distributes processing tasks to SIMT parallel processors. The pipeline manager (232) receives instructions from the scheduler (210) of FIG. 2a and manages the execution of these instructions through the graphics multiprocessor (234) and / or texture unit (236). The illustrated graphics multiprocessor (234) is an exemplary instance of a SIMT parallel processor. However, various types of SIMT parallel processors of different architectures may be included within the processing cluster (214). One or more instances of the graphics multiprocessor (234) may be included within the processing cluster (214). The graphics multiprocessor (234) can process data, and the data crossbar (240) can be used to distribute the processed data to one of a number of possible destinations, including other shader units. The pipeline manager (232) can facilitate the distribution of processed data by specifying a destination for the processed data to be distributed through the data crossbar (240).

[0035] Each graphics multiprocessor (234) within the processing cluster (214) may include the same set of functional execution logic (e.g., arithmetic logic unit, load / store unit, etc.). The functional execution logic may be configured in a pipelined manner so that new instructions can be issued before previous instructions are completed. The functional execution logic supports various operations including integer and floating-point arithmetic, comparison operations, Boolean operations, bit shifting, and calculation of various algebraic functions. The same functional unit hardware may be used to perform different operations, and any combination of functional units may exist.

[0036] Commands sent to the processing cluster (214) constitute threads. A set of threads executed across a set of parallel processing engines is a thread group. A thread group executes the same program on different input data. Each thread within a thread group may be assigned to a different processing engine within the graphics multiprocessor (234). A thread group may contain fewer threads than the number of processing engines within the graphics multiprocessor (234). If a thread group contains fewer threads than the number of processing engines, one or more of the processing engines may be idle during the cycle in which the thread group is being processed. A thread group may contain more threads than the number of processing engines within the graphics multiprocessor (234). If a thread group contains more threads than the number of processing engines within the graphics multiprocessor (234), processing may be performed over consecutive clock cycles. Optionally, multiple thread groups may be executed simultaneously on the graphics multiprocessor (234).

[0037] The graphics multiprocessor (234) can perform load and save operations including internal cache memory. Optionally, the graphics multiprocessor (234) may forgo the internal cache and use cache memory (e.g., L1 cache (248)) within the processing cluster (214). Each graphics multiprocessor (234) also has access to an L2 cache within a partition unit (e.g., partition unit (220A-220N) of FIG. 2a) shared among all processing clusters (214) and may be used to transfer data between threads. The graphics multiprocessor (234) may also access off-chip global memory, which may include one or more of local parallel processor memory and / or system memory. Any memory outside the parallel processing unit (202) may be used as global memory. An embodiment in which the processing cluster (214) includes multiple instances of the graphics multiprocessor (234) can share common instructions and data that can be stored in the L1 cache (248).

[0038] Each processing cluster (214) may include an MMU (245) (memory management unit) configured to map virtual addresses to physical addresses. In other embodiments, one or more instances of the MMU (245) may reside within the memory interface (218) of FIG. 2A. The MMU (245) includes a set of page table entries (PTEs) used to map virtual addresses to the physical addresses of tiles and optionally to cache line indices. The MMU (245) may include an address translation index buffer (TLB) or cache that may reside within the graphics multiprocessor (234), L1 cache, or processing cluster (214). Physical addresses are processed to distribute surface data access locality to allow efficient request interleaving between partition units. Cache line indices may be used to determine whether a request for a cache line is a hit or a miss.

[0039] In a graphics and computing application, the processing cluster (214) may be configured such that each graphics multiprocessor (234) is connected to a texture unit (236) to perform texture mapping operations, for example, to determine texture sample locations, read texture data, and filter texture data. Texture data is read from an internal texture L1 cache (not shown) or, in some embodiments, from an L1 cache within the graphics multiprocessor (234), and is fetched from an L2 cache, local parallel processor memory, or system memory as needed. Each graphics multiprocessor (234) outputs the processed task to a data crossbar (240) to provide the processed task to another processing cluster (214) for further processing, or stores the processed task in an L2 cache, local parallel processor memory, or system memory via a memory crossbar (216). The preROP (pre-raster operations unit) (242) is configured to receive data from the graphics multiprocessor (234) and send the data to a ROP unit that may be located together with a partition unit as described in this specification (e.g., the partition unit (220A-220N) of FIG. 2A). The preROP (242) unit may perform optimization for color blending, organize pixel color data, and perform address translation.

[0040] It will be understood that the core architecture described herein is exemplary and variations and modifications are possible. Any number of processing units, such as a graphics multiprocessor (234), a texture unit (236), a preROP (242), etc., may be included within the processing cluster (214). Additionally, although only one processing cluster (214) is illustrated, parallel processing units as described herein may include any number of instances of the processing cluster (214). Optionally, each processing cluster (214) may be configured to operate independently of other processing clusters (214) using separate, distinct processing units, an L1 cache, etc.

[0041] FIG. 2d illustrates an example of a graphics multiprocessor (234) connected to a pipeline manager (232) of a processing cluster (214). The graphics multiprocessor (234) has an execution pipeline including, but not limited to, an instruction cache (252), an instruction unit (254), an address mapping unit (256), a register file (258), one or more general-purpose graphics processing unit (GPGPU) cores (262), and one or more load / store units (266). The GPGPU cores (262) and the load / store units (266) are connected to a cache memory (272) and a shared memory (270) via a memory and cache interconnect (268). The graphics multiprocessor (234) may further include a tensor and / or ray-tracing core (263) containing hardware logic to facilitate matrix and / or ray-tracing operations.

[0042] The instruction cache (252) can receive a stream of instructions to be executed from the pipeline manager (232). Instructions are cached in the instruction cache (252) and dispatched for execution by the instruction unit (254). The instruction unit (254) can dispatch instructions as a thread group (e.g., warp), and each thread of the thread group is assigned to another execution unit within the GPGPU core (262). Instructions can access any space among the local, shared, or global address space by specifying an address within the integrated address space. The address mapping unit (256) can be used to translate addresses in the integrated address space into separate memory addresses that can be accessed by the load / store unit (266).

[0043] The register file (258) provides a set of registers to the functional units of the graphics multiprocessor (234). The register file (258) provides temporary storage for operands connected to the data paths of the functional units of the graphics multiprocessor (234) (e.g., GPGPU core (262), load / store unit (266)). The register file (258) may be divided among the functional units so that a dedicated portion of the register file (258) is allocated to each functional unit. For example, the register file (258) may be divided among several warps executed by the graphics multiprocessor (234).

[0044] Each of the GPGPU cores (262) may include a floating-point unit (FPU) and / or an integer arithmetic logic unit (ALU) used to execute instructions of the graphics multiprocessor (234). In some embodiments, the GPGPU core (262) may include hardware logic, which may reside within the tensor and / or ray tracing core (263) in other cases. The GPGPU cores (262) may have similar or different structures. For example, in one embodiment, a first portion of the GPGPU cores (262) includes a single-precision FPU and an integer ALU, while a second portion of the GPGPU cores includes a double-precision FPU. Optionally, the FPU may implement the IEEE 754-2008 standard for floating-point arithmetic or enable variable-precision floating-point arithmetic. The graphics multiprocessor (234) may additionally include one or more fixed-function or special-function units to perform specific functions such as rectangular copying or pixel blending operations. One or more GPGPU cores may also include fixed or special-function logic.

[0045] GPGPU cores (262) may include SIMD logic capable of executing a single instruction on multiple data sets. Optionally, the GPGPU core (262) may physically execute SIMD4, SIMD8, and SIMD16 instructions and logically execute SIMD1, SIMD2, and SIMD32 instructions. SIMD instructions for the GPGPU core may be generated at compile time by a shader compiler or may be automatically generated when executing a program written and compiled for a single program multiple data (SPMD) or SIMT architecture. Multiple threads of a program configured for a SIMT execution model may be executed through a single SIMD instruction. For example, in one embodiment, eight SIMT threads performing the same or similar operations may be executed in parallel through a single SIMD8 logic unit.

[0046] The memory and cache interconnection (268) is an interconnection network that connects each of the function units of the graphics multiprocessor (234) to the register file (258) and the shared memory (270). For example, the memory and cache interconnection (268) is a crossbar interconnection that allows the load / store unit (266) to implement load and store operations between the shared memory (270) and the register file (258). The register file (258) can operate at the same frequency as the GPGPU core (262), and thus data transfer between the GPGPU core (262) and the register file (258) has very low latency. The shared memory (270) can be used to enable communication between threads running on the function units within the graphics multiprocessor (234). The cache memory (272) can be used as a data cache to cache texture data communicated between the function unit and the texture unit (236), for example. The shared memory (270) can also be used as a program management cache. A thread running on the GPGPU core (262) can programmatically store data in the shared memory in addition to automatically cached data stored in the cache memory (272).

[0047] FIGS. 3a through 3c illustrate additional graphics multiprocessors according to embodiments. FIGS. 3a and 3b illustrate graphics multiprocessors (325, 350) associated with the graphics multiprocessor (234) of FIG. 2c and which may be used instead of one of them. Accordingly, the disclosure of any feature combined with the graphics multiprocessor (234) in this specification also indicates a corresponding combination with the graphics multiprocessor(s) (325, 350), but is not limited thereto. FIG. 3c illustrates a graphics processing unit (GPU) (380) comprising a dedicated set of graphics processing resources arranged in multi-core groups (365A-365N) corresponding to the graphics multiprocessors (325, 350). The illustrated graphics multiprocessor (325, 350) and multicore group (365A-365N) may be a streaming multiprocessor (SM) capable of simultaneous execution of multiple execution threads.

[0048] The graphics multiprocessor (325) of FIG. 3a includes a plurality of additional instances of execution resource units for the graphics multiprocessor (234) of FIG. 2d. For example, the graphics multiprocessor (325) may include a plurality of instances of instruction units (332A-332B), register files (334A-334B), and texture unit(s) (344A-344B). The graphics multiprocessor (325) also includes a plurality of sets of graphics or compute execution units (e.g., GPGPU cores (336A-336B), tensor cores 337A-337B, ray-tracing cores (338A-338B)) and a plurality of sets of load / store units (340A-340B). The execution resource unit has a common instruction cache (330), texture and / or data cache memory (342), and shared memory (346).

[0049] Various components can communicate through an interconnection fabric (327). The interconnection fabric (327) may include one or more crossbar switches that enable communication between various components of the graphics multiprocessor (325). The interconnection fabric (327) may be a separate high-speed network fabric layer on which each component of the graphics multiprocessor (325) is stacked. Components of the graphics multiprocessor (325) communicate with remote components through the interconnection fabric (327). For example, GPGPU cores (336A-336B, 337A-337B, and 3378A-338B) can each communicate with shared memory (346) through the interconnection fabric (327). The interconnection fabric (327) may mediate communication within the graphics multiprocessor (325) to ensure fair bandwidth allocation between components.

[0050] The graphics multiprocessor (350) of FIG. 3b includes a plurality of execution resource sets (356A-356D), each of which includes a plurality of instruction units, register files, GPGPU cores, and load storage units as illustrated in FIG. 2d and FIG. 3a. The execution resources (356A-356D) may work in cooperation with texture unit(s) (360A-360D) for texture operations while sharing an instruction cache (354) and shared memory (353). For example, the execution resources (356A-356D) may share the instruction cache (354) and shared memory (353) as well as multiple instances of texture and / or data cache memory (358A-358B). Various components may communicate through an interconnection fabric (352) similar to the interconnection fabric (327) of FIG. 3a.

[0051] Those skilled in the art will understand that the architecture described in FIGS. 1, FIGS. 2a through 2d, and FIGS. 3a-3b is illustrative and not limiting to the scope of the embodiments described herein. Accordingly, the technologies described herein may be implemented in any appropriately configured processing unit, without departing from the scope of the embodiments described herein, including, without limitation, one or more mobile application processors, one or more desktop or server central processing units (CPUs) including multi-core CPUs, one or more parallel processing units such as the parallel processing unit (202) of FIG. 2a, and one or more graphics processors or special purpose processing units.

[0052] A parallel processor or GPGPU as described herein may be communicably connected to a host / processor core to accelerate graphics operations, machine-learning operations, pattern analysis operations, and various general-purpose GPU (GPGPU) functions. The GPU may be communicably connected to the host processor / core via a bus or other interconnect (e.g., high-speed interconnects such as PCIe or NVLink). In other embodiments, the GPU may be integrated on the same package or chip as the core, or may be communicably connected to the core via an internal processor bus / interconnect (i.e., inside the package or chip). Regardless of how the GPU is connected, the processor core may assign tasks to the GPU in the form of a sequence of commands / instructions contained in a work descriptor. The GPU then uses dedicated circuitry / logic to efficiently process these commands / instructions.

[0053] FIG. 3c illustrates a graphics processing unit (GPU) (380) comprising a dedicated set of graphics processing resources arranged in multi-core groups (365A-365N). Although details of a single multi-core group (365A) are provided, it will be understood that other multi-core groups (365B-365N) may have the same or similar set of graphics processing resources. The details described for the multi-core groups (365A-365N) may also apply to any graphics multiprocessor (234, 325, 350) described herein.

[0054] As illustrated, the multi-core group (365A) may include a set of graphics cores (370), a set of tensor cores (371), and a set of ray tracing cores (372). A scheduler / dispatcher (368) schedules and dispatches graphics threads for execution on the various cores (370, 371, 372). A register file set (369) stores operand values ​​used by the cores (370, 371, 372) when executing the graphics threads. These may include, for example, integer registers for storing integer values, floating-point registers for storing floating-point values, vector registers for storing packed data elements (integer and / or floating-point data elements), and tile registers for storing tensor / matrix values. The tile registers are implemented as a combined set of vector registers.

[0055] One or more combined Level 1 (L1) cache and shared memory units (373) locally store graphic data such as texture data, vertex data, pixel data, ray data, boundary volume data, etc. within each multi-core group (365A). One or more texture units (374) may also be used to perform texturing operations such as texture mapping and sampling. A Level 2 (L2) cache (375) shared by all or part of the multi-core groups (365A-365N) stores graphic data and / or instructions for multiple concurrent graphic threads. As illustrated, the L2 cache (375) may be shared across multiple multi-core groups (365A-365N). One or more memory controllers (367) connect the GPU (380) to memory (366), which may be system memory (e.g., DRAM) and / or dedicated graphic memory (e.g., GDDR6 memory).

[0056] The input / output (I / O) circuit (363) connects the GPU (380) to one or more I / O devices (362), such as a digital signal processor (DSP), a network controller, or a user input device. On-chip interconnects may be used to connect the I / O devices (362) to the GPU (380) and memory (366). One or more I / O memory management units (IOMMUs) (364) of the I / O circuit (363) directly connect the I / O devices (362) to system memory (366). Optionally, the IOMMUs (364) manage multiple sets of page tables to map virtual addresses to physical addresses in system memory (366). Then, the I / O devices (362), CPU(s) (361), and GPU(s) (380) can share the same virtual address space.

[0057] In one embodiment of the IOMMU (364), the IOMMU (364) supports virtualization. In this case, it may manage a first set of page tables to map guest / graphics virtual addresses to guest / graphics physical addresses, and a second set of page tables to map guest / graphics physical addresses to system / host physical addresses (in system memory (366)). The base addresses of each of the first and second sets of page tables may be stored in control registers and may be swapped out on a context switch (e.g., a new context may access the associated set of page tables). Although not shown in FIG. 3c, each core (370, 371, 372) and / or multi-core group (365A-365N) may include a translation lookaside buffer (TLB) for caching translations from guest virtual to guest physical, from guest physical to host physical, and from guest virtual to host physical.

[0058] The CPU (361), GPU (380), and I / O device (362) may be integrated on a single semiconductor chip and / or chip package. The illustrated memory (366) may be integrated on the same chip or connected to a memory controller (367) via an off-chip interface. In one embodiment, the memory (366) includes GDDR6 memory that shares the same virtual address space as other physical system-level memory, but the basic principles described herein are not limited to this particular embodiment.

[0059] The tensor core (371) includes multiple execution units specifically designed to perform matrix operations, which are the basic computing operations used to perform deep learning operations. For example, simultaneous matrix multiplication operations can be used for neural network training and inference. The tensor core (371) can perform matrix processing using various operand precisions, including single-precision floating-point (e.g., 32 bits), half-precision floating-point (e.g., 16 bits), integer word (16 bits), byte (8 bits), and half-byte (4 bits). For example, a neural network implementation extracts features from each rendered scene and potentially combines details from multiple frames to construct a high-quality final image.

[0060] In a deep learning implementation example, parallel matrix multiplication operations can be scheduled to be executed on a Tensor Core (371). In particular, training of a neural network requires a significant number of matrix inner product operations. The Tensor Core (371) may include at least N inner product processing elements to process the inner product formula of an N×N×N matrix. Before starting the matrix multiplication, one entire matrix is ​​loaded into a tile register, and at least one column of a second matrix is ​​loaded for N cycles. N inner products are processed for each cycle.

[0061] Matrix elements may be stored with different precisions according to a specific implementation, including 16-bit words, 8-bit bytes (e.g., INT8), and 4-bit half-bytes (e.g., INT4). Different precision modes may be specified for the tensor core (371) to ensure that the most efficient precision is used for different workloads (e.g., inference workloads that allow quantization into bytes and half-bytes).

[0062] The ray tracing core (372) can accelerate ray tracing operations for both real-time and non-real-time ray tracing implementations. In particular, the ray tracing core (372) may include a ray search / cross circuit that performs ray search using a boundary volume hierarchy (BVH) and identifies the intersection between a ray enclosed within the BVH volume and a primitive. The ray tracing core (372) may also include a circuit that performs depth testing and culling (e.g., using a Z-buffer or similar array). In one embodiment, the ray tracing core (372) performs search and cross operations in cooperation with the image denoising techniques described herein, at least some of which may be executed on a tensor core (371). For example, the tensor core (371) implements a deep learning neural network to perform denoising of the frame generated by the ray tracing core (372). However, the CPU(s) (361), graphics core (370) and / or ray tracing core (372) may also implement all or part of the noise removal and / or deep learning algorithms.

[0063] Additionally, as previously described, a distributed approach to noise removal may be used, wherein the GPU (380) is located in a computing device connected to another computing device via a network or high-speed interconnect. In this distributed approach, the interconnected computing devices share neural network learning / training data to improve the speed at which the entire system learns to perform noise removal for different types of image frames and / or different graphics applications.

[0064] The ray tracing core (372) processes all BVH seeks and ray-primitive intersections to prevent the graphics core (370) from being overloaded with thousands of instructions per ray. For example, each ray tracing core (372) includes a first set of special circuits that perform bounding box tests (e.g., seek operations) and a second set of special circuits that perform ray-triangle intersection tests (e.g., intersecting sought rays). Thus, for example, a multi-core group (365A) can simply fire ray probes, and the ray tracing core (372) independently performs ray seeks and intersections and returns hit data (e.g., hit, no hit, multiple hits, etc.) to the thread context. While the ray tracing core (372) performs seek and intersection operations, the other cores (370, 371) are free to perform other graphics or compute tasks.

[0065] Optionally, each ray tracing core (372) includes a seek unit that performs a BVH test operation and a cross unit that performs a ray-primitive cross test. The cross unit generates a "hit," "no hit," or "multiple hit" response and provides it to the appropriate thread. During the seek and cross operations, the execution resources of other cores (e.g., graphics core (370) and tensor core (371)) are freed up to perform other forms of graphics operations.

[0066] In one optional embodiment described below, a hybrid rasterization / ray tracing approach is used in which the work is distributed between the graphics core (370) and the ray tracing core (372).

[0067] The ray tracing core (372) (and / or other cores (370, 371)) may include hardware support for a ray tracing instruction set such as Microsoft’s DXR (DirectX Ray Tracing), which includes ray-generating, nearest hit, random hit, and miss shaders as well as DispatchRays commands, thereby allowing a unique set of shaders and textures to be assigned to each object. Another ray tracing platform that may be supported by the ray tracing core (372), graphics core (370), and tensor core (371) is Vulkan 1.1.85. However, it should be noted that the basic principles described herein are not limited to any specific ray tracing ISA.

[0068] Generally, various cores (372, 371, 370) may support a set of ray tracing instructions including instructions / functions for ray generation, nearest hit, random hit, ray-primitive intersection, per primitive and hierarchical bounding box construction, comparison, visit, and exception. More specifically, a preferred embodiment includes a ray tracing instruction that performs one or more of the following functions:

[0069] Ray Generation - The ray generation command can be executed for each pixel, sample, or other user-defined task assignment.

[0070] Nearest Hit - The Nearest Hit command can be executed to find the nearest intersection of a ray and a primitive within a scene.

[0071] Random Hit - The Random Hit command identifies multiple intersections between rays and primitives within a scene to identify potentially new nearest intersection points.

[0072] Intersect - The intersect command performs a ray-primitive intersection test and outputs the result.

[0073] Construct bounding box per primitive - This command forms a bounding box around a given primitive or group of primitives (e.g., when forming a new BVH or other acceleration data structure).

[0074] Comparison difference - This indicates that the ray does not intersect any geometry within the scene or a specific area of ​​the scene.

[0075] Visit - This indicates the child volumes that the ray will explore.

[0076] Exception - This includes various types of exception handlers (e.g., called for various error conditions).

[0077] GPU-Host Processor Interconnection Techniques

[0078] FIG. 4a illustrates an exemplary architecture in which a plurality of GPUs (410-413), such as the parallel processor (200) illustrated in FIG. 2a, are communicably connected to a plurality of multi-core processors (405-406) via high-speed links (440A-440D) (e.g., buses, point-to-point interconnects, etc.). Depending on the implementation, the high-speed links (440A-440D) may support communication throughputs of 4 GB / s, 30 GB / s, 80 GB / s, or higher. Various interconnection protocols may be used, including but not limited to PCIe 4.0 or 5.0 and NVLink 2.0. However, the basic principles described herein are not limited to any specific communication protocol or throughput.

[0079] Two or more of the GPUs (410-413) may be interconnected across high-speed links, and these high-speed links may be implemented using the same or different protocols / links as those used for the high-speed links (440A-440D). Similarly, two or more of the multi-core processors (405-406) may be connected via a high-speed link (443), which may be a symmetric multi-processor (SMP) bus operating at 20 GB / s, 30 GB / s, 120 GB / s, or higher. Alternatively, all communication between the various system components illustrated in FIG. 4a (e.g., via a common interconnection fabric) may be achieved using the same protocols / links. However, as noted, the basic principles described herein are not limited to any particular type of interconnection technology.

[0080] Each multi-core processor (405-406) can be communicateably connected to each processor memory (401-402) through a memory interconnect (430A-430B), and each GPU (410-413) can be communicateably connected to each GPU memory (420-423) through a GPU memory interconnect (450A-450D). The memory interconnects (430A-430B and 450A-450D) may use the same or different memory access technologies. As an example, not a limitation, processor memory (401 to 402) and GPU memory (420 to 423) may be volatile memory such as dynamic random access memory (DRAM) (including stacked DRAM), graphics DDR SDRAM (GDDR) (e.g., GDDR5, GDDR6), or high bandwidth memory (HBM), and / or may be non-volatile memory such as 3D XPoint / Optane or Nano-Ram. For example, some of the memories may be volatile memory, and other parts may be non-volatile memory (e.g., using a 2-level memory (2LM) layer).

[0081] As described below, various processors (405-406) and GPUs (410-413) may each be physically connected to specific memories (401-402, 420-423), but an integrated memory architecture may also be implemented in which the same virtual system address space (also referred to as the "effective address" space) is distributed among all the various physical memories. For example, processor memories (401 to 402) may each contain 64 GB of the system memory address space, and GPU memories (420 to 423) may each contain 32 GB of the system memory address space (ultimately, in this example, the total addressable memory is 256 GB).

[0082] FIG. 4b illustrates additional optional details regarding the interconnection between the multi-core processor (407) and the graphics acceleration module (446). The graphics acceleration module (446) may include one or more GPU chips integrated into a line card connected to the processor (407) via a high-speed link (440). Alternatively, the graphics acceleration module (446) may be integrated into the same package or chip as the processor (407).

[0083] The illustrated processor (407) includes a plurality of cores (460A-460D), each core having a translation index buffer (461A-461D) and one or more caches (462A-462D). These cores may include various other components (e.g., instruction fetch unit, branch prediction unit, decoder, execution unit, reorder buffer, etc.) for executing instructions not illustrated and processing data so as not to obscure the basic principles of the components described herein. The caches (462A-462D) may include Level 1 (L1) and Level 2 (L2) caches. Additionally, one or more shared caches (456) may be included in the caching layer and may be shared by a set of cores (460A-460D). For example, one embodiment of the processor (407) includes 24 cores, each of which has its own L1 cache, 12 shared L2 caches, and 12 shared L3 caches. In this embodiment, one of the L2 and L3 caches is shared by two adjacent cores. The processor (407) and the graphics accelerator integration module (446) are connected to a system memory (441) which may include processor memory (401-402).

[0084] Data and instructions stored in various caches (462A-462D, 456) and system memory (441) are maintained through inter-core communication via the coherence bus (464). For example, each cache may have associated cache coherence logic / circuits to communicate via the coherence bus (464) in response to a detected read or write to a specific cache line. In one embodiment, a cache snooping protocol is implemented via the coherence bus (464) to snoop cache access. Since cache snooping / coherence techniques will be well understood by those skilled in the art, they will not be described in detail in this specification to avoid obscuring the basic principles described herein.

[0085] A proxy circuit (425) may be provided to enable the graphics acceleration module (446) to communicate with the coherence bus (464) so ​​that the graphics acceleration module (446) can participate in the cache coherence protocol as a peer of the cores. Specifically, an interface (435) provides connectivity to the proxy circuit (425) via a high-speed link (440) (e.g., PCIe bus, NVLink, etc.), and an interface (437) connects the graphics acceleration module (446) to the link (440) at high speed.

[0086] In one embodiment, the accelerator integration circuit (436) provides cache management, memory access, context management, and interrupt management services on behalf of the plurality of graphics processing engines (431, 432, N) of the graphics acceleration module (446). The graphics processing engines (431, 432, N) may each include a separate graphics processing unit (GPU). Alternatively, the graphics processing engines (431, 432, N) may include different types of graphics processing engines within the GPU, such as graphics execution units, a media processing engine (e.g., a video encoder / decoder), a sampler, and a bullet engine. That is, the graphics acceleration module may be a GPU having a plurality of graphics processing engines (431-432, N), or the graphics processing engines (431-432, N) may be individual GPUs integrated on a common package, a line card, or a chip.

[0087] The accelerator integrated circuit (436) may include a memory management unit (MMU) (439) for performing various memory management functions, such as virtual-to-physical memory translation (also called effective-to-real memory translation) and memory access protocols for accessing system memory (441). The MMU (439) may also include a translation index buffer (TLB) (not shown) for caching virtual / effective-to-physical / real address translations. In one embodiment, the cache (438) stores commands and data for efficient access by the graphics processing engine (431-432, N). Data stored in the cache (438) and graphics memory (433-434, M) may be consistent with the core cache (462A-462D, 456) and system memory (411). As mentioned, this can be achieved through a proxy circuit (425) that participates in the cache consistency mechanism on behalf of the cache (438) and memory (433-434, M) (e.g., sending updates related to the modification / access of cache lines on the processor cache (462A-462D, 456) to the cache (438) and receiving updates from the cache (438)).

[0088] The register set (445) stores context data for a thread executed by the graphics processing engine (431-432, N), and the context management circuit (448) manages the thread context. For example, the context management circuit (448) can perform save and restore operations to save and restore the context of various threads during context switching (e.g., a first thread can be saved and a second thread can be saved so that the second thread is executed by the graphics processing engine). For example, at the time of context switching, the context management circuit (448) can save the current register value to a designated area in memory (e.g., identified by the context pointer). Then, the context management circuit can restore the register value when returning to the context. The interrupt management circuit (447) can receive and process interrupts received from, for example, a system device.

[0089] In one embodiment, a virtual / effective address from the graphics processing engine (431) is translated by the MMU (439) into an actual / physical address in system memory (411). Optionally, the accelerator integration circuit (436) supports a number of (e.g., 4, 8, 16) graphics accelerator modules (446) and / or other accelerator devices. A graphics accelerator module (446) may be dedicated to a single application running on the processor (407) or may be shared among multiple applications. Optionally, a virtualized graphics execution environment is provided in which the resources of the graphics processing engine (431-432, N) are shared among multiple applications or virtual machines (VMs). The resources may be subdivided into "slices" that are allocated to different VMs or applications based on processing requirements and priorities associated with the VMs and / or applications.

[0090] Accordingly, the accelerator integration circuit (436) acts as a bridge to the system for the graphics acceleration module (446) and provides address translation and system memory cache services. In one embodiment, to facilitate the bridging function, the accelerator integration circuit (436) may also include hardware and shared I / O (497) (e.g., PCIe, USB) that enable system control of voltage, clocking, performance, heats, and security. The shared I / O (497) may use separate physical connections or traverse the high-speed link (440). Additionally, the accelerator integration circuit (436) may provide virtualization facilities to enable the host processor to easily manage the virtualization, interrupts, and memory management of the graphics processing engine.

[0091] Since the hardware resources of the graphics processing engines (431-432, N) are explicitly mapped to the actual address space shown by the host processor (407), any host processor can directly address these resources using effective address values. One optional feature of the accelerator integration circuit (436) is the physical separation of the graphics processing engines (431-432, N), so that they appear in the system as independent units.

[0092] One or more graphics memories (433-434, M) may be connected to each graphics processing engine (431-432, N). The graphics memories (433-434, M) store instructions and data processed by each graphics processing engine (431-432, N). The graphics memories (433-434, M) may be volatile memory such as DRAM (including stacked DRAM), GDDR (e.g., GDDR5, GDDR6), or HBM, and / or may be non-volatile memory such as 3D XPoint / Optane or Nano-Ram.

[0093] To reduce data traffic through the high-speed link (440), biasing techniques may be used to ensure that the data stored in the graphics memory (433-434, M) is preferably not used by the core (460A-460D) (at least not frequently) and is most frequently used by the graphics processing engine (431-432, N). Similarly, the biasing mechanism attempts to keep the data required by the core (and preferably not the graphics processing engine (431-432, N)) within the core's cache (462A-462D, 456) and system memory (411).

[0094] According to the variant illustrated in FIG. 4c, the accelerator integrated circuit (436) is integrated within the processor (407). The graphics processing engine (431-432, N) communicates with the accelerator integrated circuit (436) via the high-speed link (440) through the interface (437) and interface (435) (again, any form of bus or interface protocol may be used). The accelerator integrated circuit (436) can perform the same operations as described in relation to FIG. 4b, but potentially higher throughput may be achieved if it is very close to the coherence bus (464) and cache (462A-462D, 456).

[0095] The described embodiments may support different programming models, including a dedicated process programming model (without graphics acceleration module virtualization) and a shared programming model (with virtualization). The latter may include a programming model controlled by the accelerator integration circuit (436) and a programming model controlled by the graphics acceleration module (446).

[0096] In an embodiment of a dedicated process model, the graphics processing engine (431-432, N) may be dedicated to a single application or process under a single operating system. The single application may funnel other application requests to the graphics engine (431-432, N) to provide virtualization within a VM / partition.

[0097] In a dedicated process programming model, graphics processing engines (431-432, N) can be shared by multiple VM / application partitions. The shared model requires the system hypervisor to virtualize the graphics processing engines (431-432, N) so that each operating system can access them. In the case of a single-partition system without a hypervisor, the operating system owns the graphics processing engines (431-432, N). In both cases, the operating system can virtualize the graphics processing engines (431-432, N) to provide access to each process or application.

[0098] For a shared programming model, a graphics acceleration module (446) or an individual graphics processing engine (431-432, N) selects a process element using a process handle. The process element may be stored in system memory (411) and may be addressable using the effective address to physical address translation techniques described herein. The process handle may be an implementation-specific value provided to the host process when registering its context with the graphics processing engine (431-432, N) (i.e., when calling system software to add a process element to the process element linked list). The lower 16 bits of the process handle may be the offset of the process element within the process element linked list.

[0099] FIG. 4d illustrates an exemplary accelerator integration slice (490). As used herein, a “slice” comprises a specific portion of the processing resources of the accelerator integration circuit (436). An application effective address space (482) within system memory (411) stores a process element (483). The process element (483) may be stored in response to a GPU call (481) from an application (480) running on the processor (407). The process element (483) contains the process state for the corresponding application (480). A job descriptor (WD) (484) contained in the process element (483) may be a single job requested by the application or may contain a pointer to a queue of jobs. In the latter case, the WD (484) is a pointer to a job request queue within the application's address space (482).

[0100] The graphics acceleration module (446) and / or individual graphics processing engines (431-432, N) may be shared by all or a subset of processes within the system. For example, the technology described herein may include infrastructure for transmitting WD (484) to the graphics acceleration module (446) to set the process state and start a job in a virtualized environment.

[0101] In one embodiment, the dedicated process programming model is implementation-specific. In this model, a single process owns the graphics acceleration module (446) or the individual graphics processing engine (431). Since the graphics acceleration module (446) is owned by a single process, when allocating the graphics acceleration module (446), the hypervisor initializes the accelerator integration circuit (436) for the owning partition, and the operating system initializes the accelerator integration circuit (436) for the owning process.

[0102] During operation, the WD fetch unit (491) within the accelerator integration slice (490) fetches the next WD (484), which contains an indication of the operation to be performed by one of the graphics processing engines of the graphics acceleration module (446). Data from the WD (484) may be stored in a register (445) and may be used by the MMU (439), the interrupt management circuit (447), and / or the context management circuit (448) as illustrated. For example, the MMU (439) may include a segment / page work circuit for accessing segment / page tables (486) within the OS virtual address space (485). The interrupt management circuit (447) may process an interrupt event (492) received from the graphics acceleration module (446). When performing a graphics operation, the effective address (493) generated by the graphics processing engines (431-432, N) is converted into an actual address by the MMU (439).

[0103] The same set of registers (445) may be duplicated for each graphics processing engine (431-432, N) and / or graphics acceleration module (446) and may be initialized by a hypervisor or operating system. Each of these duplicated registers may be included in an accelerator integration slice (490). Exemplary registers that may be initialized by a hypervisor are shown in Table 1.

[0104]

[0105] Exemplary registers that can be initialized by the operating system are shown in Table 2.

[0106]

[0107] Each WD (484) may be specific to a particular graphics acceleration module (446) and / or graphics processing engine (431-432, N). The WD may contain all information that the graphics processing engine (431-432, N) requires to perform its own work, or the WD may be a pointer to a memory location that has set up a command queue for the work to be completed by the application. FIG. 4e illustrates additional optional details of the shared model. This figure includes a hypervisor physical address space (498) where a process element list (499) is stored. The hypervisor physical address space (498) is accessible through a hypervisor (496) that virtualizes the graphics acceleration module engine for the operating system (495).

[0108] The shared programming model allows all or a subset of processes from all or a subset of partitions within the system to use the graphics acceleration module (446). There are two programming models in which the graphics acceleration module (446) is shared by multiple processes and partitions: time-sharing sharing and graphics-oriented sharing.

[0109] In this model, the system hypervisor (496) owns the graphics acceleration module (446) and makes its functions available to all operating systems (495). For the graphics acceleration module (446) to support virtualization by the system hypervisor (496), the graphics acceleration module may comply with the following requirements: 1) Job requests from applications must be autonomous (i.e., state does not need to be maintained between jobs), or the graphics acceleration module (446) must provide a context save and restore mechanism. 2) Job requests from applications must be guaranteed by the graphics acceleration module (446) to be completed within a specified time, including any transformation defects, or the graphics acceleration module (446) must provide the ability to preempt the processing of jobs. 3) Fairness between processes must be guaranteed when the graphics acceleration module (446) operates in a specified shared programming model.

[0110] In the case of a shared model, the application (480) may be required to have an operating system (495) perform a system call having a graphics acceleration module (446) type, a job descriptor (WD), an authorization mask register (AMR) value, and a context save / restore area pointer (CSRP). The graphics acceleration module (446) type describes the target acceleration function for the system call. The graphics acceleration module (446) type may be a system-specific value. The WD is specifically formatted for the graphics acceleration module (446) and may be in the form of a graphics acceleration module (446) command, a valid address pointer for a user-defined structure, a valid address pointer for a queue of commands, or any other data structure to describe the work to be performed by the graphics acceleration module (446). In one embodiment, the AMR value is the AMR state to be used for the current process. The value passed to the operating system is similar to an application setting the AMR. If the implementations of the accelerator integration circuit (436) and the graphics acceleration module (446) do not support the User Authority Mask Override Register (UAMOR), the operating system may apply the current UAMOR value to the AMR value before passing the AMR in a hypervisor call. The hypervisor (496) may optionally apply the current Authority Mask Override Register (AMOR) value before placing the AMR in the process element (483). The CSRP may be one of the registers (445) containing the effective address of a region within the application's address space (482) for the graphics acceleration module (446) to save and restore the context state. This pointer is optional when no state needs to be saved between tasks or when a task is preempted. The context save / restore region may be fixed system memory.

[0111] When the system is called, the operating system (495) can verify whether the application (480) has registered and been granted permission to use the graphics acceleration module (446). Then, the operating system (495) calls the hypervisor (496) with the information shown in Table 3.

[0112]

[0113] When the hypervisor receives a call, the hypervisor (496) verifies whether the operating system (495) has registered and been granted permission to use the graphics acceleration module (446). Then, the hypervisor (496) places the process element (483) into the process element linked list for the corresponding graphics acceleration module (446) type. The process element may contain information shown in Table 4.

[0114]

[0115] The hypervisor can initialize multiple accelerator integration slice (490) registers (445). As illustrated in FIG. 4f, in one optional embodiment, addressable integration memory is employed through a common virtual memory address space used to access physical processor memory (401 to 402) and GPU memory (420 to 423). In this embodiment, operations executed on the GPU (410-413) utilize the same virtual / effective memory address space to access processor memory (401-402), and vice versa, thereby simplifying programming. A first portion of the virtual / effective address space may be allocated to processor memory (401), a second portion to second processor memory (402), and a third portion to GPU memory (420). Therefore, the entire virtual / effective memory space (sometimes referred to as the effective address space) may be distributed across processor memory (401 to 402) and GPU memory (420 to 423), respectively, so that any processor or GPU can access any physical memory having a virtual address mapped to that memory.

[0116] In one or more of the MMUs (439A-439E), a bias / coherence management circuit (494A-494E) may be provided to implement a biasing technique that ensures cache coherence between the cache of a host processor (e.g., 405) and the GPU (410-413) and indicates physical memory where specific types of data should be stored. Although multiple instances of the bias / coherence management circuit (494A-494E) are illustrated in FIG. 4f, the bias / coherence circuit may be implemented within the MMU and / or accelerator integration circuit (436) of one or more host processors (405).

[0117] GPU attached memory (420-423) is mapped as part of system memory and can be accessed using shared virtual memory (SVM) technology, but does not suffer from the typical performance drawbacks associated with full system cache coherence. The ability to access GPU-attached memory (420-423) as system memory without cumbersome cache coherence overhead provides a beneficial operating environment for GPU offload. This configuration allows host processor (405) software to set operands and access calculation results without the overhead of traditional I / O DMA data copies. These traditional copies include driver calls, interrupts, and memory-mapped I / O (MMIO) accesses, all of which are inefficient compared to simple memory access. At the same time, the ability to access GPU attached memory (420-423) without cache coherence overhead can be critical to the execution time of offloaded calculations. For example, in cases with significant streaming write memory traffic, cache coherence overhead can significantly reduce the effective write bandwidth shown by the GPU (410-413). The efficiency of operand setup, result access, and GPU computation all play a role in determining the validity of GPU offload.

[0118] The choice between GPU bias and host processor bias can be driven by a bias tracker data structure. For example, a bias table may be used that is a page-granular structure containing 1 or 2 bits per GPU-attached memory page (i.e., controlled by the granularity of the memory pages). The bias table may be implemented in a stolen memory range of one or more GPU-attached memories (420-423) with or without a bias cache in the GPU (410-413) (e.g., to cache frequently / recently used entries of the bias table). Alternatively, the entire bias table may be maintained within the GPU.

[0119] In one embodiment, the bias table entry associated with each access to the GPU-attached memory (420-423) is accessed prior to the actual access to the GPU memory, and the following actions occur. First, a local request from the GPU (410-413) seeking its own page in the GPU bias is forwarded directly to the corresponding GPU memory (420-423). A local request from the GPU seeking its own page in the host bias is forwarded to the processor (405) (e.g., via the high-speed link discussed above). Optionally, a request from the processor (405) seeking the requested page in the host processor bias completes a request such as a normal memory read. Alternatively, requests directed to the GPU-biased page may be forwarded to the GPU (410-413). The GPU may then move the page to the host processor bias if it is not currently in use.

[0120] The bias state of a page can be changed by software-based mechanisms, hardware-assisted software-based mechanisms, or, in some limited cases, by purely hardware-based mechanisms.

[0121] One mechanism for changing the bias state uses an API call (e.g., OpenCL), which in turn calls the GPU's device driver, and the device driver sends a message to the GPU instructing it to change the bias state (or encues a command descriptor), and in some variations, performs a cache flushing operation on the host. A cache flushing operation is required for the transition from the host processor (405) bias to the GPU bias, but not for the reverse transition.

[0122] Cache consistency can be maintained by temporarily rendering GPU-biased pages uncacheable by the host processor (405). To access these pages, the processor (405) may request access from the GPU (410), which may or may not immediately grant access depending on the implementation. Therefore, to reduce communication between the host processor (405) and the GPU (410), it is beneficial to ensure that the GPU-biased pages are pages requested by the GPU rather than the host processor (405), and vice versa.

[0123] Graphics processing pipeline

[0124] FIG. 5 illustrates a graphics processing pipeline (500). A graphics processing pipeline (500) illustrated with a graphics multiprocessor (234) as in FIG. 2d, a graphics multiprocessor (325) as in FIG. 3a, and a graphics multiprocessor (350) as in FIG. 3b may be implemented. A graphics multiprocessor may be associated with the parallel processor(s) (112) of FIG. 1 and may be included within a parallel processing subsystem as described herein, such as the parallel processor (200) of FIG. 2a, which may be used instead of one of these. Various parallel processing systems may implement the graphics processing pipeline (500) through one or more instances of a parallel processing unit (e.g., the parallel processing unit (202) of FIG. 2a) as described herein. For example, a shader unit (e.g., the graphics multiprocessor (234) of FIG. 2c) may be configured to perform one or more of the functions of a vertex processing unit (504), a tessellation control processing unit (508), a tessellation evaluation processing unit (512), a geometry processing unit (516), and a fragment / pixel processing unit (524). The functions of a data assembler (502), a primitive assembler (506, 514, 518), a tessellation unit (510), a rasterizer (522), and a raster operation unit (526) may also be performed by other processing engines within a processing cluster (e.g., the processing cluster (214) of FIG. 2a) and a corresponding partition unit (partition unit (220A-220N) of FIG. 2a). The graphics processing pipeline (500) may also be implemented using dedicated processing units for one or more functions. It is also possible for one or more parts of the graphics processing pipeline (500) to be performed by parallel processing logic within a general-purpose processor (e.g., CPU).Optionally, one or more parts of the graphics processing pipeline (500) may access on-chip memory (e.g., parallel processor memory (222) of FIG. 2a) through a memory interface (528) which may be an instance of the memory interface (218) of FIG. 2a. The graphics processor pipeline (500) may also be implemented through a multi-core group (365A) as in FIG. 3c.

[0125] The data assembler (502) is a processing unit capable of collecting vertex data for surfaces and primitives. The data assembler (502) outputs vertex data containing vertex attributes to the vertex processing unit (504). The vertex processing unit (504) is a programmable execution unit that executes a vertex shader program that illuminates and transforms the vertex data specified by the vertex shader program. The vertex processing unit (504) reads data that is stored in a cache, local, or system memory for use in processing the vertex data and can be programmed to transform the vertex data from an object-based coordinate representation to a world space coordinate space or a normalized device coordinate space.

[0126] A first instance of the primitive assembler (506) receives vertex attributes from the vertex processing unit (504). The primitive assembler (506) reads the stored vertex attributes as needed and constructs graphics primitives for processing by the tessellation control processing unit (508). The graphics primitives include triangles, line segments, points, patches, etc., supported by various graphics processing application programming interfaces (APIs).

[0127] The tessellation control processing unit (508) processes input vertices as control points for a geometric patch. The control points are converted from an input representation from the patch (e.g., the base of the patch) into a representation suitable for use in surface evaluation by the tessellation evaluation processing unit (512). The tessellation control processing unit (508) can also calculate a tessellation factor for the edges of the geometric patch. The tessellation factor is applied to a single edge and quantifies the view-dependent level of detail associated with the edge. The tessellation unit (510) is configured to receive the tessellation factor for the edges of the patch and to tessellate the patch into a number of geometric primitives, such as lines, triangles, or quadrilateral primitives, and these primitives are transmitted to the tessellation evaluation processing unit (512). The tessellation evaluation processing unit (512) operates on the parameterized coordinates of the subdivided patches to generate vertex attributes and surface representations for each vertex associated with geometric primitives.

[0128] A second instance of the primitive assembler (514) receives vertex attributes from the tessellation evaluation processing unit (512), reads the stored vertex attributes as needed, and constructs a graphic primitive for processing by the geometry processing unit (516). The geometry processing unit (516) is a programmable execution unit that executes a geometry shader program to convert the graphics primitive received from the primitive assembler (514) as specified by the geometry shader program. The geometry processing unit (516) can be programmed to subdivide the graphic primitive into one or more new graphic primitives and to calculate parameters used to rasterize the new graphic primitives.

[0129] The geometry processing unit (516) can add or delete elements from the geometry stream. The geometry processing unit (516) outputs parameters and vertices that specify a new graphic primitive to the primitive assembler (518). The primitive assembler (518) receives the parameters and vertices from the geometry processing unit (516) and constructs the graphic primitive for processing by the viewport scale, curl, and clip unit (520). The geometry processing unit (516) reads data stored in parallel processor memory or system memory to use for processing the geometry data. The viewport scale, curl, and clip unit (520) performs clipping, curling, and viewport scaling, and outputs the processed graphic primitive to the rasterizer (522).

[0130] The rasterizer (522) can perform depth culling and other depth-based optimizations. The rasterizer (522) also performs scan transformations on new graphic primitives to generate fragments and outputs these fragments and associated coverage data to the fragment / pixel processing unit (524). The fragment / pixel processing unit (524) is a programmable execution unit configured to execute a fragment shader program or a pixel shader program. The fragment / pixel processing unit (524) transforms the fragment or pixel received from the rasterizer (522) as specified by the fragment or pixel shader program. For example, the fragment / pixel processing unit (524) may be programmed to perform operations including, but not limited to, texture mapping, shading, blending, texture correction, and perspective correction to generate shaded fragments or pixels output to the raster operation unit (526). The fragment / pixel processing unit (524) may read data stored in parallel processor memory or system memory for use when processing fragment data. Fragment or pixel shader programs may be configured to shade at samples, pixels, tiles, or other granularities according to the sampling rate configured for the processing unit.

[0131] The raster operation unit (526) is a processing unit that performs raster operations, including but not limited to stenciling, z-testing, blending, etc., and outputs pixel data as processed graphic data to be stored in a graphic memory (e.g., parallel processor memory (222) of FIG. 2a, and / or system memory (104) of FIG. 1) for display on one or more display devices (110) or for further processing by one or more processors (102) or parallel processors (112). The raster operation unit (526) may be configured to compress z or color data written to memory and decompress z or color data read from memory.

[0132] Machine Learning Overview

[0133] The aforementioned architecture can be applied to perform training and inference operations using machine learning models. Machine learning has been successful in solving many types of tasks. The computations involved in training and using machine learning algorithms (e.g., neural networks) are inherently suitable for efficient parallel implementation. Therefore, parallel processors, such as general-purpose graphic processing units (GPGPUs), play a crucial role in the practical implementation of deep neural networks. Parallel graphics processors with a Single Instruction, Multi-Thread (SIMT) architecture are designed to maximize the amount of parallel processing in the graphics pipeline. In a SIMT architecture, groups of parallel threads attempt to execute program instructions synchronously together as often as possible to increase processing efficiency. The efficiency provided by the implementation of parallel machine learning algorithms allows for the use of high-capacity networks and enables these networks to be trained on larger datasets.

[0134] Machine learning algorithms are algorithms capable of learning based on a dataset. For example, machine learning algorithms can be designed to model high-level abstractions within a dataset. For instance, image recognition algorithms can be used to determine which category a given input belongs to, regression algorithms can output numerical values ​​given an input, and pattern recognition algorithms can be used to generate translated text or perform text-to-speech and / or speech recognition.

[0135] A representative type of machine learning algorithm is the neural network. There are various types of neural networks, and a simple type is the feedforward network. A feedforward network can be implemented as an acyclic graph in which nodes are arranged in layers. Generally, the feedforward network topology includes an input layer and an output layer separated by at least one hidden layer. The hidden layer transforms the input received by the input layer into a representation useful for generating an output at the output layer. Network nodes are fully connected to nodes in adjacent layers through edges, but there are no edges between nodes within each layer. Data received from the nodes of the input layer of a feedforward network is propagated (i.e., "feedforwarded") to the nodes of the output layer through an activation function that calculates the state of the nodes in each successive layer within the network based on the coefficients ("weights") associated with each edge connecting the layers. Depending on the specific model represented by the algorithm being executed, the output of a neural network algorithm can take various forms.

[0136] Before a machine learning algorithm can be used to model a specific problem, the algorithm is trained using a training dataset. Neural network training involves selecting a network topology, using a training dataset representing the problem to be modeled by the network, and adjusting weights until the network model performs with minimal error for every instance of the training dataset. For example, during the supervised learning training process for a neural network, the output generated by the network in response to an input representing an instance of the training dataset is compared with an output labeled as "correct" for that instance; the difference between the output and the labeled output is calculated, and the weights associated with the connection are adjusted to minimize the error as the error signal is backpropagated through the network layers. The network is considered "trained" when the error for each output generated from an instance of the training dataset is minimized.

[0137] The accuracy of machine learning algorithms can be significantly affected by the quality of the datasets used for training. The training process can be computationally intensive and may require substantial time on conventional general-purpose processors. Therefore, parallel processing hardware is used to train many types of machine learning algorithms. This is particularly useful for optimizing neural network training, as the computations performed to tune the coefficients of a neural network naturally benefit from parallel implementation. In particular, many machine learning algorithms and software applications are configured to utilize parallel processing hardware within general-purpose graphics processing units.

[0138] FIG. 6 is a generalized diagram of a machine learning software stack (600). A machine learning application (602) may be configured to train a neural network using a training data set or to implement machine intelligence using a trained deep neural network. The machine learning application (602) may include specialized software and / or training and inference functions for the neural network that can be used to train the neural network before deployment. The machine learning application (602) may implement any type of machine intelligence including, but not limited to, image recognition, mapping and localization, autonomous navigation, speech synthesis, medical imaging, or language translation.

[0139] Hardware acceleration for machine learning applications (602) can be enabled through a machine learning framework (604). The machine learning framework (604) can provide a library of machine learning primitives. Machine learning primitives are basic operations typically performed by machine learning algorithms. Without the machine learning framework (604), machine learning algorithm developers would have to generate and optimize the key computational logic associated with machine learning algorithms, and then re-optimize the computational logic when new parallel processors are developed. Instead, machine learning applications can be configured to perform necessary computations using the primitives provided by the machine learning framework (604). Exemplary primitives include tensor convolution, activation functions, and pooling, which are computational operations performed during the training of a convolutional neural network (CNN). The machine learning framework (604) can also provide primitives for implementing basic linear algebra subprograms performed by many machine learning algorithms, such as matrix and vector operations.

[0140] The machine learning framework (604) can process input data received from the machine learning application (602) and generate appropriate input for the computing framework (606). The computing framework (606) abstracts the basic commands provided to the GPGPU driver (608) so that the machine learning framework (604) can utilize hardware acceleration through the GPGPU hardware (610) without the machine learning framework (604) needing to know the architecture of the GPGPU hardware (610) in detail. Additionally, the computing framework (606) can enable hardware acceleration for the machine learning framework (604) across various types and generations of GPGPU hardware (610).

[0141] GPGPU Machine Learning Acceleration

[0142] FIG. 7 illustrates a general-purpose graphics processing unit (700) which may be the parallel processor (200) of FIG. 2a or the parallel processor(s) (112) of FIG. 1. The general-purpose processing unit (GPGPU) (700) may be configured to be particularly efficient at processing the type of computational workload associated with training deep neural networks. Additionally, the GPGPU (700) may be directly connected to other instances of the GPGPU to create a multi-GPU cluster, thereby improving the training speed, particularly for deep neural networks.

[0143] The GPGPU (700) includes a host interface (702) that enables a connection with a host processor. The host interface (702) may be a PCI Express interface. However, the host interface may be a vendor-specific communication interface or a communication fabric. The GPGPU (700) receives commands from the host processor and uses a global scheduler (704) to distribute execution threads associated with these commands to a set of processing clusters (706A-706H). The processing clusters (706A-706H) share a cache memory (708). The cache memory (708) may function as an upper-level cache for the cache memory within the processing clusters (706A-706H). The illustrated processing clusters (706A-706H) may correspond to the processing clusters (214A-214N) of FIG. 2A.

[0144] The GPGPU (700) includes memory (714A-714B) combined with a processing cluster (706A-706H) through a set of memory controllers (712A-712B). The memory (714A-714B) may include various types of memory devices, such as dynamic random access memory (DRAM) or graphics random access memory (SGRAM), including graphics double data rate (GDDR) memory. The memory (714A-714B) may also include 3D stacked memory, including but not limited to high bandwidth memory (HBM).

[0145] Each of the processing clusters (706A-706H) may include a set of graphics multiprocessors, such as the graphics multiprocessor (234) of FIG. 2d, the graphics multiprocessor (325) of FIG. 3a, or the graphics multiprocessor (350) of FIG. 3b, or may include a multi-core group (365A-365N) as in FIG. 3c. The graphics multiprocessors of the computing clusters include a number of types of integer and floating-point logic units capable of performing computing operations within a range of precision suitable for machine learning calculations. For example, at least one subset of the floating-point units of each of the processing clusters (706A-706H) may be configured to perform 16-bit or 32-bit floating-point operations, while another subset of the floating-point units may be configured to perform 64-bit floating-point operations.

[0146] Multiple instances of the GPGPU (700) may be configured to operate as a computing cluster. The communication mechanism used by the computing cluster for synchronization and data exchange varies according to the embodiment. For example, multiple instances of the GPGPU (700) communicate through a host interface (702). In one embodiment, the GPGPU (700) includes an I / O hub (709) that connects the GPGPU (700) to a GPU link (710) that enables direct access to other instances of the GPGPU. The GPU link (710) may be connected to a dedicated GPU-GPU bridge that enables communication and synchronization between multiple instances of the GPGPU (700). Optionally, the GPU link (710) is connected to a high-speed interconnect for transmitting and receiving data to and from other GPGPUs or parallel processors. Multiple instances of the GPGPU (700) are located in separate data processing systems and communicate through network devices accessible via the host interface (702). The GPU link (710) may be configured to enable access to the host processor in addition to or instead of the host interface (702).

[0147] The illustrated configuration of the GPGPU (700) may be configured to train a neural network, but other configurations of the GPGPU (700) may be configured to be deployed within a high-performance or low-power inference platform. In the inference configuration, the GPGPU (700) includes fewer processing clusters (706A-706H) compared to the training configuration. Additionally, memory technology associated with memory (714A-714B) may differ between the inference configuration and the training configuration. In one embodiment, the inference configuration of the GPGPU (700) may support inference-specific instructions. For example, the inference configuration may provide support for one or more 8-bit integer dot product instructions commonly used during inference operations for a deployed neural network.

[0148] FIG. 8 illustrates a multi-GPU computing system (800). The multi-GPU computing system (800) may include a processor (802) connected to a plurality of GPGPUs (806A-806D) via a host interface switch (804). The host interface switch (804) may be a PCI Express switch device that connects the processor (802) to a PCI Express bus (through which the processor (802) can communicate with a set of GPGPUs (806A-806D)). Each of the plurality of GPGPUs (806A-806D) may be an instance of the GPGPU (700) of FIG. 7. The GPGPUs (806A-806D) may be interconnected via a set of high-speed point-to-point GPU-GPU links (816). High-speed GPU-to-GPU links can be connected to each of the GPGPUs (806A-806D) via a dedicated GPU link, such as the GPU link (710) of FIG. 7. The P2P GPU link (816) enables direct communication between each of the GPGPUs (806A-806D) without requiring communication through the host interface bus to which the processor (802) is connected. When GPU-to-GPU traffic is directed to the P2P GPU links, the host interface bus remains available for system memory access or for communicating with other instances of the multi-GPU computing system (800), for example, through one or more network devices. In FIG. 8, the GPGPUs (806A-806D) are connected to the processor (802) via the host interface switch (804), but the processor (802) may, alternatively, include direct support for the P2P GPU link (816) and be connected directly to the GPGPUs (806A-806D).

[0149] Machine learning neural network implementation

[0150] The computing architecture provided by the embodiments described herein may be configured to perform a type of parallel processing particularly suitable for training and deploying neural networks for machine learning. Neural networks can be generalized to a network of functions having graph relationships. As is well known in the art, there are various types of neural network implementations used in machine learning. One exemplary type of neural network is a feedforward network, as described above.

[0151] The second exemplary type of neural network is the Convolutional Neural Network (CNN). A CNN is a specialized feedforward neural network designed to process data with a known grid-like topology, such as image data. Therefore, while CNNs are commonly used in computing vision and image recognition applications, they can also be used for other types of pattern recognition, such as speech and language processing. Nodes in the CNN input layer consist of a set of "filters" (feature detectors influenced by receptive fields found in the retina), and the output of each filter set is propagated to nodes within successive layers of the network. Computation for a CNN involves applying convolutional mathematical operations to each filter to generate the output of that filter. Convolution is a special type of mathematical operation performed by two functions to create a third function that is a modified version of one of the two original functions. In convolutional network terminology, the first function for the convolution can be referred to as the input, and the second function can be referred to as the convolution kernel. The output can be referred to as a feature map. For example, the input to a convolution layer can be a multidimensional array of data defining various color components of an input image. A convolution kernel can be a multidimensional array of parameters, where the parameters are adjusted by the training process for the neural network.

[0152] A Recurrent Neural Network (RNN) is a family of feedforward neural networks that includes feedback connections between layers. When used, RNNs enable the modeling of sequential data by sharing parameter data across multiple parts of the neural network. The architecture of an RNN includes periods. Since at least a portion of the RNN's output data is used as feedback to process subsequent inputs in the sequence, a period represents the influence that the current value of a variable has on its own value in the future. This feature makes RNNs particularly useful for language processing due to the variable nature that can be utilized to construct language data.

[0153] The drawings described below not only present exemplary feedforward, CNN, and RNN networks but also explain general processes for training and deploying each of these types of networks. It will be understood that these descriptions are exemplary and non-limiting with respect to any specific embodiment described herein, and that the illustrated concepts can generally be applied to deep neural networks and machine learning techniques.

[0154] The exemplary neural network described above can be used to perform deep learning. Deep learning is machine learning that uses deep neural networks. Unlike shallow neural networks, which contain only a single hidden layer, the deep neural networks used in deep learning are artificial neural networks composed of multiple hidden layers. Deep neural networks are generally more computationally intensive for training. However, the additional hidden layers of the network enable multi-level pattern recognition, which reduces output errors compared to shallow machine learning techniques.

[0155] Deep neural networks used in deep learning generally consist of a front-end network that performs feature recognition coupled to a back-end network, which represents a mathematical model capable of performing actions (e.g., object classification, speech recognition) based on features provided to the model. Deep learning enables machine learning without the need for hand-crafted feature engineering on the model. Instead, deep neural networks can learn features based on statistical structures or correlations within the input data. The learned features can be provided to a mathematical model capable of mapping detected features to the output. The mathematical model used by the network is typically specialized for the specific task to be performed, and a different model will be used to perform different tasks.

[0156] Once a neural network is constructed, a learning model can be applied to train the network to perform specific tasks. The learning model describes how to adjust the weights within the model to reduce the network's output error. Backpropagation is a common method used in neural networks. Input vectors are provided to the network for processing. The network's output is compared to the desired output using a loss function, and an error value is calculated for each neuron within the output layer. Then, the error values ​​are backpropagated until each neuron has an associated error value that roughly represents its contribution to the original output. Subsequently, the network can update its weights by learning from these errors using algorithms such as stochastic gradient descent.

[0157] FIGS. 9a and 9b illustrate an exemplary convolutional neural network. FIG. 9a illustrates various layers within a CNN. As illustrated in FIG. 9a, an exemplary CNN used to model image processing can receive an input (902) representing red, green, and blue (RGB) components. The input (902) can be processed by a plurality of convolutional layers (e.g., a first convolutional layer (904), a second convolutional layer (906)). The output from the plurality of convolutional layers can optionally be processed by a set of fully connected layers (908). Neurons in a fully connected layer have full connections to all activations in the preceding layer, as described above for a feedforward network. The output from the fully connected layer (908) can be used to generate an output result from the network. Activations in the fully connected layer (908) can be calculated using matrix multiplication instead of convolution. Not all CNN implementations use a fully connected layer (908). For example, in some implementations, a second convolution layer (906) can generate the output for the CNN.

[0158] The convolution layer is sparsely connected, which differs from the conventional neural network structure found in the fully connected layer (908). Conventional neural network layers are fully connected, so that every output unit interacts with every input unit. However, the convolution layer is sparsely connected because, as illustrated, the output of the field's convolution is an input to the nodes of the subsequent layer (instead of the respective state values ​​of each node within the field). The kernel associated with the conventional layer performs the convolution operation, and its output is transmitted to the next layer. The dimensionality reduction performed within the convolution layer is one aspect that enables the CNN to scale to process large images.

[0159] FIG. 9b illustrates exemplary computational steps within a convolution layer of a CNN. Inputs to the convolution (912) of the CNN can be processed in three stages of the convolution layer (914). These three stages may include a convolution stage (916), a detector stage (918), and a pooling stage (920). Then, the convolution layer (914) can output data to a subsequent convolution layer. The final convolution layer of the network can generate output feature map data or provide input to a fully connected layer to generate classification values ​​to be input to, for example, the CNN.

[0160] In the convolution step (916), multiple convolutions are performed in parallel to generate a linear activation set. The convolution step (916) may include an affine transformation, which is any transformation that can be specified as the sum of a linear transformation and a translation. Affine transformations include rotation, translation, expansion, and combinations of these transformations. The convolution step calculates the output of a function (e.g., a neuron) connected to a specific region of input (which may be determined as a local region associated with the neuron). The neuron calculates the inner product between the neuron's weights and the region of local input to which the neuron is connected. The output from the convolution step (916) defines a linear activation set that is processed by a subsequent stage of the convolution layer (914).

[0161] Linear activations can be processed by the detector step (918). In the detector step (918), each linear activation is processed by a non-linear activation function. The non-linear activation function increases the non-linear characteristics of the entire network without affecting the receptive field of the convolution layer. Several types of non-linear activation functions can be used. One specific type is the rectified linear unit (ReLU), which is an activation function defined as f(x)=max(0,x) such that the activation has a threshold value at 0.

[0162] The pooling step (920) uses a pooling function that replaces the output of the second convolution layer (906) with a summary statistic of the surrounding outputs. The pooling function can be used to introduce translation invariance to the neural network so that small translations of the input do not change the pooled output. Invariance to local translation can be useful in scenarios where the presence of a feature within the input data is more important than the exact location of the feature. During the pooling step (920), various types of pooling functions may be used, including max pooling, mean pooling, and l2-norm pooling. Additionally, some CNN implementations do not include a pooling step. Instead, these implementations replace it with an additional convolution step having an increased stride compared to the previous convolution step.

[0163] Then, the output of the convolution layer (914) can be processed by the next layer (922). The next layer (922) may be either an additional convolution layer or a fully connected layer (908). For example, the first convolution layer (904) of FIG. 9a may output to the second convolution layer (906), and the second convolution layer may output to the first layer of the fully connected layer (908).

[0164] FIG. 10 illustrates an exemplary recurrent neural network (1000). In a recurrent neural network (RNN), the previous state of the network influences the output of the network's current state. RNNs can be constructed in various ways using various functions. The use of RNNs generally deals primarily with predicting the future based on previous input sequences using mathematical models. For example, an RNN can be used to perform statistical language modeling to predict the next word given a previous word sequence. The illustrated RNN (1000) can be described as having an input layer (1002) that receives an input vector, a hidden layer (1004) for implementing a recurrent function, a feedback mechanism (1005) that enables 'memory' of the previous state, and an output layer (1006) that outputs the result. The RNN (1000) operates based on time steps. At a given time step, the state of the RNN is influenced based on the previous time step through the feedback mechanism (1005). In a given time step, the state of the hidden layer (1004) is defined by the previous state and the input in the current time step. The initial input (x1) in the first time step can be processed by the hidden layer (1004). The second input (x2) can be processed by the hidden layer (1004) using state information determined during the processing of the initial input (x1). The given state is s t =f(Ux t +Ws t-1 It can be calculated as ), where U and W are parameter matrices. The function (f) is generally nonlinear, such as a hyperbolic tangent function (Tanh) or a variation of the rectifier function f(x)=max(0,x). However, the specific mathematical function used in the hidden layer (1004) may vary depending on the details of the specific implementation of the RNN (1000).

[0165] In addition to the aforementioned basic CNN and RNN networks, variations of these networks may be possible. One example of an RNN variation is the Long Short-Term Memory (LSTM) RNN. The LSTM RNN can learn long-term dependencies that may be necessary for processing longer language sequences. A variation of the CNN is the Convolutional Deep Belief Network (DBN), which has a structure similar to the CNN and is trained in a manner similar to the Deep Belief Network. The Deep Belief Network (DBN) is a generative neural network composed of multiple layers of stochastic (random) variables. The DBN can be trained layer by layer using greedy unsupervised learning. Subsequently, the learned weights of the DBN can be used to provide a pre-trained neural network by determining the optimal initial set of weights for the neural network.

[0166] FIG. 11 illustrates the training and deployment of a deep neural network. Once a given network is built for a task, the neural network is trained using a training dataset (1102). Various training frameworks (1104) have been developed to enable hardware acceleration of the training process. For example, the machine learning framework (604) of FIG. 6 can be configured as a training framework (604). The training framework (604) can be connected to an untrained neural network (1106) and allows the untrained neural network to be trained using the parallel processing resources described herein to generate a trained neural network (1108).

[0167] To initiate the training process, initial weights can be selected randomly or pre-trained using a deep confidence network. Subsequently, training cycles are performed in a supervised or unsupervised manner.

[0168] Supervised learning is a learning method in which training is performed as an mediated operation, such as when the training dataset (1102) contains inputs that are paired with desired outputs for inputs, or when the training dataset contains inputs with known outputs and the output grades of the neural network are manually graded. The network processes the inputs and compares the output of the result to the expected output or the desired set of outputs. Then, the error is backpropagated through the system. The training framework (1104) may be adjusted to adjust the weights controlling the untrained neural network (1106). The training framework (1104) may provide a tool to monitor how well the untrained neural network (1106) converges into a model suitable for generating the correct answer based on known input data. The training process occurs iteratively as the network's weights are adjusted to refine the output generated by the neural network. The training process may continue until the neural network reaches a statistically desired accuracy associated with the trained neural network (1108). Afterward, the trained neural network (1108) can be deployed to implement any number of machine learning operations to generate an inference result (1114) based on the input of new data (1112).

[0169] Unsupervised learning is a learning method in which a network attempts self-learning using unlabeled data. Therefore, for unsupervised learning, the training dataset (1102) will contain input data without any associated output data. The untrained neural network (1106) can learn grouping within the unlabeled inputs and determine how individual inputs relate to the entire dataset. Unsupervised training can be used to generate a self-organizing map, which is a type of trained neural network (1108) capable of performing operations useful for reducing the dimensionality of the data. Unsupervised training can also be used to perform anomaly detection, which can identify data points in the input dataset that deviate from the normal pattern of the data.

[0170] Variations of supervised and unsupervised training may also be used. Semi-supervised learning is a technique that includes a training dataset (1102) containing a mixture of classified and unclassified data of the same distribution. Incremental learning is a variation of supervised learning that continuously uses input data to further train the model. Incremental learning enables the trained neural network (1108) to adapt to new data (1112) without forgetting the knowledge injected into the network during initial training.

[0171] Whether supervised or unsupervised, the training process, particularly for deep neural networks, can be too computationally intensive for a single computing node. Instead of using a single computing node, a distributed network of computing nodes can be used to accelerate the training process.

[0172] FIG. 12 is a block diagram illustrating distributed learning. Distributed learning is a training model that performs supervised or unsupervised training of a neural network using a plurality of distributed computing nodes. Each distributed computing node may include one or more host processors and one or more general-purpose processing nodes, such as the high-performance parallel general-purpose graphics processing unit (700) of FIG. 7, for example. As illustrated, distributed learning may be performed as model parallel processing (1202), data parallel processing (1204), or a combination of model and data parallel processing (1204).

[0173] In model parallel processing (1202), different computation nodes of a distributed system can perform training computations for different parts of a single network. For example, each layer of a neural network can be trained at different processing nodes of a distributed system. The benefits of model parallel processing include the ability to scale particularly to large models. By splitting the computations associated with different layers of a neural network, it is possible to train very large neural networks where the weights of all layers do not fit in the memory of a single computation node. In some cases, model parallel processing can be particularly useful for performing unsupervised learning of large neural networks.

[0174] In data parallel processing (1204), different nodes in a distributed network have complete instances of the model, and each node receives different parts of the data. Then, the results from the different nodes are combined. While various approaches to data parallel processing are possible, all data parallel training approaches require techniques to combine results and synchronize model parameters between each node. Exemplary approaches to data combination include parameter averaging and update-based data parallel processing. Parameter averaging trains each node on a subset of training data and sets global parameters (e.g., weights, biases) as the parameter average of each node. Parameter averaging utilizes a central parameter server that maintains parameter data. Update-based data parallel processing is similar to parameter averaging except that updates to the model are transmitted instead of parameters being transmitted from the nodes to the parameter server. Additionally, update-based data parallel processing can be performed in a distributed manner, where updates are compressed and transmitted between nodes.

[0175] Combined model and data parallel processing (1206) can be implemented, for example, in a distributed system where each computation node includes multiple GPUs. Each node may have a complete instance of the model, having a separate GPU used to train different parts of the model within each node.

[0176] Distributed training has increased overhead compared to training on a single machine. However, the parallel processors and GPGPUs described here can implement various techniques to reduce the overhead of distributed training, including techniques that enable high-bandwidth GPU-GPU data transfer and accelerated remote data synchronization, respectively.

[0177] Exemplary machine learning applications

[0178] Machine learning can be applied to solve a wide range of technical problems, including but not limited to computer vision, autonomous driving and navigation, and speech recognition and language processing. Computer vision has typically been one of the most active research areas within machine learning applications. Applications of computer vision range from replicating human visual capabilities, such as face recognition, to generating new categories of visual capabilities. For example, computer vision applications can be configured to recognize sound waves from vibrations induced in objects visible in video. Parallel processor-accelerated machine learning enables the training of computer vision applications using significantly larger training datasets than previously possible, and allows for the deployment of inference systems using low-power parallel processors.

[0179] Parallel processor-accelerated machine learning includes autonomous driving applications such as lane and road sign recognition, obstacle avoidance, navigation, and driving control. Accelerated machine learning technology can be used to train driving models based on datasets that define appropriate responses to specific training inputs. The parallel processors described herein enable the rapid training of increasingly complex neural networks used in autonomous driving solutions and enable the deployment of low-power inference processors on mobile platforms suitable for integration into autonomous vehicles.

[0180] Parallel processor-accelerated deep neural networks have enabled machine learning approaches to automatic speech recognition (ASR). ASR involves generating a function that computes the most likely language sequence given an input acoustic sequence. Accelerated machine learning using deep neural networks can replace Hidden Markov Models (HMMs) and Gaussian Mixture Models (GMMs) previously used in ASR.

[0181] Parallel processor-accelerated machine learning can also be used to accelerate natural language processing. Automated learning procedures can generate robust models for error- or unfamiliar inputs by utilizing statistical inference algorithms. Exemplary natural language processor applications include automatic machine translation between human languages.

[0182] Parallel processing platforms used for machine learning can be divided into training platforms and batch platforms. Training platforms are generally highly parallel and include optimization functions that accelerate multi-GPU single-node training and multi-node multi-GPU training. Exemplary parallel processors suitable for training include the general-purpose graphics processing unit (700) of FIG. 7 and the multi-GPU computing system (800) of FIG. 8. On the other hand, batch machine learning platforms generally include low-power parallel processors suitable for use in products such as cameras, autonomous robots, and autonomous vehicles.

[0183] FIG. 13 illustrates an exemplary inference SOC (system on a chip) (1300) suitable for performing inference using a trained model. The SOC (1300) may incorporate processing components including a media processor (1302), a vision processor (1304), a GPGPU (1306), and a multi-core processor (1308). The GPGPU (1306) may be a GPGPU as described herein, such as the GPGPU (700), and the multi-core processor (1308) may be a multi-core processor as described herein, such as the multi-core processor (405-406). The SOC (1300) may further include an on-chip memory (1305) that enables a shared on-chip data pool accessible to each processing component. The processing components are optimized for low-power operation to enable deployment to various machine learning platforms, including autonomous vehicles and autonomous robots. For example, one embodiment of the SOC (1300) may be used as part of a main control system for an autonomous vehicle. When the SOC (1300) is configured to be used in an autonomous vehicle, the SOC is designed and configured to comply with the relevant functional safety standards of the jurisdiction in which it is deployed.

[0184] During operation, the media processor (1302) and the vision processor (1304) can work together to accelerate computer vision operations. The media processor (1302) can enable low-latency decoding of multiple high-resolution (e.g., 4K, 8K) video streams. The decoded video streams can be written to a buffer in the on-chip memory (1305). The vision processor (1304) can then perform preliminary processing operations on the frames of the decoded video to parse the decoded video and prepare the frames for processing using a trained image recognition model. For example, the vision processor (1304) can accelerate convolution operations for the CNN used to perform image recognition on high-resolution video data, while back-end model computation is performed by the GPGPU (1306).

[0185] The multi-core processor (1308) may include control logic that assists in the sequencing and synchronization of data transfer and shared memory operations performed by the media processor (1302) and the vision processor (1304). The multi-core processor (1308) may also function as an application processor for executing software applications that can utilize the inference computing capabilities of the GPGPU (1306). For example, at least part of the navigation and driving logic may be implemented as software running on the multi-core processor (1308). This software may issue computational workloads directly to the GPGPU (1306) or may issue computational workloads to the multi-core processor (1308), and the multi-core processor may offload at least part of these operations to the GPGPU (1306).

[0186] The GPGPU (1306) may include a computing cluster, such as a low-power configuration of a processing cluster (706A-706H), within a general-purpose graphics processing unit (700). The computing cluster within the GPGPU (1306) may support instructions specifically optimized to perform inference calculations in a trained neural network. For example, the GPGPU (1306) may support instructions that perform low-precision calculations, such as 8-bit and 4-bit integer vector operations.

[0187] Additional System Overview

[0188] FIG. 14 is a block diagram of a processing system (1400). Elements of FIG. 14 having the same or similar names as elements of any other drawings in this specification may describe the same elements as those in other drawings, may operate or function in a similar manner, may include the same components, and may be linked to other entities such as those described in this specification, though not limited to. The system (1400) may be used in a single-processor desktop system, a multi-processor workstation system, or a server system having multiple processors (1402) or processor cores (1407). The system (1400) may be a processing platform integrated within a system-on-chip (SoC) integrated circuit used in mobile, portable, or embedded devices, such as within an Internet of Things (IoT) device that can be connected wired or wirelessly to a local or wide-area network.

[0189] The system (1400) may be a processing system having components corresponding to the components of FIG. 1. For example, in other configurations, the processor(s) (1402) or processor core(s) (1407) may correspond to the processor(s) (102) of FIG. 1. The graphics processor(s) (1408) may correspond to the parallel processor(s) (112) of FIG. 1. The external graphics processor (1418) may be one of the add-in device(s) (120) of FIG. 1.

[0190] The system (1400) may include, be connected to, or be integrated within a game console, including a server-based gaming platform, a game and media console, a mobile gaming console, a portable game console, or an online game console. The system (1400) is part of a mobile internet-connected device, such as a mobile phone, a smartphone, a tablet computing device, or a laptop with small internal storage capacity. The processing system (1400) may also include, be connected to, or be integrated within a wearable device, such as a smart watch wearable device; smart eyewear or clothing enhanced with augmented reality (AR) or virtual reality (VR) capabilities that provide visual, audio, or haptic output to complement a real visual, audio, or haptic experience, or provide text, audio, graphics, video, holographic images, or video or haptic feedback; other augmented reality (AR) devices; or other virtual reality (VR) devices. The processing system (1400) may include or be part of a television or set-top box device. The system (1400) may include, be connected to, or be integrated within autonomous vehicles such as buses, tractor trailers, automobiles, motor or electric cycles, airplanes or gliders (or any combination thereof). The autonomous vehicle may use the system (1400) to process the environment detected around the vehicle.

[0191] Each of one or more processors (1402) may include one or more processor cores (1407) that process instructions for performing actions on system or user software when executed. In some embodiments, at least one of the one or more processor cores (1407) is configured to process a specific instruction set (1409). The instruction set (1409) may facilitate computing via Complex Instruction Set Computing (CISC), Reduced Instruction Set Computing (RISC), or Very Long Instruction Word (VLIW). One or more processor cores (1407) may process other instruction sets (1409) that may include instructions that facilitate emulation of other instruction sets. The processor cores (1407) may also include other processing devices, such as Digital Signal Processors (DSPs).

[0192] The processor (1402) may include a cache memory (1404). Depending on the architecture, the processor (1402) may have a single internal cache or multiple levels of internal cache. In some embodiments, the cache memory is shared among various components of the processor (1402). In some embodiments, the processor (1402) also uses an external cache (e.g., a Level 3 (L3) cache or an LLC (Last Level Cache)) (not shown), and this external cache may be shared among processor cores (1407) using known cache coherence techniques. A register file (1406) may be additionally included in the processor (1402) and may include different types of registers (e.g., integer registers, floating-point registers, status registers, and instruction pointer registers) that store different types of data. Some registers may be general-purpose registers, but others may be specific to the design of the processor (1402).

[0193] One or more processor(s) (1402) are connected to one or more interface bus(s) (1410) to transmit communication signals, such as addresses, data, or control signals, between the processor(s) (1402) of the system (1400) and other components. The interface bus (1410) may be a processor bus, such as a version of the Direct Media Interface (DMI) bus. However, the processor bus is not limited to a DMI bus and may include one or more Peripheral Component Interconnect (PCI) buses (e.g., PCI, PCI Express), memory buses, or other types of interface buses. For example, the processor(s) (1402) may include an integrated memory controller (1416) and a platform controller hub (1430). The memory controller (1416) facilitates communication between a memory device and other components of the system (1400), and the platform controller hub (PCH) (1430) provides access to I / O devices via a local I / O bus.

[0194] The memory device (1420) may be a dynamic random-access memory (DRAM) device, a static random-access memory (SRAM) device, a flash memory device, a phase-change memory device, or some other memory device having performance suitable for functioning as a process memory. In one embodiment, the memory device (1420) may operate as a system memory for a system (1400) that stores data (1422) and instructions (1421) used by one or more processors (1402) when executing an application or process. The memory controller (1416) is also connected to an optional external graphics processor (1418) that can communicate with one or more graphics processors (1408) within the processor (1402) to perform graphics and media operations. In some embodiments, graphics, media, and computing operations may be supported by an accelerator (1412), which is an auxiliary processor that can be configured to perform a special set of graphics, media, or computing operations. For example, the accelerator (1412) is a matrix multiplication accelerator used to optimize machine learning or computing operations. The accelerator (1412) may be a ray-tracing accelerator that can be used to perform ray-tracing operations in cooperation with the graphics processor (1408). In one embodiment, an external accelerator (1419) may be used instead of or in conjunction with the accelerator (1412).

[0195] A display device (1411) may be connected to a processor(s) (1402). The display device (1411) may be one or more of an internal display device, such as in a mobile electronic device or a laptop device, or an external display device attached via a display interface (e.g., DisplayPort, etc.). The display device (1411) may be a head-mounted display (HMD), such as a stereoscopic display device used in a virtual reality (VR) application or an augmented reality (AR) application.

[0196] The platform controller hub (1430) enables peripheral devices to be connected to the memory device (1420) and the processor (1402) via a high-speed I / O bus. I / O peripheral devices include, but are not limited to, an audio controller (1446), a network controller (1434), a firmware interface (1428), a wireless transceiver (1426), a touch sensor (1425), and a data storage device (1424) (e.g., non-volatile memory, volatile memory, hard disk drive, flash memory, NAND, 3D NAND, 3D XPoint / Optane, etc.). The data storage device (1424) may be connected via a storage interface (e.g., SATA) or via a peripheral device bus such as a PCI bus (e.g., PCI, PCI Express). The touch sensor (1425) may include a touch screen sensor, a pressure sensor, or a fingerprint sensor. The wireless transceiver (1426) may be a Wi-Fi transceiver, a Bluetooth transceiver, or a mobile network transceiver such as a 3G, 4G, 5G, or LTE (Long-Term Evolution) transceiver. The firmware interface (1428) enables communication with system firmware and may be, for example, a unified extensible firmware interface (UEFI). The network controller (1434) may enable network access to a wired network. In some embodiments, a high-performance network controller (not shown) is connected to the interface bus (1410). The audio controller (1446) is a multi-channel high-definition audio controller. In some of these embodiments, the system (1400) includes an optional legacy I / O controller (1440) for connecting legacy (e.g., PS / 2 (Personal System 2)) devices to the system.The platform controller hub (1430) can also be connected to one or more USB (Universal Serial Bus) controllers (1442) to connect to input devices such as a keyboard and mouse combination (1443), a camera (1444), or other USB input devices.

[0197] It will be understood that the illustrated system (1400) is exemplary and not limited, as other types of data processing systems configured differently may also be used. For example, the memory controller (1416) and the platform controller hub (1430) may be integrated into a separate external graphics processor, such as an external graphics processor (1418). The platform controller hub (1430) and / or the memory controller (1416) may be located outside of one or more processor(s) (1402). For example, the system (1400) may include an external memory controller (1416) and a platform controller hub (1430), which may be configured as a memory controller hub and a peripheral device controller hub within a system chipset that communicates with the processor(s) (1402).

[0198] For example, circuit boards ("sleds") equipped with components such as CPUs, memory, and other components are used and can be designed to improve thermal performance. Processing components, such as processors, are located on the top of the sled, while near memory, such as DIMMs, are located on the bottom of the sled. Due to the improved airflow provided by this design, components can operate at higher frequencies and power levels than in typical systems, thereby improving performance. Additionally, since the sleds are configured to be arbitrarily coupled with power and data communication cables in a rack, the ability to quickly remove, upgrade, reinstall, and / or replace them can be enhanced. Similarly, individual components located on the sleds, such as processors, accelerators, memory, and data storage drives, are configured to be easily upgraded as they are spaced further apart from one another. In an exemplary embodiment, the components additionally include hardware authentication features to prove their authenticity.

[0199] Data centers can utilize a single network architecture ("fabric") that supports multiple different network architectures, including Ethernet and Omni-Path. Sleds can be connected to switches via fiber optics, which provides higher bandwidth and lower latency than typical twisted-pair cabling (e.g., Category 5, Category 5e, Category 6, etc.). Due to high-bandwidth, low-latency interconnects and network architectures, data centers pool resources such as memory, accelerators (e.g., GPUs, graphics accelerators, FPGAs, ASICs, neural networks and / or artificial intelligence accelerators, etc.), and physically separated data storage drives during use, and provide computing resources (e.g., processors) to them as needed, allowing access to the pooled resources as if the computing resources were local.

[0200] The power source may provide voltage and / or current to the system (1400) or any component described herein. In one example, the power supply includes an AC-DC (alternating current to direct current) adapter for plugging into a wall outlet. This AC power may be a renewable energy source (e.g., solar power). In one example, the power source includes a DC source, such as an external AC-DC converter. The power source or power supply may include wireless charging hardware that charges by placing it in proximity to a charging field. The power source may include an internal battery, an AC power source, a motion-based power source, a solar power source, or a fuel cell power source.

[0201] FIGS. 15a through 15c illustrate a computing system and a graphics processor. Elements of FIGS. 15a through 15c having the same or similar names as elements of any other figure in this specification describe the same elements as those in the other figures, may operate or function in a similar manner, may include the same components, and may be linked to other entities such as those described in this specification, though not limited to.

[0202] FIG. 15a is a block diagram of a processor (1500) which may be a variant of one of the processors (1402) and may be used in place of one of them. Accordingly, the disclosure of any features combined with the processor (1500) in this specification also indicates a corresponding combination with the processor(s) (1402), but is not limited thereto. The processor (1500) may have one or more processor cores (1502A-1502N), an integrated memory controller (1514), and an integrated graphics processor (1508). If the integrated graphics processor (1508) is excluded, the system including the processor will include a graphics processor device connected within the system chipset or via the system bus. The processor (1500) may include additional cores, including an additional core (1502N) indicated by a dotted box. Each of the processor cores (1502A-1502N) includes one or more internal cache units (1504A-1504N). In some embodiments, each processor core (1502A-1502N) may also access one or more shared cache units (1506). The internal cache units (1504A-1504N) and the shared cache units (1506) represent a cache memory hierarchy within the processor (1500). The cache memory hierarchy may include at least one level of instruction and data cache within each processor core and one or more levels of shared intermediate-level caches, such as level 2 (L2), level 3 (L3), level 4 (L4), or other levels of cache, and the highest level cache prior to external memory is classified as LLC. In some embodiments, cache consistency logic maintains consistency between the various cache units (1506 and 1504A-1504N).

[0203] The processor (1500) may also include a set of one or more bus controller units (1516) and system agent cores (1510). One or more bus controller units (1516) manage a set of peripheral buses, such as one or more PCI or PCI Express buses. The system agent cores (1510) provide management functions for various processor components. The system agent cores (1510) may include one or more integrated memory controllers (1514) that manage access to various external memory devices (not shown).

[0204] For example, one or more processor cores (1502A-1502N) may include support for simultaneous multi-threading. A system agent core (1510) includes components that coordinate and operate the cores (1502A-1502N) during multi-thread processing. The system agent core (1510) may additionally include a power control unit (PCU), which includes logic and components that regulate the power state of the processor cores (1502A-1502N) and the graphics processor (1508).

[0205] The processor (1500) may further include a graphics processor (1508) that executes graphics processing operations. In some of these embodiments, the graphics processor (1508) is connected to a system agent core (1510) that includes a set of shared cache units (1506) and one or more integrated memory controllers (1514). The system agent core (1510) may also include a display controller (1511) that drives the graphics processor output to one or more connected displays. The display controller (1511) may also be a separate module connected to the graphics processor through at least one interconnect, or it may be integrated within the graphics processor (1508).

[0206] A ring-based interconnect unit (1512) may be used to connect internal components of a processor (1500). However, other interconnect units may be used, including point-to-point interconnects, switched interconnects, or other technologies known in the art. In some of these embodiments having a ring-based interconnect (1512), a graphics processor (1508) is connected to the ring interconnect (1512) via an I / O link (1513).

[0207] An exemplary I / O link (1513) represents at least one of a plurality of various I / O interconnects including an on-package I / O interconnect that facilitates communication between various processor components and a high-performance embedded memory module (1518), such as an eDRAM module. Optionally, each processor core (1502A-1502N) and graphics processor (1508) may use the embedded memory module (1518) as a shared LLC (Last Level Cache).

[0208] For example, the processor cores (1502A-1502N) may be homogeneous cores executing the same instruction set architecture. Alternatively, the processor cores (1502A-1502N) may be heterogeneous in terms of instruction set architecture (ISA), where one or more processor cores (1502A-1502N) execute a first instruction set, and at least one of the other cores executes a subset of the first instruction set or another instruction set. The processor cores (1502A-1502N) may be heterogeneous in terms of microarchitecture, where one or more cores consuming relatively high power are connected to one or more power cores consuming low power. In another example, the processor cores (1502A-1502N) may be heterogeneous in terms of computing power. Additionally, the processor (1500) may be implemented on one or more chips having the illustrated components in addition to other components, or implemented as an SoC integrated circuit.

[0209] FIG. 15b is a block diagram of the hardware logic of a graphics processor core (1519) according to some embodiments described herein. A graphics processor core (1519), commonly referred to as a core slice, may be one or more graphics cores within a modular graphics processor. A graphics processor core (1519) is an example of a single graphics core slice, and a graphics processor described herein may include multiple graphics core slices according to target power and performance envelopes. Each graphics processor core (1519) may include a fixed-function block (1530) connected to a plurality of sub-cores (1521A-1521F), also referred to as sub-slices, which include modular blocks of general-purpose and fixed-function logic.

[0210] The fixed-function block (1530) may include a geometry / fixed-function pipeline (1531) that can be shared by all sub-cores of the graphics processor core (1519), for example in a low-performance and / or low-power graphics processor implementation. The geometry / fixed-function pipeline (1531) includes a 3D fixed-function pipeline (e.g., a 3D pipeline (1612) as in FIG. 16a described below), a video front-end unit, a thread generator and a thread dispatcher, and a combined return buffer manager that manages a combined return buffer (e.g., a combined return buffer (1718) in FIG. 17 as described below).

[0211] The fixed function block (1530) may also include a graphics SoC interface (1532), a graphics microcontroller (1533), and a media pipeline (1534). The graphics SoC interface (1532) provides an interface between the graphics processor core (1519) and other processor cores within the system-on-chip integrated circuit. The graphics microcontroller (1533) is a programmable sub-processor that can be configured to manage various functions of the graphics processor core (1519), including thread dispatch, scheduling, and pre-emption. The media pipeline (1534) (e.g., the media pipeline (1616) of FIG. 16a and FIG. 17) includes logic that facilitates the decoding, encoding, pre-processing, and / or post-processing of multimedia data, including image and video data. The media pipeline (1534) implements media operations through requests to computing or sampling logic within the sub-cores (1521A-1521F).

[0212] The SoC interface (1532) enables the graphics processor core (1519) to communicate with other components within the SoC, including a general-purpose application processor core (e.g., CPU) and / or memory hierarchy elements such as shared LLC (last level cache) memory, system RAM and / or embedded on-chip or on-package DRAM. The SoC interface (1532) may also enable communication with fixed-function devices within the SoC, such as a camera imaging pipeline, and enable the use and / or implementation of global memory atoms that can be shared between the graphics processor core (1519) and the CPU within the SoC. The SoC interface (1532) may also implement power management control for the graphics processor core (1519) and enable an interface between the clock domain of the graphics processor core (1519) and other clock domains within the SoC. Optionally, the SoC interface (1532) may enable the reception of command buffers from a command streamer and a global thread dispatcher configured to provide commands and instructions to each of one or more graphics cores within the graphics processor. The commands and instructions may be dispatched to the media pipeline (1534) when a media operation is to be performed, or to the geometry and fixed-function pipeline (e.g., geometry and fixed-function pipeline (1531), geometry and fixed-function pipeline (1537)) when a graphics processing operation is to be performed.

[0213] A graphics microcontroller (1533) may be configured to perform various scheduling and management tasks for a graphics processor core (1519). In one embodiment, the graphics microcontroller (1533) may perform graphics and / or compute workload scheduling on various graphics parallel engines within an array of execution units (EUs) (1522A-1522F, 1524A-1524F) within a sub-core (1521A-1521F). In this workload scheduling, host software running on a CPU core of an SoC containing a graphics processor core (1519) may submit a workload to one of a plurality of graphics processor doorbells, which calls a scheduling operation on the appropriate graphics engine. Scheduling operations include determining the next workload to execute, submitting the workload to the command streamer, preempting existing workloads running in the engine, monitoring the progress of the workload, and notifying the host software when the workload is completed. Optionally, the graphics microcontroller (1533) also facilitates a low-power or idle state of the graphics processor core (1519) so that the graphics processor core (1519) has the ability to save and restore registers within the graphics processor core (1519) independently of the operating system and / or the graphics driver software of the operating system during low-power state transitions.

[0214] The graphics processor core (1519) may have up to N more or fewer modular sub-cores than the illustrated sub-cores (1521A-1521F). In each set of N sub-cores, the graphics processor core (1519) may also include shared function logic (1535), shared and / or cache memory (1536), geometry / fixed function pipeline (1537), and additional fixed function logic (1538) for accelerating various graphics and computing processing operations. The shared function logic (1535) may include logic units (e.g., samplers, arithmetic and / or inter-thread communication logic) associated with the shared function logic (1720) of FIG. 17, which may be shared by each of the N sub-cores within the graphics processor core (1519). The shared and / or cache memory (1536) may be an LLC for a set of N sub-cores (1521A-1521F) within the graphics processor core (1519) and may operate as a shared memory accessible by multiple sub-cores. The geometry / fixed function pipeline (1537) may be included in place of the geometry / fixed function pipeline (1531) within the fixed function block (1530) and may include the same or similar logic unit.

[0215] The graphics processor core (1519) includes additional fixed-function logic (1538) that may include various fixed-function acceleration logic used by the graphics processor core (1519). Optionally, the additional fixed-function logic (1538) includes an additional geometry pipeline used only in position-only shading. In position-only shading, there are two geometry pipelines: a full geometry pipeline within the geometry / fixed-function pipeline (1538, 1531) and a cull pipeline, which is an additional geometry pipeline that may be included within the additional fixed-function logic (1538). For example, the cull pipeline may be a reduced version of the full geometry pipeline. The full pipeline and the cull pipeline may run different instances of the same application, and each instance has a distinct context. Position-only shading can hide long cull runs of discarded triangles, allowing shading to be completed faster in some instances. For example, since the curl pipeline only fetches and shades the position attributes of vertices and does not perform rendering and rasterization of pixels for the frame buffer, the curl pipeline logic within additional fixed-function logic (1538) can run position shaders in parallel with the main application and generally generates critical results faster than the pull pipeline. The curl pipeline can use the generated critical results to calculate visibility information for all triangles, regardless of whether the triangles are culled. The pull pipeline (which may also be called a regeneration pipeline in this case) can use the visibility information to skip culled triangles and shade only the visible triangles that are finally passed to the rasterization stage.

[0216] Optionally, additional fixed-function logic (1538) may include machine-learning acceleration logic, such as fixed-function matrix multiplication logic, for an implementation that includes optimization for machine learning training or inference.

[0217] Each graphics sub-core (1521A-1521F) includes a set of execution resources that can be used to perform graphics, media, and computing operations in response to requests from a graphics pipeline, a media pipeline, or a shader program. The graphics sub-core (1521A-1521F) includes a plurality of EU arrays (1522A-1522F, 1524A-1524F), thread dispatch and inter-thread communication (TD / IC) logic (1523A-1523F), a 3D (e.g., texture) sampler (1525A-1525F), a media sampler (1506A-1506F), a shader processor (1527A-1527F), and a shared local memory (SLM) (1528A-1528F). Each EU array (1522A-1522F, 1524A-1524F) includes multiple execution units, which are general-purpose graphics processing units capable of performing floating-point and integer / fixed-point logic operations during graphics, media, or computing operations, including graphics, media, or computing shader programs. TD / IC logic (1523A-1523F) performs local thread dispatch and thread control operations for execution units within the sub-core and facilitates communication between threads running on the execution units of the sub-core. 3D samplers (1525A-1525F) can read textures or other 3D graphics-related data into memory. 3D samplers can read texture data differently depending on the configured sample state and the texture format associated with a given texture. Media samplers (1506A-1506F) can perform similar read operations depending on the type and format associated with the media data. For example, each graphics sub-core (1521A-1521F) may alternately include integrated 3D and media samplers.Threads executed on execution units within each sub-core (1521A-1521F) use shared local memory (1528A-1528F) within each sub-core, allowing threads executed within a thread group to use a common pool of on-chip memory.

[0218] FIG. 15c is a block diagram of a general-purpose graphics processing unit (GPGPU) (1570) that may be configured as a graphics processor, e.g., a graphics processor (1508) and / or a computing accelerator, according to an embodiment described herein. The GPGPU (1570) may be interconnected with a host processor (e.g., one or more CPUs (1546)) and memory (1571, 1572) via one or more system and / or memory buses. Memory (1571) is system memory that may be shared with one or more CPU(s) (1546), whereas memory (1572) is device memory dedicated to the GPGPU (1570). For example, components and device memory (1572) within the GPGPU (1570) may be mapped to memory addresses accessible by one or more CPU(s) (1546). Access to memory (1571, 1572) can be facilitated through a memory controller (1568). The memory controller (1568) may include an internal DMA (direct memory access) controller (1569) or may include logic for performing operations to be performed by the DMA controller.

[0219] The GPGPU (1570) includes a plurality of cache memories, including an L2 cache (1553), an L1 cache (1554), and an instruction cache (1555), and a shared memory (1556), at least a portion of the shared memory may also be partitioned as cache memory. The GPGPU (1570) also includes a plurality of computing units (1560A-1560N). Each computing unit (1560A-1560N) includes a set of a vector register (1561), a scalar register (1562), a vector logic unit (1563), and a scalar logic unit (1564). The computing units (1560A-1560N) may also include a local shared memory (1565) and a program counter (1566). The computing units (1560A-1560N) may be connected to a constant cache (1567), which may be used to store constant data, which is data that does not change during the execution of a kernel or shader program running on the GPGPU (1570). The constant cache (1567) is a scalar data cache, and the cached data may be fetched directly into a scalar register (1562).

[0220] During operation, one or more CPU(s) (1546) may write commands to registers or memory of the GPGPU (1570) mapped to an accessible address space. A command processor (1557) may read commands from registers or memory and determine how the commands will be processed within the GPGPU (1570). A thread dispatcher (1558) may then dispatch threads to computing units (1560A-1560N) to be used to execute these commands. Each computing unit (1560A-1560N) may execute threads independently of other computing units. Additionally, each computing unit (1560A-1560N) may be configured independently to enable conditional computation and may conditionally output the results of the computation to memory. The command processor (1557) may interrupt one or more CPUs (1546) when the submitted commands are completed.

[0221] FIGS. 16a through 16c illustrate block diagrams of additional graphics processor and computing accelerator architectures provided by embodiments described herein, for example, according to FIGS. 15a through 15c. Elements of FIGS. 16a through 16c having the same or similar names as elements of any other figure in this specification describe the same elements as in the other figures, may operate or function in a similar manner, may include the same components, and may be linked to other entities such as those described herein, but not limited to.

[0222] FIG. 16a is a block diagram of a graphics processor (1600) which may be an individual graphics processing unit, or a graphics processor incorporating multiple processing cores, memory devices, or other semiconductor devices such as but not limited to network interfaces. The graphics processor (1600) may be a variant of the graphics processor (1508) or may be used in place of the graphics processor (1508). Accordingly, any feature disclosed herein in combination with the graphics multiprocessor (1508) also indicates a corresponding combination with the graphics multiprocessor (1600), but is not limited thereto. The graphics processor may communicate with registers on the graphics processor and commands located in processor memory through a memory-mapped I / O interface. The graphics processor (1600) may include a memory interface (1614) for accessing memory. The memory interface (1614) may be an interface to local memory, one or more internal caches, one or more shared external caches, and / or system memory.

[0223] Optionally, the graphics processor (1600) also includes a display controller (1602) that drives display output data to a display device (1618). The display controller (1602) includes hardware for configuring one or more overlay planes for display and multiple layers of video or user interface elements. The display device (1618) may be an internal or external display device. In one embodiment, the display device (1618) is a head-mounted display device, such as a virtual reality (VR) display device or an augmented reality (AR) display device. The graphics processor (1600) may include a video codec engine (1606) that encodes, decodes, or transcodes media from or between one or more media encoding formats, including but not limited to MPEG (Moving Picture Experts Group) formats such as MPEG-2, H.264 / MPEG-4 AVC, H.265 / HEVC, AOMedia (Alliance for Open Media) VP8, VP9, ​​and SMPTE (Society of Motion Picture & Television Engineers) 421M / VC-1, and JPEG formats such as JPEG (Joint Photographic Experts Group) and MJPEG (Motion JPEG).

[0224] The graphics processor (1600) may include a block image transfer (BLIT) engine (1604) that performs two-dimensional (2D) rasterization operations, for example, including bit-bounded block transfers. However, alternatively, 2D graphics operations may be performed using one or more components of a graphics processing engine (GPE) (1610). In some embodiments, the GPE (1610) is a computing engine that performs graphics operations, including three-dimensional (3D) graphics operations and media operations.

[0225] The GPE (1610) may include a 3D pipeline (1612) that performs 3D operations, such as rendering 3D images and scenes, using processing functions that act on 3D primitive shapes (e.g., rectangles, triangles, etc.). The 3D pipeline (1612) includes programmable and fixed functional elements that perform various tasks within the elements and / or create execution threads in the 3D / media sub-system (1615). While the 3D pipeline (1612) may be used to perform media operations, embodiments of the GPE (1610) also include a media pipeline (1616) that is particularly used to perform media operations, such as video post-processing and image enhancement.

[0226] The media pipeline (1616) may include a fixed function or programmable logic unit that performs one or more special media operations, such as video decoding acceleration, video deinterlacing, and video encoding acceleration, on behalf of or for the video codec engine (1606). The media pipeline (1616) may further include a thread creation unit that creates a thread that runs on the 3D / media sub-system (1615). The created thread performs calculations for media operations on one or more graphics execution units included in the 3D / media sub-system (1615).

[0227] The 3D / media sub-system (1615) may include logic for executing threads generated by the 3D pipeline (1612) and the media pipeline (1616). The pipeline may send thread execution requests to the 3D / media sub-system (1615), and the 3D / media sub-system includes thread dispatch logic for mediating and sending various requests for available thread execution resources. Execution resources include an array of graphics execution units that process 3D and media threads. The 3D / media sub-system (1615) may include one or more internal caches for thread instructions and data. The 3D / media sub-system (1615) may also include shared memory including registers and addressable memory that share data among threads and store output data.

[0228] FIG. 16b illustrates a graphic processor (1620) that can be used instead of the graphic processor (1600) as a variation of the graphic processor (1600) (and vice versa). Accordingly, any disclosure of features combined with the graphic processor (1600) in this specification also indicates a corresponding combination with the graphic processor (1620), but is not limited thereto. The graphic processor (1620) has a tiled architecture according to the embodiments described in this specification. The graphic processor (1620) may include a graphic processing engine cluster (1622) having a plurality of instances of the graphic processing engine (1610) of FIG. 16a within graphic engine tiles (1610A-1610D). Each graphic engine tile (1610A-1610D) may be interconnected through a set of tile interconnects (1623A-1623F). Each graphics engine tile (1610A-1610D) may also be connected to a memory module or memory device (1626A-1626D) via a memory interconnect (1625A-1625D). The memory device (1626A-1626D) may use any graphics memory technology. For example, the memory device (1626A-1626D) may be GDDR memory. The memory device (1626A-1626D) may be a high bandwidth memory (HBM) module that may be in an on-die format together with each graphics engine tile (1610A-1610D). The memory device (1626A-1626D) may be a stacked memory device that may be stacked on top of each graphics engine tile (1610A-1610D). Each graphics engine tile (1610A-1610D) and associated memory (1626A-1626D) may be located on individual chiplets bonded to a base die or base substrate, as described in more detail in FIGS. 24b through 24d.

[0229] The graphics processor (1620) may be configured as a non-uniform memory access (NUMA) system, wherein memory devices (1626A-1626D) are connected to associated graphics engine tiles (1610A-1610D). A given memory device may be accessed by graphics engine tiles rather than directly connected tiles. However, access latency for memory devices (1626A-1626D) may be lowest when accessing local tiles. In one embodiment, a cache coherent NUMA (ccNUMA) system using tile interconnects (1623A-1623F) that enable communication between cache controllers within graphics engine tiles (1610A-1610D) can maintain a consistent memory image when more than one cache stores the same memory location.

[0230] The graphics processing engine cluster (1622) may be connected to an on-chip or on-package fabric interconnect (1624). The fabric interconnect (1624) may enable communication between components such as graphics engine tiles (1610A-1610D), a video codec (1606), and one or more copy engines (1604). The copy engine (1604) may be used to move data from, to, or between memory devices (1626A-1626D) and memory (e.g., system memory) located outside the graphics processor (1620). The fabric interconnect (1624) may also be used to interconnect the graphics engine tiles (1610A-1610D). The graphics processor (1620) may optionally include a display controller (1602) that enables connection with an external display device (1618). The graphics processor may also be configured as a graphics or computing accelerator. In the accelerator configuration, the display controller (1602) and the display device (1618) may be omitted.

[0231] The graphics processor (1620) may be connected to a host system via a host interface (1628). The host interface (1628) may enable communication between the graphics processor (1620), system memory, and / or other system components. The host interface (1628) may be, for example, a PCI Express bus or another type of host system interface.

[0232] FIG. 16c illustrates a computing accelerator (1630) according to an embodiment described herein. The computing accelerator (1630) may have structural similarity to the graphics processor (1620) of FIG. 3b and is optimized for computing acceleration. A computing engine cluster (1632) may include a set of computing engine tiles (1640A-1640D) containing execution logic optimized for parallel or vector-based general-purpose computing operations. In some embodiments, one or more computing engine tiles (1640A-1640D) may include logic for performing media acceleration, but computing engine tiles (1640A-1640D) may not include fixed-function graphics processing logic. The computing engine tiles (1640A-1640D) may access memory (1626A-1626D) through memory interconnects (1625A-1625D). The memory (1626A-1626D) and memory interconnects (1625A-1625D) may be similar to the technology in the graphics processor (1620) or may be different. The graphics computing engine tiles (1640A-1640D) may also be interconnected via a set of tile interconnects (1623A-1623F) and may be connected to and / or interconnected by the fabric interconnects (1624). In one embodiment, the computing accelerator (1630) includes a large L3 cache (1636) which may be configured as a cache for the entire device. The computing accelerator (1630) may also be connected to the host processor and memory via a host interface (1628) in a manner similar to the graphics processor (1620) of FIG. 16b.

[0233] graphics processing engine

[0234] FIG. 17 is a block diagram of a graphics processing engine (1710) of a graphics processor according to some embodiments. The graphics processing engine (GPE) (1710) may be a version of the GPE (1610) shown in FIG. 16a and may also represent the graphics engine tiles (1610A-1610D) of FIG. 16b. Elements of FIG. 17 having the same or similar names as elements of any other drawings of this specification may describe the same elements as those in other drawings, may operate or function in a similar manner, may include the same components, and may be linked to other entities such as those described in this specification, though not limited to them. For example, the 3D pipeline (1612) and media pipeline (1616) of FIG. 16a are shown in FIG. 17. The media pipeline (1616) is optional in some embodiments of the GPE (1710) and may not be explicitly included within the GPE (1710). For example, in at least one embodiment, a separate media and / or image processor is connected to the GPE (1710).

[0235] The GPE (1710) is connected to or includes a command streamer (1703) that provides a command stream to the 3D pipeline (1612) and / or the media pipeline (1616). In addition to or instead of, the command streamer (1703) may be directly connected to an integrated return buffer (1718). The integrated return buffer (1718) may be communicably connected to a graphics core array (1714). Optionally, the command streamer (1703) is connected to memory, which may be system memory or one or more internal cache memory and shared cache memory. The command streamer (1703) may receive commands from memory and transmit the commands to the 3D pipeline (1612) and / or the media pipeline (1616). A command is a directive fetched from a ring buffer, and the ring buffer stores commands for the 3D pipeline (1612) and the media pipeline (1616). The ring buffer may additionally include a batch command buffer that stores a batch of multiple commands. A command for the 3D pipeline (1612) may also include references to data stored in memory, such as but not limited to vertex and geometric data for the 3D pipeline (1612) and / or image data and memory objects for the media pipeline (1616). The 3D pipeline (1612) and the media pipeline (1616) process the commands and data by performing operations through logic within their respective pipelines or by dispatching one or more execution threads to the graphics core array (1714). The graphics core array (1714) may include one or more blocks of graphics cores (e.g., graphics core(s) (1715A), graphics core(s) (1715B)), and each block includes one or more graphics cores.Each graphics core includes general-purpose and graphics execution logic that performs graphics and computing operations, and a set of graphics execution resources including fixed-function texture processing and / or machine learning and artificial intelligence acceleration logic.

[0236] In various embodiments, the 3D pipeline (1612) may include fixed functions and programmable logic for processing one or more shader programs, such as vertex shaders, geometry shaders, pixel shaders, fragment shaders, compute shaders, or other shader programs, by processing instructions and dispatching execution threads to the graphics core array (1714). The graphics core array (1714) provides a combined block of execution resources used to process these shader programs. The multi-purpose execution logic (e.g., execution units) within the graphics core(s) (1715A, 1715B) of the graphics core array (1714) may support various 3D API shader languages ​​and execute multiple concurrent execution threads associated with multiple shaders.

[0237] The graphics core array (1714) may also include execution logic that performs media functions such as video and / or image processing. The execution unit may include general-purpose logic that can be programmed to perform parallel general-purpose computing operations in addition to graphics processing operations. The general-purpose logic may perform processing operations in parallel with or together with the general-purpose logic within the processor core(s) (1407) of FIG. 14 or the core (1502A-1502N) as in FIG. 15a.

[0238] Output data generated by a thread running on the graphics core array (1714) can be output to memory in the integrated return buffer (URB) (1718). The URB (1718) can store data for multiple threads. The URB (1718) can be used to transfer data between different threads running on the graphics core array (1714). The URB (1718) can also be additionally used for synchronization between threads on the graphics core array (1714) and fixed function logic within the shared function logic (1720).

[0239] Optionally, the graphics core array (1714) is scalable, so the array includes a variable number of graphics cores, and each graphics core has a variable number of execution units according to the target power and performance level of the GPE (1710). Since the execution resources are dynamically scalable, the execution resources can be enabled or disabled as needed.

[0240] A graphics core array (1714) is connected to a shared function logic (1720) that includes a plurality of resources shared among the graphics cores of the graphics core array. The shared function within the shared function logic (1720) is a hardware logic unit that provides special supplementary functions to the graphics core array (1714). In various embodiments, the shared function logic (1720) includes, but is not limited to, sampler (1721), arithmetic (1722), and inter-thread communication (ITC) (1723) logic. Additionally, one or more cache(s) (1725) may be implemented within the shared function logic (1720).

[0241] A shared function is implemented at least when the requirement for a given special function is insufficient to be included within the graphics core array (1714). Instead, a single instantiation of that special function is implemented as an independent entity of the shared function logic (1720) and is shared among the execution resources within the graphics core array (1714). The exact set of functions shared among the graphics core arrays (1714) and included within the graphics core array (1714) varies according to the embodiment. A specific shared function within the shared function logic (1720) that is extensively used by the graphics core array (1714) may be included within the shared function logic (1716) within the graphics core array (1714). Optionally, the shared function logic (1716) within the graphics core array (1714) may include some or all of the logic within the shared function logic (1720). All logic elements within the shared function logic (1720) may be duplicated within the shared function logic (1716) of the graphics core array (1714). Alternatively, the shared function logic (1720) is excluded for the shared function logic (1716) within the graphics core array (1714).

[0242] Execution unit

[0243] FIGS. 18a and 18b illustrate thread execution logic (1800) comprising an array of processing elements used in a graphics processor core according to an embodiment described herein. The elements of FIGS. 18a through 18b having the same or similar names as the elements of any other figure in the specification describe the same elements as those in the other figures, may operate or function in a similar manner, may include the same components, and may be linked to other entities such as those described herein, though not limited to. FIGS. 18a and 18b illustrate an overview of thread execution logic (1800), which may represent hardware logic illustrated as each sub-core (1521A-1521F) of FIG. 18a illustrates an execution unit within a general-purpose graphics processor, and FIG. 18b illustrates an execution unit that may be used within a computing accelerator.

[0244] As illustrated in FIG. 18a, the thread execution logic (1800) includes a shader processor (1802), a thread dispatcher (1804), an instruction cache (1806), an expandable execution unit array comprising a plurality of execution units (1808A-1808N), a sampler (1810), shared local memory (1811), a data cache (1812), and a data port (1814). Optionally, the expandable execution unit array may be dynamically expanded by enabling or disabling one or more execution units (e.g., any execution units (1808A, 1808B, 1808C, 1808D, to 1808N-1 and 1808N)) according to the computational requirements of the workload. The included components may be interconnected through an interconnection fabric connected to each component. The thread execution logic (1800) may include one or more connections to memory, such as system memory or cache memory, through one or more of the instruction cache (1806), data port (1814), sampler (1810), and execution units (1808A-1808N). Each execution unit (e.g., 1808A) may be a standalone, programmable general-purpose computing unit capable of executing multiple concurrent hardware threads while processing multiple data elements in parallel for each thread. In various embodiments, an array of execution units (1808A-1808N) may be extended to include any number of individual execution units.

[0245] Execution units (1808A-1808N) can be used primarily to execute shader programs. A shader processor (1802) can process various shader programs and dispatch execution threads associated with shader programs via a thread dispatcher (1804). The thread dispatcher may include logic for mediating thread initiation requests from the graphics and media pipelines and instantiating the requested threads on one or more execution units (1808A-1808N). For example, a geometry pipeline may dispatch vertex, tessellation, or geometry shaders to thread execution logic for processing. Optionally, the thread dispatcher (1804) may also process requests for runtime thread creation from execution shader programs.

[0246] Execution units (1808A–1808N) can support instruction sets that include basic support for many standard 3D graphics shader instructions so that shader programs from graphics libraries (e.g., Direct 3D and OpenGL) can be executed with minimal translation. Execution units support vertex and geometry processing (e.g., vertex programs, geometry programs, vertex shaders), pixel processing (e.g., pixel shaders, fragment shaders), and general-purpose processing (e.g., compute and media shaders). Each execution unit (1808A–1808N) is capable of Multiple Issue Single Instruction Multiple Data (SIMD) execution, and multi-threaded operation enables an efficient execution environment despite memory access with longer latency. Each hardware thread within each execution unit has a dedicated high-bandwidth register file and an associated independent thread state. Execution is issued multiple times per clock to a pipeline capable of performing integer, single-precision and double-precision floating-point operations, SIMD branch functions, logical operations, transcendental operations, and other operations. While waiting for data from memory or from one of the shared functions, dependent logic within the execution unit (1808A-1808N) puts the waiting thread to sleep until the requested data is returned. While the waiting thread is in a sleep state, hardware resources may be used to process other threads. For example, during the delay associated with vertex shader operations, the execution unit may perform operations on other types of shader programs, including pixel shaders, fragment shaders, or other vertex shaders such as the vertex shader (2107) shown in FIG. 21. Various embodiments may be executed by using SIMT (Single Instruction Multiple Thread) instead of SIMD or in addition to the use of SIMD.References to SIMD cores or operations may also be applied to SIMT or to SIMD together with SIMT.

[0247] Each execution unit of the execution unit (1808A-1808N) operates on an array of data elements. The number of data elements is the "execution size" or the number of channels for an instruction. An execution channel is a logical unit of execution for accessing data elements within an instruction, masking, and flow control. The number of channels may be independent of the number of physical arithmetic logic units (ALUs) or floating-point units (FPUs) or other logic units (e.g., tensor cores, ray tracing cores, etc.) for a particular graphics processor. Additionally, the execution units (1808A-1808N) support integer and floating-point data types.

[0248] The execution unit instruction set includes SIMD instructions. Various data elements can be stored as data types packed in registers, and the execution unit will process various elements depending on the data size of the elements. For example, when operating on a 256-bit wide vector, the 256-bit vector is stored in registers, and the execution unit operates on the vector as four individual 64-bit packed data elements (Quad-Word (QW) size data elements), eight individual 32-bit packed data elements (DW (Double Word) size data elements), sixteen individual 16-bit packed data elements (Word (W) size data elements), or thirty-two individual 8-bit packed data elements (byte (B) size data elements). However, other vector widths and register sizes are also possible.

[0249] Optionally, one or more execution units may be combined into a combined execution unit (EU) (1809A-1809N) having thread control logic (1807A-1807N) common to the combined EU. Multiple EUs may be combined into an EU group. Each EU in a combined EU group may be configured to execute a separate SIMD hardware thread. The number of EUs in a combined EU group may vary depending on the embodiment. Additionally, various SIMD widths, including but not limited to SIMD8, SIMD16, and SIMD32, may be performed for each EU. Each combined graphics execution unit (1809A-1809N) includes at least two execution units. For example, a combined execution unit (1809A) includes a first EU (1808A), a second EU (1808B), and thread control logic (1807A) common to the first EU (1808A) and the second EU (1808B). Thread control logic (1807A) controls threads executed on the combined graphics execution unit (1809A) so that each EU within the combined execution unit (1809A-1809N) can be executed using a common instruction pointer register.

[0250] One or more internal instruction caches (e.g., 1806) are included in the thread execution logic (1800) to cache thread instructions for the execution unit. One or more data caches (e.g., 1812) that cache thread data while the thread is running may be included in the thread execution logic (1800). A thread running on the execution logic (1800) may also store explicitly managed data in shared local memory (1811). A sampler (1810) may be included to provide texture sampling for 3D motion and media sampling for media motion. The sampler (1810) may include a special texture or media sampling function that processes texture or media data during the sampling process before providing the sampled data to the execution unit.

[0251] During execution, the graphics and media pipeline sends a thread initiation request to the thread execution logic (1800) via the thread creation and dispatch logic. Once a group of geometric objects is processed and rasterized into pixel data, the pixel processor logic (e.g., pixel shader logic, fragment shader logic, etc.) within the shader processor (1802) is called to further calculate output information and write the result to an output surface (e.g., color buffer, depth buffer, stencil buffer, etc.). The pixel shader or fragment shader can calculate the values ​​of various vertex attributes to be interpolated across the rasterized objects. Subsequently, the pixel processor logic within the shader processor (1802) can execute an application programming interface (API)-provided pixel or fragment shader program. To execute the shader program, the shader processor (1802) dispatches the thread to an execution unit (e.g., 1808A) via the thread dispatcher (1804). The shader processor (1802) can access texture data of a texture map stored in memory using the texture sampling logic of the sampler (1810). Arithmetic operations on the texture data and input geometric data calculate pixel color data for each geometric fragment, or discard one or more pixels in other processing.

[0252] Additionally, the data port (1814) may provide a memory access mechanism for thread execution logic (1800) that outputs processed data to memory for further processing on the graphics processor output pipeline. The data port (1814) includes or is connected to one or more cache memories (e.g., data cache (1812)) that cache data for memory access through the data port (1814).

[0253] Optionally, the execution logic (1800) may also include a ray tracer (1805) capable of providing ray tracing acceleration capabilities. The ray tracer (1805) may support a ray tracing instruction set containing instructions / functions for generating rays. The ray tracing instruction set may be similar to or different from the ray tracing instruction set supported by the ray tracing core (372) of FIG. 3c.

[0254] FIG. 18b illustrates exemplary internal details of an execution unit (1808). The graphics execution unit (1808) may include an instruction fetch unit (1837), a general register file (GRF) array (1824), an architecture register file (ARF) array (1826), a thread arbitrator (1822), a transmit unit (1830), a branch unit (1832), a set of SIMD floating point units (1834), and optionally a set of dedicated integer SIMD ALUs (1835). The GRF (1824) and ARF (1826) include sets of general register files and architecture register files associated with each concurrent hardware thread that can be activated in the graphics execution unit (1808). The structural state per thread is maintained in the ARF (1826), and data used during the execution of the thread may be stored in the GRF (1824). The execution state of each thread, including the instruction pointer for each thread, can be held in the thread-specific register of the ARF (1826).

[0255] The graphics execution unit (1808) may have an architecture that is a combination of Simultaneous Multi-Threading (SMT) and fine-grained Interleaved Multi-Threading (IMT). This architecture may have a modular configuration that can be fine-tuned at design time according to the target number of concurrent threads and the number of registers per execution unit, wherein execution unit resources are distributed across the logic used to execute multiple concurrent threads. The number of logic threads that can be executed by the graphics execution unit (1808) is not limited to the number of hardware threads, and multiple logic threads may be assigned to each hardware thread.

[0256] Optionally, the graphics execution unit (1808) may co-issue multiple instructions, each of which may be different instructions. The thread arbitrator (1822) of the graphics execution unit thread (1808) may dispatch instructions for execution to one of the transfer unit (1830), the branch unit (1832), or the SIMD FPU(s) (1834). Each execution thread may access 128 general-purpose registers within the GRF (1824), each register may store 32 bytes accessible as a SIMD 8-element vector of 32-bit data elements. Each execution unit thread may access 4KB within the GRF (1824), but embodiments are not limited thereto, and more or fewer register resources may be provided in other embodiments. The graphics execution unit (1808) is divided into seven hardware threads capable of performing computing operations independently, but the number of threads per execution unit may also vary depending on the embodiment, for example, up to 16 hardware threads may be supported. In an embodiment where seven threads can access 4KB, the GRF (1824) can store a total of 28KB. In another embodiment, where 16 threads can access 4KB, the GRF (1824) can store a total of 64KB. However, the number of threads per execution unit is not limited to these examples and may be more or less than the given number. A flexible addressing mode can address registers together to effectively form a wider register or represent a strided rectangular block data structure.

[0257] In addition to, or instead of, memory operations, sampler operations, and other long-latency system communications are dispatched via a "transmission" command executed by a message transmission unit (1830). Branch instructions are dispatched to a dedicated branch unit (1832) to facilitate SIMD divergence and final convergence.

[0258] The graphics execution unit (1808) includes one or more SIMD floating-point units (FPUs) (1834) that perform floating-point operations. The FPU(s) (1834) may also support integer calculations. In some examples, the FPU(s) (1834) may perform SIMD operations up to M 32-bit floating-point (or integer) operations, or SIMD operations up to 2M 16-bit integer or 16-bit floating-point operations. Optionally, at least one of the FPU(s) provides extended arithmetic capabilities supporting high-throughput transcendental arithmetic functions and double-precision 64-bit floating-point. A set of 8-bit integer SIMD ALUs (1835) is also present and may be specifically optimized to perform operations related to machine learning calculations.

[0259] Optionally, an array of multiple instances of the graphics execution unit (1808) may be instantiated in a graphics sub-core group (e.g., a sub-slice). For scalability, the product designer may select the exact number of execution units per sub-core group. The execution unit (1808) may execute instructions across multiple execution channels. Each thread executed on the graphics execution unit (1808) may be executed on a different channel.

[0260] FIG. 19 illustrates another exemplary execution unit (1900). The elements of FIG. 19 having the same or similar names as the elements of any other drawings of this specification describe the same elements as those in the other drawings, may operate or function in a similar manner, may include the same components, and may be linked to other entities such as those described in this specification, though not limited to them. The execution unit (1900) may be a computationally optimized execution unit for use in computing engine tiles (1640A-1640D), for example, as in FIG. 16c, but is not limited thereto. The execution unit (1900) may also be used in graphics engine tiles (1610A-1610D), as in FIG. 16b. The execution unit (1900) may include a thread control unit (1901), a thread state unit (1902), an instruction fetch / prefetch unit (1903), and an instruction decoding unit (1904). The execution unit (1900) may additionally include a register file (1906) that stores registers that can be allocated to hardware threads within the execution unit. The execution unit (1900) may additionally include a transmission unit (1907) and a branch unit (1908). The transmission unit (1907) and the branch unit (1908) may operate similarly to the transmission unit (1830) and the branch unit (1832) of the graphics execution unit (1808) of FIG. 18b.

[0261] The execution unit (1900) may also include a computing unit (1910) comprising a plurality of different types of functional units. The computing unit (1910) may also include an ALU unit (1911) comprising an array of arithmetic logic units. The ALU unit (1911) may be configured to perform 64-bit, 32-bit, and 16-bit integer and floating-point operations. Integer and floating-point operations may be performed simultaneously. The computing unit (1910) may also include a systolic array (1912) and an arithmetic unit (1913). The systolic array (1912) includes a W (wide) and D (deep) network of data processing units that can be used to perform vector or other data-parallel operations in a systolic manner. The systolic array (1912) may be configured to perform matrix operations such as matrix inner product operations. The systolic array (1912) supports 8-bit and 4-bit integer operations as well as 16-bit floating-point operations. The systolic array (1912) may be configured to accelerate machine learning operations. The systolic array (1912) may be configured to support the bfloat16, 16-bit floating-point format. The operation unit (1913) may be included to perform a specific subset of mathematical operations in a more efficient and low-power manner than the ALU unit (1911). The arithmetic unit (1913) may include arithmetic logic that can be found in the shared function logic of the graphics processing engine provided in other embodiments (e.g., the arithmetic logic (1722) of the shared function logic (1720) of FIG. 17). The arithmetic unit (1913) may be configured to perform 32-bit and 64-bit floating-point operations.

[0262] The thread control unit (1901) includes logic for controlling the execution of threads within the execution unit. The thread control unit (1901) may include thread arbitration logic for starting, stopping, and preempting the execution of threads within the execution unit (1900). The thread state unit (1902) may be used to store the thread state for threads assigned to be executed on the execution unit (1900). By storing the thread state within the execution unit (1900), threads can be quickly preempted when they are blocked or idle. The instruction fetch / prefetch unit (1903) may fetch instructions from the instruction cache of a higher-level execution logic (e.g., the instruction cache (1806) as in FIG. 18a). The instruction fetch / prefetch unit (1903) may also issue a prefetch request to allow instructions to be loaded into the instruction cache based on an analysis of the currently executing thread. The instruction decoding unit (1904) can be used to decode instructions to be executed by the computing unit. The instruction decoding unit (1904) can be used as a secondary decoder to decode complex instructions into micro-operations that make up the structure.

[0263] The execution unit (1900) further includes a register file (1906) that can be used by a hardware thread executed on the execution unit (1900). The registers of the register file (1906) may be divided across logic used to execute multiple concurrent threads within the computing unit (1910) of the execution unit (1900). The number of logical threads that can be executed by the graphics execution unit (1900) is not limited to the number of hardware threads, and multiple logical threads may be assigned to each hardware thread. The size of the register file (1906) may vary from embodiment to embodiment depending on the number of supported hardware threads. Register renaming may be used to dynamically assign registers to hardware threads.

[0264] FIG. 20 is a block diagram illustrating a graphics processor instruction format (2000). A graphics processor execution unit supports an instruction set having multiple formats of instructions. Solid boxes generally represent components included in the execution unit instructions, while dotted lines represent components that are optional or included only in a subset of instructions. The instruction format (2000) described and illustrated is a macro-instruction in that, unlike micro-operations derived from decoded instructions once the instruction is processed, it is an instruction provided to the execution unit.

[0265] The graphics processor execution unit can basically support instructions in a 128-bit instruction format (2010). A 64-bit compressed instruction format (2030) is available for some instructions depending on the number of selected instructions, instruction options, and operands. While the basic 128-bit instruction format (2010) provides access to all instruction options, some options and operations are limited to the 64-bit format (2030). The basic instructions available in the 64-bit format (2030) vary depending on the embodiment. Instructions are partially compressed using a set of index values ​​in the index field (2013). The execution unit hardware refers to a set of compression tables based on the index values ​​and uses the compression table output to reconstruct the basic instructions into the 128-bit instruction format (2010). Instructions of different sizes and formats may be used.

[0266] For each format, the instruction opcode (2012) defines the operation to be performed by the execution unit. The execution unit executes each instruction in parallel across multiple data elements of each operand. For example, in response to an add instruction, the execution unit performs simultaneous addition operations across each color channel representing a texture element or image element. Basically, the execution unit executes each instruction across all data channels of the operand. The instruction control field (2014) enables control over specific execution options, such as channel selection (e.g., prediction) and data channel order (e.g., swizzle). For instructions in the 128-bit instruction format (2010), the execution-size field (2016) limits the number of data channels to be executed in parallel. The execution-size field (2016) is not available in the 64-bit compressed instruction format (2030).

[0267] Some execution unit instructions have up to three operands, including two source operands SRC0 (2020) and SRC1 (2022) and one destination (2018). The execution unit may support dual destination instructions, one of which destination is implied. Data manipulation instructions may have a third source operand (e.g., SRC2 (2024)), where the instruction operation code (2012) determines the number of source operands. The last source operand of an instruction may be an immediate (e.g., hard-coded) value passed along with the instruction.

[0268] The 128-bit instruction format (2010) includes an access / address mode field (2026) that specifies, for example, whether a direct register addressing mode or an indirect register addressing mode is used. When a direct register addressing mode is used, the register addresses of one or more operands are provided directly by the bits of the instruction.

[0269] A 128-bit instruction format (2010) may include an access / address mode field (2026) that specifies an address mode and / or access mode for the instruction. The access mode may be used to define data access alignment for the instruction. Access modes including a 16-byte aligned access mode and a 1-byte aligned access mode may be supported, wherein the byte alignment of the access mode determines the access alignment of the instruction operands. For example, when in the first mode, the instruction may use 1-byte aligned addressing for source and destination operands, and when in the second mode, the instruction may use 16-byte aligned addressing for all source and destination operands.

[0270] The address mode portion of the access / address mode field (2026) can determine whether the instruction should use direct or indirect addressing. When the direct register addressing mode is used, the bits of the instruction directly provide the register addresses of one or more operands. When the indirect register addressing mode is used, the register addresses of one or more operands can be calculated based on the address register value and the address immediate field of the instruction.

[0271] Instructions may be grouped according to an operation code (2012) bit field that simplifies operation code decoding (2040). For an 8-bit operation code, bits 4, 5, and 6 allow the execution unit to determine the type of operation code. The specific operation code groups described are merely examples. The move and logic operation code group (2042) may include data move and logic instructions (e.g., move (mov), compare (cmp)). The move and logic group (2042) may share five most significant bits (MSB), where the move (mov) instruction is in the form 0000xxxxb and the logic instruction is in the form 0001xxxxb. The flow control instruction group (2044) (e.g., call, jump (jmp)) includes instructions in the form 0010xxxxb (e.g., 0x20). The other instruction group (2046) includes a mixture of instructions including synchronization instructions (e.g., wait, transmit) of the form 0011xxxxb (e.g., 0x30). The parallel arithmetic instruction group (2048) includes component-specific arithmetic instructions (e.g., add, multiply (mul)) of the form 0100xxxxb (e.g., 0x40). The parallel arithmetic group (2048) performs arithmetic operations in parallel across data channels. The vector arithmetic group (2050) includes arithmetic instructions (e.g., dp4) of the form 0101xxxxb (e.g., 0x50). The vector arithmetic group performs arithmetic operations such as inner product calculations on vector operands. In one embodiment, the described operation code decoding (2040) may be used to determine which part of the execution unit is used to execute the decoded instruction. For example, some instructions may be designated as systolic instructions to be executed by a systolic array. Other instructions, such as ray tracing instructions (not shown), may be routed to a ray tracing core or ray tracing logic within a slice or partition of execution logic.

[0272] Graphics Pipeline

[0273] FIG. 21 is a block diagram of a graphics processor (2100) according to another embodiment. The elements of FIG. 21 having the same or similar names as the elements of any other drawings in this specification describe the same elements as those in the other drawings, may operate or function in a similar manner, may include the same components, and may be linked to other entities such as those described in this specification, though not limited to.

[0274] The graphics processor (2100) may include different types of graphics processing pipelines, such as a geometry pipeline (2120), a media pipeline (2130), a display engine (2140), thread execution logic (2150), and a rendering output pipeline (2170). The graphics processor (2100) may be a graphics processor within a multi-core processing system that includes one or more general-purpose processing cores. The graphics processor may be controlled by register writing to one or more control registers (not shown) or by commands issued to the graphics processor (2100) through a ring interconnect (2102). The ring interconnect (2102) may connect the graphics processor (2100) to other processing components, such as other graphics processors or general-purpose processors. Commands from the ring interconnect (2102) are interpreted by a command streamer (2103), and this command streamer provides commands to individual components of the geometric pipeline (2120) or the media pipeline (2130).

[0275] The command streamer (2103) can read vertex data from memory and direct the operation of the vertex fetcher (2105) to execute the vertex-processing command provided by the command streamer (2103). The vertex fetcher (2105) can provide the vertex data to the vertex shader (2107), and the vertex shader (2107) performs coordinate space transformation and lighting operations on each vertex. The vertex fetcher (2105) and the vertex shader (2107) can execute the vertex-processing command by dispatching the execution thread to the execution unit (2152A, 2152B) through the thread dispatcher (2131).

[0276] The execution units (2152A, 2152B) may be an array of vector processors having instruction sets for performing graphics and media operations. The execution units (2152A, 2152B) may have an attached L1 cache (2151) that is specific to each array or shared between the arrays. The cache may consist of a data cache, an instruction cache, or a single cache partitioned to contain data and instructions in different partitions.

[0277] The geometry pipeline (2120) may include a tessellation component that performs hardware-accelerated tessellation of 3D objects. A programmable hull shader (2111) may configure the tessellation operation. A programmable domain shader (2117) may provide back-end evaluation of the tessellation output. A tessellator (2113) may include special-purpose logic that operates in the direction of the hull shader (2111) and generates a set of detailed geometric objects based on a coarse geometric model provided as input to the geometry pipeline (2120). Additionally, if tessellation is not used, the tessellation component (e.g., hull shader (2111), tessellator (2113), and domain shader (2117)) may be bypassed.

[0278] A complete geometric object may be processed by the geometry shader (2119) through one or more threads dispatched to the execution unit (2152A, 2152B), or may proceed directly to the clipper (2129). The geometry shader may operate on the entire geometric object rather than vertices or a patch of vertices, as in the previous stage of the graphics pipeline. When tessellation is disabled, the geometry shader (2119) receives input from the vertex shader (2107). When the tessellation unit is disabled, the geometry shader (2119) may be programmed into a geometry shader program that performs geometric tessellation.

[0279] Before rasterization, the clipper (2129) processes vertex data. The clipper (2129) may be a fixed-function clipper or a programmable clipper with clipping and geometry shader functions. The rasterizer and depth test component (2173) of the rendering output pipeline (2170) may dispatch a pixel shader to convert geometric objects into per-pixel representations. The pixel shader logic may be included in the thread execution logic (2150). Optionally, the application may bypass the rasterizer and depth test component (2173) and access unrasterized vertex data through the stream-out unit (2123).

[0280] The graphics processor (2100) has an interconnection bus, an interconnection fabric, or other interconnection mechanism that allows data and message transfer between the main components of the processor. In some embodiments, execution units (2152A, 2152B) and associated logic units (e.g., L1 cache (2151), sampler (2154), texture cache (2158), etc.) are interconnected via a data port (2156) that performs memory access and communicates with the rendering output pipeline component of the processor. In some embodiments, the sampler (2154), cache (2151, 2158), and execution units (2152A, 2152B) may each have a separate memory access path. Optionally, the texture cache (2158) may also be configured as a sampler cache.

[0281] The rendering output pipeline (2170) may include a rasterizer and a depth test component (2173) that convert vertex-based objects into their corresponding pixel-based representations. The rasterizer logic may include a window / masker unit that performs fixed-function triangle and line rasterization. The associated render cache (2178) and depth cache (2179) may also be available in some embodiments. The pixel action component (2177) performs pixel-based actions on the data, but in some examples, pixel actions related to 2D actions (e.g., bit-block image transfer via blending) are performed by the 2D engine (2141) or replaced by the display controller (2143) using an overlay display plane at display time. A shared L3 cache (2175) may be available to all graphics components to allow data to be shared without using main system memory.

[0282] The graphics processor media pipeline (2130) may include a media engine (2137) and a video front-end (2134). The video front-end (2134) may receive pipeline commands from a command streamer (2103). The media pipeline (2130) may include a separate command streamer. The video front-end (2134) may process the media commands before sending them to the media engine (2137). The media engine (2137) may include a thread creation function that creates a thread to be dispatched to thread execution logic (2150) via a thread dispatcher (2131).

[0283] The graphics processor (2100) may include a display engine (2140). The display engine (2140) is located outside the processor (2100) and may be connected to the graphics processor via a ring interconnect (2102) or some other interconnect bus or fabric. The display engine (2140) may include a 2D engine (2141) and a display controller (2143). The display engine (2140) may include special purpose logic capable of operating independently of the 3D pipeline. The display controller (2143) may be connected to a display device (not shown), which may be a system-integrated display device, such as in a laptop computer, or an external display device attached via a display device connector.

[0284] The geometry pipeline (2120) and the media pipeline (2130) may be configured to perform operations according to multiple graphics and media programming interfaces and are not limited to any single application programming interface (API). Driver software for the graphics processor may convert API calls limited to a specific graphics or media library into commands that can be processed by the graphics processor. Support for the Khronos Group’s OpenGL (Open Graphics Library), OpenCL (Open Computing Language), and / or Vulkan graphics and computing APIs may be provided. Support for Microsoft Corporation’s Direct3D library may also be provided. A combination of these libraries may be supported. Support for the Open Source Computer Vision Library (OpenCV) may also be provided. Future APIs with compatible 3D pipelines will also be supported when mapping from the API pipeline to the graphics processor pipeline becomes possible.

[0285] Graphics Pipeline Programming

[0286] FIG. 22a is a block diagram illustrating a graphics processor command format (2200) used to program a graphics processing pipeline, such as the pipeline described herein together with FIG. 16a, 17, and 21. FIG. 22b is a block diagram illustrating a graphics processor command sequence (2210) according to one embodiment. The solid boxes in FIG. 22a illustrate components generally included in a graphics command, and the dotted lines include components that are optional or included only in a subset of graphics commands. An exemplary graphics processor command format (2200) of FIG. 22a includes a client (2202), a command opcode (2204), and a data field identifying data (2206) for the command. A sub-opcode (2205) and a command size (2208) are also included in some commands.

[0287] The client (2202) specifies a client unit of a graphics device that processes command data. A graphics processor command parser may examine the client field of each command to coordinate further processing of the command and to route the command data to the appropriate client unit. The graphics processor client unit may include a memory interface unit, a render unit, a 2D unit, a 3D unit, and a media unit. Each client unit may have a corresponding processing pipeline that processes the command. When a command is received by the client unit, the client unit reads the operation code (2204) and, if present, the sub-operation code (2205) to determine the action to be performed. The client unit performs the command using the information in the data field (2206). For some commands, an explicit command size (2208) is expected to specify the size of the command. The command parser may automatically determine the size of at least some of the commands based on the command operation code. Commands can be arranged using multiple double words. Other command formats may also be used.

[0288] The flowchart of FIG. 22b illustrates an exemplary graphics processor command sequence (2210). Software or firmware of a data processing system featuring an embodiment of a graphics processor may use a version of the illustrated command sequence to set, execute, and terminate a set of graphics operations. Sample command sequences are illustrated and described for illustrative purposes only and are not limited to these specific commands or this command sequence. Furthermore, since commands may be issued as a batch of commands in a command sequence, the graphics processor will process the sequence of commands at least partially simultaneously.

[0289] The graphics processor command sequence (2210) may begin with a pipeline flush command (2212) that causes any active graphics pipeline to complete any command currently pending in the pipeline. Optionally, the 3D pipeline (2222) and the media pipeline (2224) may not operate simultaneously. A pipeline flush is performed to allow the active graphics pipeline to complete any ongoing command. In response to the pipeline flush, the command parser for the graphics processor will pause command processing until the active drawing engine completes the ongoing task and the associated read cache is invalidated. Optionally, all data in the render cache marked as 'dirty' may be flushed into memory. The pipeline flush command (2212) may be used for pipeline synchronization or before putting the graphics processor into a low-power state.

[0290] A pipeline select command (2213) may be used when a command sequence requires the graphics processor to explicitly switch between pipelines. The pipeline select command (2213) may be required only once within the execution context before issuing a pipeline command, provided that the execution context does not issue commands for both pipelines. A pipeline flush command (2212) may be required immediately before switching pipelines via the pipeline select command (2213).

[0291] A pipeline control command (2214) can configure a graphics pipeline for operation and can be used to program a 3D pipeline (2222) and a media pipeline (2224). A pipeline control command (2214) can configure the pipeline state for an active pipeline. A pipeline control command (2214) can be used for pipeline synchronization and to delete data from one or more cache memories within the active pipeline before processing a batch of commands.

[0292] The return buffer status command (2216) may be used to configure a set of return buffers for each pipeline to write data. Some pipeline operations require the allocation, selection, or configuration of one or more return buffers during processing, for which the operation writes intermediate data. The graphics processor may also use one or more return buffers to store output data and perform cross-thread communication. The return buffer status (2216) may include selecting the size and number of return buffers to use for a set of pipeline operations.

[0293] The remaining commands in the command sequence vary depending on the active pipeline for the operation. Depending on the pipeline determination (2220), the command sequence is aligned with the 3D pipeline (2222) starting from the 3D pipeline state (2230) or the media pipeline (2224) starting from the media pipeline state (2240).

[0294] The commands that configure the 3D pipeline state (2230) include 3D state setting commands for vertex buffer state, vertex element state, constant color state, depth buffer state, and other state variables to be configured before the 3D primitive command is processed. The values ​​of these commands are determined at least partially based on the specific 3D API being used. The 3D pipeline state (2230) commands may also optionally disable or bypass specific pipeline elements if they are not being used.

[0295] The 3D primitive (2232) command can be used to submit a 3D primitive to be processed by the 3D pipeline. The command and associated parameters passed to the graphics processor via the 3D primitive (2232) command are passed to the vertex fetch function of the graphics pipeline. The vertex fetch function creates a vertex data structure using the 3D primitive (2232) command data. The vertex data structure is stored in one or more return buffers. The 3D primitive (2232) command can be used to perform vertex operations on the 3D primitive through a vertex shader. To process the vertex shader, the 3D pipeline (2222) dispatches a shader execution thread to the graphics processor execution unit.

[0296] The 3D pipeline (2222) can be triggered via an execute (2234) command or event. A register can record the execution of the trigger command. The execution can be triggered via a 'go' or 'kick' command in the command sequence. The command execution can be triggered using a pipeline synchronization command that flushes the command sequence through the graphics pipeline. The 3D pipeline will perform geometric processing on the 3D primitives. When the operation is complete, the resulting geometric object is rasterized and the pixel engine colors the resulting pixels. Additional commands controlling pixel shading and pixel backend operations may also be included in these operations.

[0297] A graphics processor command sequence (2210) may follow a media pipeline (2224) path when performing a media operation. Generally, the specific use and method of programming for the media pipeline (2224) depends on the media or computing operation to be performed. During media decoding, specific media decoding operations may be offloaded to the media pipeline. The media pipeline may also be bypassed, and media decoding may be performed wholly or partially using resources provided by one or more general-purpose processing cores. The media pipeline may also include elements for general-purpose graphics processor unit (GPGPU) operations, where the graphics processor is used to perform SIMD vector operations using compute shader programs that are not explicitly involved in the rendering of graphics primitives.

[0298] The media pipeline (2224) can be configured in a manner similar to the 3D pipeline (2222). A set of commands that constitute the media pipeline state (2240) is dispatched or placed in a command queue prior to the media object command (2242). The commands for the media pipeline state (2240) may include data that constitutes media pipeline elements to be used to process media objects. This includes data that constitutes video decoding and video encoding logic within the media pipeline, such as encoding or decoding formats. The commands for the media pipeline state (2240) may also support the use of one or more pointers to "indirect" state elements that include the placement of state settings.

[0299] A media object command (2242) may provide a pointer to a media object for processing by the media pipeline. The media object contains a memory buffer containing video data to be processed. Optionally, all media pipeline states must be valid before issuing the media object command (2242). When the pipeline state is configured and the media object command (2242) is queued, the media pipeline (2224) is triggered via an execution command (2244) or an equivalent execution event (e.g., register write). The output from the media pipeline (2224) may then be post-processed by the 3D pipeline (2222) or by an operation provided by the media pipeline (2224). GPGPU operations may be configured and executed in a manner similar to media operations.

[0300] Graphics software architecture

[0301] FIG. 23 illustrates an exemplary graphics software architecture for a processing system (2300). The software architecture may include a 3D graphics application (2310), an operating system (2320), and at least one processor (2330). The processor (2330) may include a graphics processor (2332) and one or more general-purpose processor core(s) (2334). The processor (2330) may be a variant of the processor (1402) or any other processor described herein. The processor (2330) may be used in place of the processor (1402) or any other processor described herein. Thus, the disclosure of any features combined with the processor (1402) or any other processor described herein also indicates, but is not limited to, a corresponding combination with the graphics processor (2330). Additionally, elements of FIG. 23 having the same or similar names as elements of any other drawings in this specification may describe the same elements as those in other drawings, operate or function in a similar manner, include the same components, and may be linked to other entities such as those described in this specification, though not limited to them. The graphic application (2310) and the operating system (2320) are each executed in the system memory (2350) of the data processing system.

[0302] A 3D graphics application (2310) may include one or more shader programs that include shader instructions (2312). Shader language instructions may be written in a high-level shader language such as Direct3D’s HLSL (High-Level Shader Language), GLSL (OpenGL Shader Language), etc. The application may also include executable instructions (2314) written in machine language suitable for execution by a general-purpose processor core (2334). The application may also include a graphics object (2316) defined by vertex data.

[0303] The operating system (2320) may be Microsoft Corporation’s Microsoft®Windows® operating system, a proprietary UNIX-like operating system, or an open source UNIX-like operating system using a variant of the Linux kernel. The operating system (2320) may support graphics APIs (2322) such as the Direct3D API, OpenGL API, or Vulkan API. When the Direct3D API is in use, the operating system (2320) uses a front-end shader compiler (2324) to compile all shader instructions (2312) written in HLSL into a low-level shader language. The compilation may be Just-In-Time (JIT) compilation, or the application may perform shader pre-compilation. High-level shaders may be compiled into low-level shaders while compiling the 3D graphics application (2310). Shader instructions (2312) may be provided in an intermediate form, such as a version of SPIR (Standard Portable Intermediate Representation) used in the Vulkan API.

[0304] A user mode graphics driver (2326) may include a back-end shader compiler (2327) that converts shader instructions (2312) into a hardware-specific representation. When the OpenGL API is in use, shader instructions (2312) written in the GLSL high-level language are passed to the user mode graphics driver (2326) for compilation. The user mode graphics driver (2326) may use an operating system kernel mode function (2328) that communicates with a kernel mode graphics driver (2329). The kernel mode graphics driver (2329) may communicate with a graphics processor (2332) to dispatch commands and instructions.

[0305] IP Core Implementation

[0306] One or more aspects may be implemented by typical code stored on a machine-readable medium that represents and / or defines logic within an integrated circuit, such as a processor. For example, the machine-readable medium may contain instructions that represent various logic within a processor. When read by a machine, the instructions may cause the machine to manufacture logic that performs the technology described herein. Such representation, known as an "IP core," is a reusable unit of logic for an integrated circuit that may be stored on a tangible machine-readable medium as a hardware model describing the structure of the integrated circuit. The hardware model may be supplied to various customers or manufacturing facilities, which load the hardware model onto manufacturing machines that manufacture the integrated circuit. The integrated circuit may be manufactured to perform the operation described in connection with any embodiment described herein.

[0307] FIG. 24a is a block diagram illustrating an IP core development system (2400) that can be used to fabricate an integrated circuit to perform operations according to one embodiment. The IP core development system (2400) can be used to generate a modular, reusable design that can be integrated into a larger design or used to construct an entire integrated circuit (e.g., an SOC integrated circuit). The design facility (2430) can generate a software simulation (2410) of the IP core design using an advanced programming language (e.g., C / C++). The software simulation (2410) can be used to design, test, and verify the behavior of the IP core using a simulation model (2412). The simulation model (2412) may include function, behavior, and / or timing simulations. Subsequently, a register transfer level (RTL) design (2415) can be generated or synthesized from the simulation model (2412). The RTL design (2415) refers to an abstraction of the operation of an integrated circuit that models the flow of digital signals between hardware registers containing related logic performed using the modeled digital signals. In addition to the RTL design (2415), lower-level designs at the logic level or transistor level may also be generated, designed, or synthesized. Thus, specific details of the initial design and simulation may differ.

[0308] The RTL design (2415) or equivalent may be further synthesized into a hardware model (2420) by a design facility, and this hardware model (2420) may be written in a hardware description language (HDL) or may be another representation of physical design data. The HDL may be further simulated or tested to verify the IP core design. The IP core design may be stored using non-volatile memory (2440) (e.g., a hard disk, flash memory, or any non-volatile storage medium) for transfer to a third-party manufacturing facility (2465). Alternatively, the IP core design may be transmitted via a wired connection (2450) or a wireless connection (2460) (e.g., via the Internet). The manufacturing facility (2465) may then manufacture an integrated circuit based at least partially on the IP core design. The manufactured integrated circuit may be configured to perform an operation according to at least one embodiment described herein.

[0309] FIG. 24b illustrates a side cross-sectional view of an integrated circuit package assembly (2470). The integrated circuit package assembly (2470) illustrates an embodiment of one or more processor or accelerator devices as described herein. The package assembly (2470) includes a plurality of units of hardware logic (2472, 2474) connected to a substrate (2480). The logic (2472, 2474) may be implemented as at least partially configurable logic or fixed-function logic hardware and may include one or more parts of any of the processor core(s), graphics processor(s), or other accelerator devices described herein. Each unit of the logic (2472, 2474) may be implemented within a semiconductor die and connected to the substrate (2480) via an interconnection structure (2473). The interconnect structure (2473) may be configured to route electrical signals between the logic (2472, 2474) and the substrate (2480) and includes, but is not limited to, interconnects such as bumps or pillars. The interconnect structure (2473) may be configured to route electrical signals, for example, input / output (I / O) signals and / or power or ground signals related to the operation of the logic (2472, 2474). Optionally, the substrate (2480) may be an epoxy-based laminate substrate. The substrate (2480) may also include other suitable types of substrates. The package assembly (2470) may be connected to other electrical devices through the package interconnect (2483). The package interconnect (2483) may be connected to the surface of the substrate (2480) to route electrical signals to other electrical devices such as a motherboard, other chipsets, or multi-chip modules.

[0310] Units of logic (2472, 2474) may be electrically connected to a bridge (2482) configured to route electrical signals between logic (2472, 2474). The bridge (2482) may be a dense interconnect structure that provides a path for electrical signals. The bridge (2482) may include a bridge substrate composed of glass or a suitable semiconductor material. Electrical routing features may be formed on the bridge substrate to provide a connection between chips between logic (2472, 2474).

[0311] Although two units of logic (2472, 2474) and a bridge (2482) are illustrated, the embodiments described herein may include more or fewer logic units on one or more dies. One or more dies may be connected by zero or more bridges, as the bridge (2482) may be excluded when logic is included on a single die. Alternatively, multiple dies or units of logic may be connected by one or more bridges. Additionally, multiple logic units, dies, and bridges may be connected together in other possible configurations, including three-dimensional configurations.

[0312] FIG. 24c illustrates a package assembly (2490) comprising a plurality of units of hardware logic chiplets connected to a substrate (2480) (e.g., a base die). The graphics processing unit, parallel processor, and / or computing accelerator described herein may be composed of various individually manufactured silicon chiplets. In this context, a chiplet is at least partially packaged integrated circuit comprising a distinct unit of logic that can be assembled into a larger package together with other chiplets. Various sets of chiplets having different IP core logic may be assembled into a single device. Additionally, chiplets may be integrated into a base die or base chiplet using active interposer technology. The concepts described herein enable interconnection and communication between different forms of IP within a GPU. Since IP cores may be manufactured using different process technologies and configured during the manufacturing process, this avoids the complexity of converging multiple IPs into the same manufacturing process, particularly in large SoCs with various characteristic IPs. Since multiple process technologies can be used, a cost-effective method is provided to shorten time-to-market and also to generate multiple product SKUs. In addition, isolated IPs can be powered on and off independently, and power to components not used on a given workload can be cut off, thereby reducing overall power consumption.

[0313] Hardware logic chiplets may include special purpose hardware logic chiplets (2472), logic or I / O chiplets (2474), and / or memory chiplets (2475). The hardware logic chiplets (2472) and logic or I / O chiplets (2474) may be implemented as at least partially configurable logic or fixed-function logic hardware and may include one or more parts of processor core(s), graphics processor(s), parallel processors, or any other accelerator devices described herein. The memory chiplets (2475) may be DRAM (e.g., GDDR, HBM) memory or cache (SRAM) memory.

[0314] Each chiplet may be manufactured as an individual semiconductor die and connected to a substrate (2480) via an interconnect structure (2473). The interconnect structure (2473) may be configured to route electrical signals between the various chiplets and the logic within the substrate (2480). The interconnect structure (2473) may include interconnects such as bumps or pillars, but is not limited thereto. In some embodiments, the interconnect structure (2473) may be configured to route electrical signals, such as I / O signals and / or power or ground signals related to the operation of logic, input / output (I / O), and memory chiplets, for example.

[0315] The substrate (2480) may be an epoxy-based laminate substrate, but is not limited thereto, and the substrate (2480) may also include other suitable types of substrates. The package assembly (2490) may be connected to other electrical devices through a package interconnect (2483). The package interconnect (2483) is connected to the surface of the substrate (2480) to route electrical signals to other electrical devices, such as a motherboard, other chipsets, or multi-chip modules.

[0316] A logic or I / O chiplet (2474) and a memory chiplet (2475) may be electrically connected via a bridge (2487) configured to route electrical signals between the logic or I / O chiplet (2474) and the memory chiplet (2475). The bridge (2487) may be a dense interconnect structure that provides a path for electrical signals. The bridge (2487) may include a bridge substrate composed of glass or a suitable semiconductor material. Electrical routing features may be formed on the bridge substrate to provide a chip-to-chip connection between the logic or I / O chiplet (2474) and the memory chiplet (2475). The bridge (2487) may also be referred to as a silicon bridge or an interconnect bridge. For example, the bridge (2487) is an Embedded Multi-die Interconnect Bridge (EMIB). Alternatively, the bridge (2487) may simply be a direct connection from one chiplet to another.

[0317] The substrate (2480) may include hardware components for I / O (2491), cache memory (2492), and other hardware logic (2493). The fabric (2485) may be embedded in the substrate (2480) to enable communication between various logic chiplets and logic (2491, 2493) within the substrate (2480). The I / O (2491), fabric (2485), cache, bridge, and other hardware logic (2493) may be integrated into a base die that forms a layer on top of the substrate (2480). The fabric (2485) may be a different type of packet switching fabric that exchanges data packets between components of a network or package assembly on a chip interconnect.

[0318] Additionally, the package assembly (2490) may include fewer or more components and chiplets interconnected by the fabric (2485) or one or more bridges (2487). The chiplets within the package assembly (2490) may be arranged in a 3D or 2.5D array. Generally, the bridge structure (2487) may be used to facilitate point-to-point interconnection between, for example, logic or I / O chiplets and memory chiplets. The fabric (2485) may be used to interconnect various logic and / or I / O chiplets (e.g., chiplets (2472, 2474, 2491, 2493)) to other logic and / or I / O chiplets. The cache memory (2492) within the substrate may operate as a global cache for the package assembly (2490), as part of a distributed global cache, or as a dedicated cache for the fabric (2485).

[0319] FIG. 24d illustrates a package assembly (2494) including an interchangeable chiplet (2495) according to one embodiment. The interchangeable chiplet (2495) may be assembled into a standardized slot on one or more base chiplets (2496, 2498). The base chiplets (2496, 2498) may be connected via a bridge interconnect (2497) which may be similar to other bridge interconnects described herein, for example, EMIB. The memory chiplet may also be connected to a logic or I / O chiplet via the bridge interconnect. The I / O and logic chiplets may communicate through an interconnect fabric. Each base chiplet may support one or more slots in a standardized format for logic, I / O, or memory / cache.

[0320] The SRAM and power transfer circuits may be manufactured with one or more base chiplets (2496, 2498), and these base chiplets (2496, 2498) may be manufactured using a process technology different from that of the interchangeable chiplets (2495) stacked on top of the base chiplets. For example, the base chiplets (2496, 2498) may be manufactured using a larger process technology, while the interchangeable chiplets may be manufactured using a smaller process technology. One or more interchangeable chiplets (2495) may be memory (e.g., DRAM) chiplets. Different memory densities may be selected for the package assembly (2494) depending on the target performance for the product using the power and / or package assembly (2494). Additionally, logic chiplets having different numbers of functional units may be selected at assembly depending on the target performance for the power and / or product. In addition, by inserting chiplets containing different types of IP logic cores into interchangeable chiplet slots, it is possible to design a hybrid processor capable of mixing and matching IP blocks of different technologies.

[0321] Exemplary System-on-Chip Integrated Circuit

[0322] FIGS. 25 and 26 illustrate exemplary integrated circuits and associated graphics processors that may be manufactured using one or more IP cores. In addition to what is illustrated, other logic and circuits including additional graphics processors / cores, peripheral interface controllers, or general-purpose processor cores may be included. Elements of FIGS. 25 and 26 having the same or similar names as elements of any other figure in this specification may describe the same elements as in the other figures, may operate or function in a similar manner, may include the same components, and may be linked to other entities such as those described in this specification, but are not limited to.

[0323] FIG. 25 is a block diagram illustrating an exemplary system-on-chip integrated circuit (2500) that may be manufactured using one or more IP cores according to one embodiment. The exemplary integrated circuit (2500) includes one or more application processor(s) (2505) (e.g., CPU) and at least one graphics processor (2510), which may be a variant of a graphics processor (1408, 1508, 2510), any graphics processor described herein, or may be used instead of any graphics processor described herein. Accordingly, the disclosure of any features combined with a graphics multiprocessor in this specification also indicates a corresponding combination with a graphics multiprocessor (2510), but is not limited thereto. The integrated circuit (2500) may further include a processor (2515) and / or a video processor (2520), any of which may be a modular IP core manufactured from the same or a plurality of different design facilities. The integrated circuit (2500) may include peripheral devices or bus logic including a USB controller (2525), a UART controller (2530), an SPI / SDIO controller (2535), and an I2S / I2C controller (2540). Additionally, the integrated circuit may include one or more HDMI (high-definition multimedia interface) controllers (2550) and a display device (2545) connected to a MIPI (mobile industry processor interface) display interface (2555). Storage devices may be provided by a flash memory subsystem (2560) including flash memory and a flash memory controller. Memory interfaces may be provided through a memory controller (2565) for accessing SDRAM or SRAM memory devices. Some integrated circuits further include an embedded security engine (2570).

[0324] FIGS. 26a and 26b are block diagrams illustrating an exemplary graphics processor for use within an SoC according to an embodiment described herein. The graphics processor may be a variant of the graphics processor (1408, 1508, 2510) or any other processor described herein. The graphics processor may be used in place of the graphics processor (1408, 1508, 2510) or any other processor described herein. Accordingly, the disclosure of any features combined with the graphics processor (1408, 1508, 2510) or any other processor described herein also indicates, but is not limited to, a corresponding combination with the graphics processor of FIGS. 26a and 26b. FIG. 26a illustrates an exemplary graphics processor (2610) of a system-on-chip integrated circuit that may be manufactured using one or more IP cores according to one embodiment. FIG. 26b illustrates an additional exemplary graphics processor (2640) of a system-on-chip integrated circuit that can be manufactured using one or more IP cores according to one embodiment. The graphics processor (2610) of FIG. 26a is an example of a low-power graphics processor core. The graphics processor (2640) of FIG. 26b is an example of a high-performance graphics processor core. For example, each of the graphics processors (2610, 2640) may be a variation of the graphics processor of FIG. 25, as mentioned at the beginning of this paragraph.

[0325] As illustrated in FIG. 26a, the graphics processor (2610) includes a vertex processor (2605) and one or more fragment processor(s) (2615A-2615N) (e.g., 2615A, 2615B, 2615C, 2615D, to 2615N-1 and 2615N). Since the graphics processor (2610) can execute different shader programs through separate logic, the vertex processor (2605) is optimized to execute operations for a vertex shader program, and one or more fragment processor(s) (2615A-2615N) execute fragment (e.g., pixel) shading operations for a fragment or pixel shader program. The vertex processor (2605) performs the vertex processing stage of the 3D graphics pipeline and generates primitive and vertex data. Fragment processor(s) (2615A-2615N) create a frame buffer that is displayed on a display device using primitive and vertex data generated by the vertex processor (2605). The fragment processor(s) (2615A-2615N) may be optimized to execute a fragment shader program provided by the OpenGL API, which can be used to perform operations similar to a pixel shader program provided by the Direct 3D API.

[0326] The graphics processor (2610) further includes one or more memory management units (MMUs) (2620A, 2620B), cache(s) (2625A, 2625B), and circuit interconnect(s) (2630A, 2630B). One or more MMU(s) (2620A, 2620B) map virtual addresses to physical addresses for the graphics processor (2610), which includes a vertex processor (2605) and / or fragment processor(s) (2615A-2615N) capable of referencing vertex or image / texture data stored in memory in addition to vertex or image / texture data stored in one or more cache(s) (2625A, 2625B). One or more MMU(s) (2620A, 2620B) are synchronized with other MMUs in the system, including one or more MMUs associated with one or more application processors (2505), image processors (2515), and / or video processors (2520) of FIG. 25, so that each processor (2505-2520) can participate in a shared or integrated virtual memory system. The component of the graphics processor (2610) may correspond to the component of another processor described herein. One or more MMU(s) (2620A-2620B) may correspond to the MMU (245) of FIG. 2c. The vertex processor (2605) and fragment processors (2615A-2615N) may correspond to the graphics multiprocessor (234). One or more circuit interconnect(s) (2630A, 2630B) enable the graphics processor (2610) to interface with other IP cores within the SoC via the internal bus of the SoC or via direct connection, depending on the embodiment. One or more circuit interconnect(s) (2630A-2630B) may correspond to the data crossbar (240) of FIG. 2c. Other correspondences may be found between similar components of the graphics processor (2610) and the various graphics processor architectures described herein.

[0327] As illustrated in FIG. 26b, the graphics processor (2640) includes one or more MMU(s) (2620A, 2620B), cache(s) (2625A, 2625B), and circuit interconnect(s) (2630A, 2630B) of the graphics processor (2610) of FIG. 26a. The graphics processor (2640) comprises one or more shader core(s) (2655A-2655N) (e.g., 2655A, 2655B, 2655C, 2655D, 2655E, 2655F through 2655N-1 and 2655N), which provides an integrated shader core architecture capable of executing any type of programmable shader code, including shader program code that implements a vertex shader, a fragment shader, and / or a compute shader, on a single core or any type of core. The exact number of shader cores present may vary depending on the embodiment and implementation. Additionally, the graphics processor (2640) includes an inter-core task manager (2645) that operates as a thread dispatcher and dispatches execution threads to one or more shader cores (2655A-2655N), and a tiling unit (2658) that accelerates tiling operations for tile-based rendering, wherein the tiling unit subdivides rendering operations for a scene in image space, for example, by utilizing local spatial consistency within the scene or optimizing the use of an internal cache. The shader cores (2655A-2655N) may correspond, for example, to the graphics multiprocessor (234) of FIG. 2d, the graphics multiprocessor (325, 350) of FIG. 3a, or to the multicore group (365A) of FIG. 3c.

[0328] The embodiments described herein include software, firmware, and hardware logic that provide techniques for performing arithmetic on sparse data through a systolic processing unit. One embodiment provides techniques for optimizing training and inference on a systolic array when using sparse data. One embodiment provides techniques for using decompression information when performing sparse computing operations. An architecture is provided that enables independent extension of matrix and / or tensor processing from other processing resources. One embodiment considers packed data compression and extension operations on a GPGPU.

[0329] GPGPU equipped with tensor acceleration logic and integrated memory

[0330] FIG. 27 is a block diagram of a processing system (2700) according to one embodiment. The data processing system (2700) is a heterogeneous processing system having a processor (2702), integrated memory (2710), and a GPGPU (2720) including machine learning acceleration logic. The processor (2702) and the GPGPU (2720) may be any of the processor and GPGPU / parallel processor as described herein. The processor (2702) may execute instructions for a compiler (2715) stored in system memory (2712). The compiler (2715) is executed on the processor (2702) to compile source code (2714A) into compiled code (2714B). The compiled code (2714B) may include instructions that can be executed by the processor (2702) and / or instructions that can be executed by the GPGPU (2720). During compilation, the compiler (2715) may perform an operation to insert metadata, which includes hints regarding the level of data parallelism present in the compiled code (2714B) and / or hints regarding data locations associated with threads to be dispatched based on the compiled code (2714B). The compiler (2715) may include information necessary to perform such an operation, or these operations may be performed with the help of a runtime library (2716). The runtime library (2716) may also support the compiler (2715) in compiling the source code (2714A) and may include instructions linked with the compiled code (2714B) at runtime to facilitate the execution of the compiled instructions on the GPGPU (2720).

[0331] The integrated memory (2710) represents an integrated address space that can be accessed by the processor (2702) and the GPGPU (2720). The integrated memory may include system memory (2712) as well as GPGPU memory (2718). The GPGPU memory (2718) is memory within the address space of the GPGPU (2720) and may include part or all of the system memory (2712). In one embodiment, the GPGPU memory (2718) may also include at least a portion of any memory dedicated for exclusive use by the GPGPU (2720). In one embodiment, compiled code (2714B) stored in the system memory (2712) may be mapped to the GPGPU memory (2718) for access by the GPGPU (2720).

[0332] The GPGPU (2720) comprises a plurality of computing blocks (2724A-2724N) that may include one or more of the various computing units or execution elements described herein. Processing resources may be or may include various computing resources, such as execution units, computing units, streaming multiprocessors, graphics multiprocessors, or multi-core groups. In one embodiment, the GPGPU (2720) further comprises a tensor accelerator (2723) that may include one or more special function computing units designed to facilitate a subset of matrix operations (e.g., inner product, etc.). The tensor accelerator (2723) may be referred to as a tensor accelerator or a tensor core. In one embodiment, logic components within the tensor accelerator (2723) may be distributed across the processing resources of the plurality of computing blocks (2724A-2724N).

[0333] The GPGPU (2720) may also include a set of resources that can be shared by computing blocks (2724A-2724N) and a tensor accelerator (2723), including but not limited to a set of registers (2725), a power and performance module (2726), and a cache (2727). In one embodiment, the registers (2725) include registers that are directly and indirectly accessible, wherein the indirectly accessible registers are optimized to be used by the tensor accelerator (2723). The power and performance module (2726) may be configured to power the computing blocks (2724A-2724N) and to adjust the clock frequency to power the gate idle components within the computing blocks (2724A-2724N). In various embodiments, the cache (2727) may include an instruction cache and / or a low-level data cache.

[0334] The GPGPU (2720) may further include an L3 data cache (2730) which can be used to cache data accessed from integrated memory (2710) by a tensor accelerator (2723) and / or a computing element within a computing block (2724A-2724N). In one embodiment, the L3 data cache (2730) includes a shared local memory (2732) which can be shared by the computing element within the computing block (2724A-2724N) and the tensor accelerator (2723).

[0335] In one embodiment, the GPGPU (2720) includes instruction processing logic such as a fetch and decoding unit (2721) and a scheduler controller (2722). The fetch and decoding unit (2721) includes a fetch unit and a decoding unit for fetching and decoding instructions to be executed by one or more of a computing block (2724A-2724N) or a tensor accelerator (2723). Instructions may be scheduled to an appropriate function unit or tensor accelerator within the computing block (2724A-2724N) via the scheduler controller (2722). In one embodiment, the scheduler controller (2722) is an ASIC configurable to perform advanced scheduling tasks. In one embodiment, the scheduler controller (2722) is a microcontroller or a low energy-per-instruction processing core capable of executing scheduler instructions loaded from a firmware module.

[0336] In one embodiment, some functions performed by the computing blocks (2724A-2724N) may be directly scheduled or offloaded to the tensor accelerator (2723). In various embodiments, the tensor accelerator (2723) includes processing element logic configured to efficiently perform matrix computing operations, such as multiplication, addition, and inner product, used in 3D graphics or computing shader programs. In one embodiment, the tensor accelerator (2723) may be configured to accelerate operations used by a machine learning framework. In one embodiment, the tensor accelerator (2723) is an application-specific integrated circuit explicitly configured to perform a specific set of parallel matrix multiplications and / or additions. In one embodiment, the tensor accelerator (2723) is a field programmable gate array (FPGA) that provides fixed-function logic that can be updated between workloads. The series of matrix operations that can be performed by the tensor accelerator (2723) may be limited to the operations that can be performed by the computing blocks (2724A-2724N). However, the tensor accelerator (2723) can perform these operations with significantly higher throughput than the computing blocks (2724A-2724N).

[0337] FIG. 28 illustrates a matrix operation (2805) performed by an instruction pipeline (2800) according to one embodiment. The instruction pipeline (2800) may be configured to perform matrix operations (2805), such as an inner product operation (but not limited thereto). The inner product of two vectors is a scalar value equal to the sum of the products of the corresponding components of the vectors. The inner product can be calculated as shown in Equation (1) below.

[0338]

[0339] The dot product can be used for convolution operations on a convolutional neural network (CNN). FIG. 28 illustrates a two-dimensional (2D) convolution using a matrix operation (2805) that includes a dot product operation. Although a 2D convolution is illustrated, an N-dimensional convolution can be performed on an N-dimensional volume using an N-dimensional filter. A receptive field tile (2802) highlights a portion of the input volume within an input volume buffer (2804). The input volume buffer can be stored in memory (2830). A dot matrix operation (2805) can be performed between the data within the receptive field tile (2802) and the convolution filter to generate data points within an output buffer (2806), which can be stored in memory (2830). The memory (2830) may be any memory described in this specification, including system memory (2712), GPGPU memory (2718), or one or more cache memories (2727, 2730), as shown in FIG. 27.

[0340] A combination of data points within the output buffer (2806) represents an activation map generated by a convolution operation. Each point within the activation map is generated by sliding a receptive field tile across the input volume buffer (2804). The activation map data can be input to an activation function to determine an output activation value. In one embodiment, the convolution of the input volume buffer (2804) can be defined as a high-level matrix operation (2805) within the framework. The high-level matrix operation can be performed through primitive operations, such as basic linear algebra subprogram (BLAS) operations. The primitive operation can be accelerated through hardware instructions executed by the instruction pipeline (2800).

[0341] An instruction pipeline (2800) used to accelerate hardware instructions may include an instruction fetch and decode unit (2721) capable of fetching and decoding hardware instructions and a scheduler controller (2722) capable of scheduling decoded instructions to one or more processing units and / or a tensor accelerator (2723) within a computing block (2724A-2724N). In one embodiment, hardware instructions may be scheduled to the computing block (2724A-2724N) and offloaded to the tensor accelerator (2723). One or more hardware instructions and associated data for performing matrix operations (2805) may be stored in memory (2830). The output of the hardware instructions may also be stored in memory (2830).

[0342] In one embodiment, the tensor accelerator (2723) may execute one or more hardware instructions to perform matrix operations (2805) using an integrated systolic array (2808) (DP logic). The systolic array (2808) may include a combination of programmable and fixed-function hardware configurable to perform inner product operations. While functional units within the computing blocks (2724A-2724N) may be configured to perform inner product operations, the systolic array (2808) may be configured to perform a limited subset of inner product operations with a much higher throughput than the computing blocks (2724A-2724N).

[0343] FIGS. 29a and 29b illustrate details of a hardware-based systolic array (2808) according to some embodiments. FIG. 29a illustrates a grid of multiple functional units configurable to perform multiple inner product operations within a single clock cycle. FIG. 29b illustrates one exemplary functional unit. During systolic matrix calculation,

[0344] As illustrated in FIG. 29a, in one embodiment, the systolic array (2808) can be configured to perform a set of parallel dot product operations using various function units. The inner product can be performed in a 'systolic' manner where SIMD data is pumped across multiple layers of function units. As illustrated in FIG. 29a, in one embodiment, the systolic array (2808) can be configured to perform a set of parallel dot product operations using various function units. The inner product can be performed in a 'systolic' manner where SIMD data is pumped across multiple layers of function units. The systolic array (2808) is a set of function units arranged in a grid. The grid of function units operates in lockstep mode and is optimized to perform multiplication-accumulation operations. The matrix operated by the systolic array (2808) is divided into sub-matrices that are pumped across the grid of function units.

[0345] In one embodiment, the systolic array (2808) can process a configurable number of SIMD channels of data using a configurable systolic depth. For a given instruction, the SIMD width and systolic depth can be selected to process a set of source data. The systolic depth defines the number of systolic layers of hardware logic to be used to process the instruction. A systolic layer is a group of multiplier and adder logic units having a variable SIMD width, wherein the systolic layer can receive an initial accumulator value as an input and generate an inner product value to be output to a successive systolic layer or to an output register.

[0346] In some embodiments, three sources may be processed, each source may be a vector register or an immediate value. In one embodiment, the source (2900) (SRC0) may be one or more initial accumulator values, which may be a single value or a vector of accumulator values. The initial accumulator values ​​will be added to a first set of inner products calculated by each function unit within the first systolic layer. The inner products calculated by the function units may be provided to the next systolic layer for a given SIMD channel. The inner products may be calculated based on the source (2901) (SRC1) and source (2902) (SRC2), which are vector registers that may contain one or more channels of packed data, each channel containing a 4-element vector. In one embodiment, each channel is 32 bits wide and provides four 8-bit vector elements. Some embodiments are configurable to calculate the inner product from an input vector having 8-bit elements, 4-bit elements, and / or 2-bit elements. In one embodiment, mixed-precision operation may be performed using any combination of supported element sizes (e.g., 8 bits x 2 bits, 8 bits x 4 bits, 4 bits x 4 bits, etc.). In one embodiment, the systolic array (2808) is configured for integer calculation, but automatic fixed-point operation may be configured in some embodiments. Although the instruction described herein is a 4-element inner product, in some embodiments, the systolic array (2808) may be configured to support floating-point inner product calculations for a different number of elements per vector.

[0347] In one embodiment, multiple channels of a 4-element vector may be packed into a single vector register of varying widths (e.g., 64 bits, 128 bits, 256 bits, 512 bits, etc.). Simultaneous dot products may be calculated via a systolic array (2808) for multiple channels of vector elements provided through source (2901) and source (2902). The number of channels of vector elements to be processed may be configured based on the execution size and systolic depth selected for the dot product calculation. In one embodiment, source vectors wider than the specified execution size and / or systolic depth may be calculated using multiple cycles of the systolic array (2808).

[0348] The number of calculations that can be performed within a given clock cycle may vary based on the number of SIMD lanes and systolic layers. As illustrated, the systolic array (2808) can perform 16 inner products per SIMD lane of throughput using a systolic depth of 4. When configured for 8 SIMD lanes, the logic can perform 128 8-bit integer (INT8) inner products within a given cycle. When configured for 8 SIMD lanes and a systolic depth of 8, each lane can perform 32 8-bit integer (INT8) inner products and a total of 256 inner products. These specific numbers of operations are examples of one embodiment, and other embodiments vary in throughput. Additionally, if the data types are different, the number of operations will be scaled based on the different data types.

[0349] In each function unit, the dot product is calculated through multiplier and adder logic, and the dot product is added to the accumulator value. The data of the result can be output to a destination register or provided to the accumulator of the next systolic layer. Details of the function unit (2912) are shown in FIG. 29b.

[0350] As illustrated in FIG. 29b, the functional unit (2912) may include a set of input data buffers (2904, 2906) and an accumulator (2922), each of which may receive input data. In one embodiment, the data buffer (2906) may receive a source (2902) (SRC2) which may be a packed vector of input data. The data buffer (2904) may receive a source (2901) (SRC1) which may be a packed vector of input data. The accumulator (2922) may receive a source (2900) (SRC0) which provides an initial accumulator value for the functional unit (2912). The initial accumulator value is added to the inner product calculated from the elements of the source (2901) and the source (2902). The inner product is calculated through element-wise multiplication of the source vector using a set of multipliers (2923A-2923D) and an adder (2924). The multipliers (2923A-2923D) are used to calculate a set of products. The sum of the set of products is calculated by the adder (2924). This sum can be accumulated (e.g., added) with any initial value provided through the source (2900). In one embodiment, this accumulated value can be provided as an input value (2926) for the next accumulator that may reside in a subsequent systolic layer. In one embodiment, the source (2901) may include a plurality of input data channels. Additional channels of the source (2901) may be relayed to an additional SIMD lane (2928) as SRC1 inputs. In one embodiment, the source (2902) may include a plurality of input data channels. Additional channels of the source (2902) can be used as SRC2 input data for logic units within additional systolic depths. In one embodiment, the source (2900) may optionally include multiple channels, and additional channels are provided as inputs to accumulators within additional function units. In one embodiment, the source (2900) may be a single value added to each accumulator within each function unit of the initial systolic layer.

[0351] FIG. 30 illustrates a systolic array (3000) comprising circuitry that facilitates partial sum loopback and sparse matrix multiplication. In the aforementioned systolic array (2808), operands containing weight data may be fixed within the array, and partial sums are propagated throughout the array structure. While other details regarding the systolic array (2808) may be applicable, in the systolic array (3000), partial sums are recirculated instead of being propagated to the next systolic layer. In one embodiment, the systolic array (3000) may consist of M rows and N columns of process elements (PE(3012AA)-PE(3012MN)). Process elements may access registers that store input data in the form of row and column data for an input matrix. A register may be stored in a register file local to a systolic array (3000) or in a register file of a processing resource connected to or containing systolic arrays (3000). A register may store row elements of matrix A (3002A to 3002M), which will be multiplied by column elements of matrix B (3001A to 3002N).

[0352] In one embodiment, fused multiplication-addition (FMA) can be performed in each processing element (PE(3012AA)-PE(3012MN)) during each clock cycle. An element of matrix A is multiplied by the corresponding element of matrix B and then added to an accumulator value or, during the first cycle, to an optional initial input value (e.g., SRC0). A partial sum loopback can be configured in each processing element. After each cycle, the accumulator value can be looped back within the processing element and used as an input for the next cycle. When the operation is performed on the entire row, the result can be stored in a register file. Data movement between the processing elements (PE(3012AA)-PE(3012MN)) after one set of calculation cycles can vary based on the instruction or macro-operation being performed.

[0353] Acceleration of Sparse Matrix Multiplier

[0354] When performing matrix multiplication operations on sparse input data, if either input element A or input element B is 0, the result of the multiplication operation will be 0. Therefore, if either input is 0 and the accumulator value can be output as the result of the FMA operation without performing multiplication, the operation can be bypassed, and as a result, the performance and efficiency of sparse matrix operations are improved.

[0355] FIGS. 31a through 31c illustrate a technique for accelerating sparse matrix multiplication by skipping operations for zero value inputs. A systolic array (2808) and a systolic array (3000) can track metadata for input elements and bypass multiplications with one or more zero inputs. FIG. 31a illustrates a system (3100) in which the systolic array (2808) includes a load filter that skips the load of sparse inputs. FIG. 31b illustrates a system (3150) in which the systolic array (3000) includes logic that skips matrix multiplication operations involving zero value operands. FIG. 31c illustrates a method (3180) for accelerating sparse matrix multiplication. Aspects of the techniques of FIGS. 31a through 31c can be applied cross-referenced to both the systolic array (2808) and the systolic arrays (3000). These techniques are applied to facilitate sparse matrix multiplication described in relation to the systolic array, but these techniques are applicable to other types of matrices or tensor accelerator units.

[0356] As illustrated in FIG. 31a, the memory (3120) can store the matrix (3102) and the matrix (3104). The systolic array (2808) may include a matrix A load unit (3126), a matrix B load unit (3122), a matrix A feed unit (3128), and a matrix B feed unit (3124). The matrix (3102) can be loaded and supplied as matrix B, while the matrix (3104) can be loaded and supplied as matrix A. Submatrices of matrix A and matrix B can be loaded and supplied through a functional unit (3130) that operates as a processing element of the systolic array (2808).

[0357] In one embodiment, the load B filter (3127) and the load A filter (2127) may include a buffer for storing a sparsity map (3112) for the matrix (3102) and a sparsity map (3114) for the matrix (3104). The load B filter (3121) may bypass the load of zero-value elements by the matrix B load unit (3122). The load A filter (3127) may bypass the load of zero-value elements by the matrix A load unit (3126). If bypass occurs for an element of either the matrix (3102) or the matrix (3104), bypass also occurs for the corresponding element. The elements that are not bypassed may be processed by the function unit (3130).

[0358] As illustrated in FIG. 31b, the system (3150) includes a systolic array (3000) having logic that omits matrix multiplication operations involving zero-value operands. As previously mentioned, each PE (3012AA-3013MN) includes hardware logic that performs calculations for matrix operations. A (A0, A1 to A M ) and B(B0, B1 to B N ) are elements of an input matrix associated with an inner product, matrix multiplication, multiplication / addition, or multiplication-accumulation operation. In one embodiment, each PE (3012AA-3013MN) is associated with metadata (3151a, 3151b, …, 3151m; 3152a, 3152b, …, 3152n; 3660a, 3660b, …, 3160n; 3170a, 3170b, …, 3170n) for the input operands associated with the operation to be performed. The metadata may include an indication of which input operands have a value of 0.

[0359] For example, for an operation to be performed by PE (3012AA) on elements A0 and B0, metadata (3050a) can be checked to determine whether input A0 is a value input of 0. Metadata (3152a) can be checked to determine whether input B0 is 0. If PE (3012AA) is configured to perform matrix multiplication and any one of the inputs is 0, the operation in PE (3012AA) may be omitted.

[0360] In one embodiment, in addition to omitting operations to be performed, the load of operands is bypassed based on metadata (3150a, 3150b, ..., 3150n; 3152a, 3152b, ..., 3152n; 3660a, 3660b, ..., 3160n; 3170a, 3170b, ..., 3170n). In one embodiment, if input A0 is 0, the load of both input A0 and input element B0 may be bypassed. If input B0 is 0, the load of both input B0 and input element A0 may be bypassed. If the bypass of the load is omitted, the next element to be processed in the same cycle in which the load is omitted may be loaded. In this configuration, only non-zero values ​​are input into the systolic array (3000). If an entire row of matrix A or a column of matrix B is 0, operations on the entire row or column may be bypassed.

[0361] As illustrated in FIG. 31c, logic associated with a matrix accelerator, tensor accelerator, tensor processor, tensor core, etc., described herein, such as but not limited to a systolic array (2808) and / or a systolic array (3000), can accelerate matrix multiplication operations through the method (3180). The method (3180) can be performed by control logic within the accelerator or by logic within the accelerator, and allows the accelerator to omit operations for zero value inputs.

[0362] The method (3180) includes logic associated with a matrix / tensor accelerator (3182) that loads metadata identifying zero and / or non-zero elements of an input matrix. The metadata may be in the form of a bit field having an active bit for each non-zero element, wherein the bit field indicates which element should be loaded. The metadata may be in the form of a bit field having an active bit for each zero element, wherein the bit field indicates which element should be omitted. The metadata may be generated by the matrix or tensor acceleration logic or supplied as input along with the input matrix to be processed.

[0363] After that, logic can analyze metadata for the input elements for the multiplication operation to be performed (3184). If a zero value input is detected (in 3185, if 'yes'), logic will select the next set of input elements (3186) and then analyze metadata for these input elements (3184). If a zero value input is not detected (in 3185, if 'no'), logic can load the input elements into a processing element to process them (3186). Once the input elements are loaded, logic can perform matrix multiplication operations on the input elements (3186). After that, logic can add the result of the multiplication operation to an accumulator value. The accumulator value may be the accumulated result of a series of multiplication-addition operations or the input value provided for the multiplication-addition operation. After that, logic can select the next set of input elements if there are additional elements to process (3186). If there are no additional elements remaining to process, the result may be output.

[0364] Using the above techniques, a person skilled in the art may implement a general-purpose graphics processing unit including a tensor accelerator that includes logic to bypass matrix multiplication operations for zero value inputs. Bypassing may be performed based on metadata for the inputs, wherein the metadata may be generated within the tensor accelerator or outside the tensor accelerator. The metadata may be generated in advance for the entire set of inputs, or may be generated row / column based on the tensor accelerator while performing matrix multiplication operations on the rows and columns of the input matrix data. The tensor accelerator may include a plurality of processing elements, which may be a systolic array of processing elements. In one embodiment, the metadata may be analyzed or generated in relation to the sub-matrix of the input before the input is loaded into the processing element. In one embodiment, each processing element includes logic to detect zero value inputs or to analyze metadata identifying zero value inputs. In one embodiment, if an operation is omitted due to zero value inputs, the next set of input operands may be loaded within the same clock cycle in which the operand loading is omitted. In this configuration, the processing element can load these operands for multiplication only if all input operands are not zero, and any multiplication operations where any input is zero can be omitted before these operands are loaded into the processing element.

[0365] Separation of special function calculation arrays via shared register files

[0366] In varying degrees of abstraction, the processing resources described herein may represent a graphics processor or graphics processor architecture (e.g., a parallel processing unit, a graphics processing engine, a multi-core group, a computing unit, a computing unit of the next graphics core) in a GPU as described herein, and a processing element (e.g., a GPGPU core, a ray tracing core, a tensor core, an execution resource, an execution unit (EU), a stream processor, a streaming multiprocessor (SM), a graphics multiprocessor). For example, the processing resources may be one of the GPGPU cores (262) or tensor / ray tracing cores (263) of a graphics multiprocessor (234); ray-tracing cores (338A-338B), tensor cores (337A-337B), or GPGPU cores (336A-336B) of a graphics multiprocessor (324); execution resources (356A-356D) of a graphics multiprocessor (350); One of the GFX core (370), tensor core (371), or ray tracing core (372) of the multi-core group (365A-365N); one of the vector logic unit (1563) or scalar logic unit (1564) of the computing unit (1506A-1506N); an execution unit having an EU array (1522A-1522F) or an EU array (1524A-1524F); an execution unit (1808A-1808N) of the execution logic (1800); and / or an execution unit (1900). The processing resources may also be execution resources within, for example, a graphics processing engine (431-432), GPGPU hardware (610), GPGPU (700), a processing cluster (706A-706H), GPGPU (806A-806D), GPGPU (1306), a graphics processing engine (1610), a graphics processing engine cluster (1622), and / or a graphics processing engine (1710). The processing resources may also be processing resources within a graphics processor (2510), a graphics processor (2610), and / or a graphics processor (2640).

[0367] For example, when considering processing resources in the form of a graphics multiprocessor (324), the graphics multiprocessor (324) may include tensor and / or ray tracing cores (263). The graphics multiprocessor (325) may include tensor cores (337A-337B) and / or ray tracing cores (338A-338B). The multi-core group (365A) may include tensor cores (371) and ray tracing cores (372). The execution logic (1800) may include a ray tracer (1805). The execution logic (1900) may include a systolic array (1912). There is an advantage to tightly coupling this special function logic with more general-purpose logic found in an ALU and / or FPU. However, this architecture may require the special function logic to scale directly proportionally to the more general-purpose logic.

[0368] The embodiments described herein provide an architecture in which special function logic is separated from processing resources containing general-purpose execution logic. Such special function logic may include matrix / tensor acceleration used to accelerate ray-tracing or systolic arrays used for ray-tracing cores. Such an architecture may provide advantages including the ability to scale the number of general-purpose processing units independently of the special function logic. While examples relating to systolic arrays are given below, the separation technique described herein may also be generally applied to special function logic within a GPGPU.

[0369] FIG. 32 illustrates a computing block (3200) comprising separated systolic arrays (3212A-3212B) according to one embodiment. Instead of including a systolic array (2808) in a separate tensor accelerator (2723) as in FIG. 28, or including a systolic array (1912) in each execution unit (1900) as in FIG. 19, a set of separated systolic arrays (3212A-3212B) may be included in a computing block (3200) similar to one of the computing blocks (2724A-2724N) of FIG. 27. The computing block (3200) may also include components of the execution logic (1800) of FIG. 18a, comprising a plurality of interconnected processing resources (PR(3208A-3208O)) that may be similar to EU (1808A-1808N) or any other processing resource described herein.

[0370] A systolic array (3212A-3212B) comprises a W (wide) and D (deep) network of data processing units that can be used to perform vector or other data-parallel operations in a systolic manner, similar to other systolic arrays described herein. In one embodiment, the systolic array (3212A-3212B) may be configured to perform matrix operations, such as matrix inner product operations. In one embodiment, the systolic array (3212A-3212B) supports 16-bit floating-point operations and 8-bit and 4-bit integer operations. In one embodiment, the systolic array (3212) may be configured to accelerate machine learning operations. In this embodiment, the systolic array (3212) may be configured to support the bfloat 16-bit floating-point format. By including systolic arrays (3212A-3212B) outside of PRs (3208A-3208O) within a computing block (3200), the size and number of systolic arrays (3212A-3212B) can be expanded independently of the number of PRs (3208A-3208O). Additionally, communication bandwidth within the PRs that would be consumed by systolic array activities can be preserved. Furthermore, the systolic arrays (3212A to 3212B) can be clocked / power gated when no matrix workload is being performed.

[0371] Communication between the systolic array (3212A-3212) and PR (3208A-3208O) may be performed via a cache or shared local memory (cache / SLM (3210)) and / or a shared register file (3214). In one embodiment, instead of a separate shared register file (3214), the cache / SLM (3210) may be partitioned to be used as a shared register file. The shared register file (3214) may be configured similarly to other GPGPU register files, such as the register file (1906) of FIG. 19. The shared register file may also include a special-purpose set of registers used to configure the interaction between the systolic array (3212A-3212B) and PR (3208A-3208O). The cache / SLM (3210) may be a block of L1 cache, L2 cache, and / or explicitly addressable on-die memory.

[0372] Matrix data for processing by the systolic array (3212A-3212B) may be stored in the cache / SLM (3210). Processing commands or instructions may be provided to the systolic array (3212A-3212B) via the shared register file (3214). Processing results may be read by the PR (3208A-3208O) from the cache / SLM (3210) or from the destination / output registers within the shared register file. During operation, instead of consuming bus / fabric bandwidth within the PR (3208A-3208O), communication traffic may be localized to the systolic array (3212A-3212B), cache / SLM (3210), and / or the shared register file (3214). Any PR (3208A-3208O) within the computing block (3200) may offload a matrix workload to one or both of the systolic arrays (3212A-3212B). Messages may be transmitted from the PR to the systolic array along with commands specifying the operation to be performed and the operands for the operation. The systolic arrays (3212A-3212B) may perform the requested operation (multiplication / addition, fused multiplication / addition, multiplication / accumulation, inner product, etc.) and output the result to the shared register file (3214). Input, intermediate, and / or output data for the requested operation may be stored in the cache / SLM (3210), and multiple dependent operations may be chained. In one embodiment, when processing operations for training or inference on a neural network are performed, the systolic array (3128A-3128B) may also perform activation functions, including, but not limited to, sigmoid, ReLU, and hyperbolic tangent (TanH) activations. In this embodiment, operations on the neural network may be offloaded to the systolic array (3212A-3212B) with approximate granularity.

[0373] Using the above techniques, a person skilled in the art may implement a general-purpose graphics processing unit comprising, for example, a computing block comprising a first general-purpose processing resource set and a first matrix accelerator, wherein the first general-purpose processing resource set is connected to the first matrix accelerator through a shared register file. In one embodiment, the computing block comprises a second general-purpose processing resource set connected to the first matrix accelerator. In one embodiment, the computing block comprises a second matrix accelerator connected to the first general-purpose processing resource set and the second general-purpose processing resource set. A processing resource within the first general-purpose processing resource set or the second general-purpose processing resource set receives a decoded instruction associated with a matrix operation to be performed and may offload the matrix operation to the first matrix accelerator or the second matrix accelerator.

[0374] A matrix accelerator to which matrix operations are offloaded can perform matrix operations while the offloading general-purpose processing resource can perform other operations in parallel. For example, a thread running on the offloading general-purpose processing resource can offload an operation and stop execution. The offloading general-purpose processing resource can then context-switch to a different thread and continue thread execution until it receives notification of the completed offloaded operation. Afterward, the offloading general-purpose processing resource can context-switch back to the offloading thread, and the offloading thread reads the result of the offloaded operation through a shared register file and / or memory shared between the general-purpose processing resource and the matrix accelerator. The offloaded operation may include the matrix accelerator applying a neural network activation function to the processed neural network data.

[0375] Packed data compression and extension operations for GPGPU

[0376] Sparse data to be processed by the matrix accelerator can be compressed for transmission between function units or for storage in memory. Then, the sparse data can be expanded at the source. Enabling packed data compression and expansion operations can reduce the amount of bandwidth consumed on the device fabric and / or memory bus when transmitting sparse data.

[0377] FIG. 33 illustrates data types for packed bytes (3310), packed words (3320), and packed double words (3330) for operands with a width of 128 bits. The packed byte format (3310) of this example has a length of 128 bits and includes 16 packed byte data elements. Here, a byte is defined as 8 bits of data. Information for each byte data element is stored in bits 7 through 0 for byte 0, bits 15 through 8 for byte 1, bits 23 through 16 for byte 2, and finally bits 120 through 127 for byte 15. Thus, all available bits are used in the registers. This storage configuration increases the storage efficiency of the processor. Additionally, when 16 data elements are accessed, a single operation can now be performed in parallel on the 16 data elements.

[0378] Generally, a data element is an individual piece of data stored in a single register or memory location along with other data elements of the same length. In a packed data sequence, the number of data elements stored in a register may be 128 bits divided by the bit length of the individual data element. Similarly, in another packed data sequence, the number of data elements stored in a register may be 64 bits divided by the bit length of the individual data element. Although the illustrated data type has a length of 128 bits, the processor described herein may be configured for 64-bit operands, 32-bit operands, or operands of other sizes. The packed word format (3320) of this example has a length of 128 bits and contains eight packed word data elements. Each packed word contains 16 bits of information. The packed double word format (3330) has a length of 128 bits and contains four packed double word data elements. Each packed double word data element contains 32 bits of information. The packed quad-word has a length of 128 bits and includes two packed quad-word data elements. In one embodiment, the two packed 128-bit operands may be stored in a 256-bit GRF register (1824) as in FIG. 18b.

[0379] FIG. 34 illustrates a processing system (3400) comprising logic for executing packed data compression and expansion operations for a GPGPU. The processing system (3400) includes memory (3402, 3432) and registers for storing an operation code (3404), a source operand (3410), a control operand (3420), and a destination operation (3430). The processing system (3400) is configured to execute one or more GPGPU instructions for performing vector and / or packed data compression and at least one GPGPU instruction for performing vector and / or packed data expansion. The GPGPU instructions may be in the form of a 128-bit instruction format (2010) as in FIG. 20, or in the form of a 64-bit format (2030).

[0380] GPGPU instructions for compression and expansion may provide register operands to the source (3419) and / or destination (3430), and may implicitly load from memory (3402) or store in memory (3432) by referencing a memory address. When the source (3410) is loaded from memory (3402) or the destination (3430) is stored in memory (3432), the load or store may traverse one or more cache memories.

[0381] In the illustrated embodiment, the control operand (3420) is provided as a register, but other embodiments may be modified so that the source and / or destination operands each refer to a memory location. In one embodiment, the control operand (3420) may be an immediate value. The operation code (3404) may be fetched from a memory location and stored in an instruction buffer for decoding by a decoder (3406). The decoded operation code (3404) may be provided to a processing resource (3408) for execution.

[0382] A processing resource (3408) represents a processing element or functional unit found in one or more of a GPGPU core, ray tracing core, tensor core, execution resource, execution unit (EU), stream processor, streaming multiprocessor (SM), or graphics multiprocessor associated with a graphics processor or graphics processor structure (e.g., parallel processing unit, graphics processing engine, multi-core group, computing unit, computing unit of the next graphics core) in a GPU as described herein. For example, a processing resource (3408) may be an execution resource (e.g., ALU or FPU) of a GPGPU core (262) or tensor / ray tracing core (263) of a graphics multiprocessor (234); a ray-tracing core (338A-338B), tensor core (337A-337B), or GPGPU core (336A-336B) of a graphics multiprocessor (325); Execution resources (356A-356D) of a graphics multiprocessor (350); one of a GFX core (370), a tensor core (371), or a ray tracing core (372) of a multi-core group (365A-365N); one of a vector logic unit (1563) or a scalar logic unit (1564) of a computing unit (1506A-1506N); an execution unit having an EU array (1522A-1522F) or an EU array (1524A-1524F); an execution unit (1808A-1808N) of an execution logic (1800); and / or an execution unit (1900). The processing resource (3408) may also be, for example, an execution resource within a graphics processing engine (431-432), GPGPU hardware (610), GPGPU (700), processing cluster (706A-706H), GPGPU (806A-806D), GPGPU (1306), graphics processing engine (1610), graphics processing engine cluster (1622), and / or a graphics processing engine (1710).The processing resource (3408) may also be an execution resource within the graphics processor (2510), the graphics processor (2610), and / or the graphics processor (2640).

[0383] FIGS. 35a and FIGS. 35b illustrate GPGPU packed data compression and expansion operations. FIGS. 35a illustrates GPGPU packed data compression (3500). FIGS. 35b illustrates GPGPU packed data expansion (3550). GPGPU packed data compression (3500) and expansion (3550) can be performed by the processing system (3400) of FIG. 34. GPGPU packed data compression (3500) and expansion (3550) may be particularly useful when processing sparse data sets, but potential uses are not limited thereto. To compress a sparse data set, a comparison operation can be performed on the source data to generate a control vector that identifies non-zero elements of the packed data. The compression operation can be used to pack the sparse data for operation. The expansion operation can then be used to expand the result of the compressed operation.

[0384] As illustrated in FIG. 35a, a source (3410), which may be designated as a register or memory address, can store packed data in the form of elements H (3514) to A (3521). A source (3410) using a GPGPU packed data compression instruction can be compressed to a destination (3430) based on a control operand (3420). The control operand (3420) selects elements of the source (3410) to be copied to the destination (3430). The elements of the source (3410) are selected based on the corresponding elements (3524 to 3531) of the control operand (3420). The selected elements are copied to the destination (3430). As illustrated, the source elements B (3520), E (3517), F (3516), and H (3514) are selected by elements (3530), elements (3527), elements (3526), ​​and elements (3525). The selected elements (B 3531, E 3530, F 3529, H 2038) are compressed into the destination (3430), and the remaining elements of the destination (3430) (elements (3527), elements (3526), ​​elements (2025), and elements (3524)) are unaffected and remain 0 when the destination (3430) is initialized to 0, or may remain as any value or values ​​used to initialize the destination (3530). The unselected elements (G 3515, D 3518, C 3519, A 3521) of the source (3410) are not copied to the destination. In the case of sparse data, a comparison can be performed on the source (3410) to generate a mask of non-zero values, and the mask of non-zero values ​​can be used to generate a mask for the control operand (3420), so that a compression operation can be used to collect non-zero values ​​from the sparse dataset. The control operand data is preserved and can be used to expand the data after processing.

[0385] The source and destination may be bytes, 16-bit words, 32-bit double words, or 64-bit quad words. In one embodiment, the control operand (3420) uses 1 byte per vector element. The vector may contain multiple values ​​(e.g., 16, 128, 256, etc.). The GPGPU packed data compression instruction may be performed as shown in Table 5 below, where A is the control data, B is the destination, and C is the source data. NUM_ELEMENTS is the number of elements in the vector register or memory to be compressed.

[0386]

[0387] As illustrated in FIG. 35b, a GPGPU packed data extension (3550) operation can be used to extend data from a source (3410) to a destination (3430) based on a control operand (3420). The source (3410) and / or the destination (3430) may specify a register or memory. The control operand (3420) provides a map of elements within the destination (3430) to which elements of the source (3410) are to be written. Unselected elements of the destination (3430) may not be written and may retain their initial values. The number of elements of the source (3410) written to the destination depends on the number of destination elements selected for output. For example, the illustrated source (3410) stores elements (W (3554) to B (3561)). The control operand (3420) contains a selected value (e.g., "1") within the elements (3564), (3566), (3567), and (3570) corresponding to the elements (3574), (3576), (3577), and (3580) of the destination (3430). The elements of the source (3410) are selected from the lowest address of the vector register or memory toward a higher address. Since four elements are selected by the control operand (3420), the first four elements of the source (B (3561), E (3560), F (3559), H (3558)) are written to the indicated location of the destination (3430). The elements Z (3557), Y (3556), X (3555), and W (3554) are not copied to the destination. Vector extension movables can be expressed as shown in Table 6 below.

[0388]

[0389] Using the above techniques, a person skilled in the art may implement a graphics multiprocessor comprising, for example, a graphics processor core and an instruction decoder for decoding instructions for execution by the graphics processor core. The instruction may specify a source operand, a destination operand, and a control operand. When executed by the graphics processor core, the instruction causes the graphics processor core to read one or more elements of a set of packed data from a location identified by the source operand and to compress one or more elements to a location identified by the destination operand. One or more elements of the set of packed data are selected by corresponding elements of a control vector associated with the control operand. In one embodiment, the location identified by the source operand and / or the destination operand is a register in the register file of the graphics multiprocessor. In one embodiment, the location identified by the source operand and / or the destination operand is a register in the register file shared between a process element of the graphics multiprocessor and a matrix accelerator within or connected to the graphics multiprocessor. The location identified by the source operand may be a memory address in memory accessible to the graphics multiprocessor. The location identified by the destination operand may also be a memory address in memory accessible to the graphics multiprocessor. Using the above techniques, a person skilled in the art may implement a graphics multiprocessor comprising, for example, a graphics processor core and an instruction decoder for decoding instructions for execution by the graphics processor core.The instruction may specify a source operand, a destination operand, and a control operand, wherein, when executed, the instruction causes a graphics processor core to read one or more elements of a set of packed data from a location identified by the source operand, extend one or more elements to a location identified by the destination operand, and one or more elements of the set of packed data are written to elements at the location identified by the destination operand by corresponding elements of a control vector associated with the control operand. In one embodiment, the location identified by the source operand and / or the destination operand is a register in the register file of the graphics multiprocessor. In one embodiment, the location identified by the source operand and / or the destination operand is a register in the register file shared between a process element of the graphics multiprocessor and a matrix accelerator within or connected to the graphics multiprocessor. The l...

Claims

Claim 1 A general-purpose graphics processing unit comprising a matrix accelerator, wherein the matrix accelerator comprises a memory for storing input data, a systolic array coupled to the memory—the systolic array comprises a plurality of stages, each of the plurality of stages comprises a plurality of processing elements—and a circuit for bypassing a matrix multiplication operation having zero-value inputs, wherein the bypass is performed based on metadata associated with the inputs, and each of the plurality of processing elements comprises hardware logic for detecting zero-value inputs and bypassing the matrix multiplication operation based on zero-value inputs. Claim 2 In claim 1, the matrix accelerator is a general-purpose graphics processing unit that receives as input the metadata associated with an operand specifying the position for the input of the zero value. Claim 3 In paragraph 1, the metadata is a general-purpose graphics processing unit that is generated in advance for the entire input data set. Claim 4 A general-purpose graphics processing unit, wherein the metadata is generated in advance based on the rows of a first matrix to be input to the matrix accelerator or the columns of a second matrix to be input to the matrix accelerator. Claim 5 A general-purpose graphics processing unit according to claim 1, wherein the matrix accelerator generates the metadata based on data referenced by an input operand, and the data includes the input of the value of 0. Claim 6 In paragraph 5, the matrix accelerator is a general-purpose graphics processing unit that generates the metadata based on the rows of a first matrix to be input to the matrix accelerator and the columns of a second matrix to be input to the matrix accelerator. Claim 7 In paragraph 5, the matrix accelerator is a general-purpose graphics processing unit that generates the metadata in relation to the sub-matrix of the input before the input is loaded into the plurality of processing elements. Claim 8 A general-purpose graphics processing unit according to claim 7, wherein, in order to bypass the matrix multiplication operation based on the input of a value of 0, the processing element of the plurality of processing elements detects, based on the metadata, that at least one of the plurality of inputs for the first part of the matrix multiplication operation is an input of a value of 0, and bypasses the loading of the plurality of inputs to the processing element associated with the first part of the matrix multiplication operation. Claim 9 In claim 8, the processing element is a general-purpose graphics processing unit that bypasses a first matrix multiplication operation having an input of 0 within a single clock cycle and loads an input for a second matrix multiplication operation. Claim 10 A method comprising, in a general-purpose graphics processor having a matrix accelerator, a step of analyzing metadata to be input to a matrix multiplication operation performed by the matrix accelerator, wherein the input to the matrix multiplication operation comprises one or more elements of a plurality of input matrices; a step of determining, based on the metadata, whether the input to the matrix multiplication operation includes an input with a value of 0; and a step of bypassing at least a first part of the matrix multiplication operation in response to the determination that the matrix multiplication operation includes an input with a value of 0, wherein the step of bypassing at least the first part of the matrix multiplication operation comprises a step of determining that at least one of a plurality of inputs to the first part of the matrix multiplication operation is an input with a value of 0; and a step of bypassing the loading of the plurality of inputs to a processing element associated with the first part of the matrix multiplication operation. Claim 11 In claim 10, the step of bypassing at least the first part of the matrix multiplication operation further comprises: determining that each of a plurality of inputs for the second part of the matrix multiplication operation is a non-zero-value input; loading the plurality of inputs for the second part of the matrix multiplication operation into the processing element; and performing the second part of the matrix multiplication operation through the processing element. Claim 12 A method according to claim 11, wherein the loading of the plurality of inputs to the processing element associated with the first part of the matrix multiplication operation during the first clock cycle is bypassed, and the plurality of inputs for the second part of the matrix multiplication operation during the first clock cycle are loaded to the processing element. Claim 13 A method according to claim 10, further comprising the step of generating the metadata based on the rows of the first matrix to be input to the matrix accelerator and the columns of the second matrix to be input to the matrix accelerator. Claim 14 A method according to claim 10, further comprising the step of generating the metadata in relation to the sub-matrix of the input before the input is loaded into the plurality of processing elements. Claim 15 A data processing system comprising a memory device and a general-purpose graphics processing unit connected to the memory device, wherein the general-purpose graphics processing unit comprises a matrix accelerator comprising logic for bypassing a matrix multiplication operation having an input of a value of 0, wherein the bypass is performed based on metadata associated with the input, wherein the matrix accelerator comprises a plurality of processing elements, each of the plurality of processing elements comprises hardware logic for detecting an input of a value of 0 and bypassing the matrix multiplication operation based on the input of a value of 0, wherein the matrix accelerator receives as input the metadata associated with an operand specifying the location of the input of a value of 0, or generates the metadata based on data referenced by the input operand, wherein the data comprises the input of a value of 0. Claim 16 A data processing system according to claim 15, wherein the plurality of processing elements are configured as a systolic array of processing elements. Claim 17 In paragraph 15, the metadata is analyzed or generated in relation to a sub-matrix of the input before the input is loaded into the plurality of processing elements, in a data processing system. Claim 18 A data processing system according to claim 15, wherein the matrix accelerator generates the metadata based on the data referenced by the input operand, and the matrix accelerator generates the metadata based on the row of a first matrix to be input to the matrix accelerator and the column of a second matrix to be input to the matrix accelerator. Claim 19 A data processing system according to claim 15, wherein, in order to bypass the matrix multiplication operation based on the input of a value of 0, the processing element of the plurality of processing elements detects, based on the metadata, that at least one of the plurality of inputs for the first part of the matrix multiplication operation is an input of a value of 0, and bypasses the loading of the plurality of inputs to the processing element associated with the first part of the matrix multiplication operation. Claim 20 In claim 19, the processing element is a data processing system that bypasses a first matrix multiplication operation having an input of 0 value within a single clock cycle and loads an input for a second matrix multiplication operation.

Images