Variable precision and mixed-type representation of multiple layers in a network

Abstract

In one example, an apparatus comprises a plurality of execution units including at least a first type of execution unit and a second type of execution unit and logic, at least partially including hardware logic, to: expose an embedded projection operation in at least one of a load instruction or a store instruction; determine a target precision level for the projection operation; and load the projection operation at the target precision level. Other embodiments are also disclosed and claimed.

Description

TECHNICAL FIELD

[0001] Embodiments relate generally to data processing, and more specifically, to machine learning processing via general purpose graphics processing units. BACKGROUND

[0002] Machine learning has been successful in solving many types of tasks. The computations that arise when training and using machine learning algorithms (e.g., neural networks) naturally lend themselves to efficient parallel implementation. As a result, parallel processors such as general purpose graphics processing units (GPGPUs) play an important role in the practical implementation of deep neural networks. Parallel graphics processors with single instruction multiple thread (SIMT) architectures are designed to maximize the amount of parallel processing in a graphics pipeline. In a SIMT architecture, groups of parallel threads attempt to execute program instructions together as often as possible to improve processing efficiency. The efficiency provided by parallel machine learning algorithm implementations allows for the use of high-capacity networks and enables these networks to be trained on larger datasets. BRIEF DESCRIPTION OF DRAWINGS

[0003] So that the manner in which the above recited features of the present embodiments can be understood in detail, a brief description of the embodiments can be had by reference to a embodiment, some of which are illustrated in the appended drawings. It is to be noted, however, that the appended drawings illustrate only typical embodiments and are therefore not to be considered limiting of its scope.

[0004] Figure 1 is a block diagram illustrating a computer system configured to implement one or more aspects of the embodiments described herein;

[0005] Figures 2A-2D illustrates a parallel processor component in accordance with an embodiment;

[0006] Figures 3A-3B is a block diagram of a graphics multiprocessor in accordance with an embodiment;

[0007] Figures 4A-4F illustrates an exemplary architecture in which multiple GPUs are communicatively coupled to multiple multi-core processors;

[0008] Figure 5 is a conceptual diagram of a graphics processing pipeline in accordance with an embodiment.

[0009] Figures 6A-6B and Figure 7 illustrates exemplary architectures and operations in techniques in accordance with an embodiment.

[0010] Figure 8 illustrates a machine learning software stack in accordance with an embodiment;

[0011] Figure 9A highly parallel general purpose graphics processing unit is shown in accordance with an embodiment;

[0012] Figure 10 A multi-GPU computing system is shown in accordance with an embodiment;

[0013] Figures 11A-11B Layers of an exemplary deep neural network are shown;

[0014] Figure 12 An exemplary recurrent neural network is shown;

[0015] Figure 13 Training and deployment of a deep neural network is shown;

[0016] Figure 14 is a block diagram showing distributed learning;

[0017] Figure 15 An exemplary inference system-on-a-chip (SOC) suitable for performing inference using a trained model is shown;

[0018] Figure 16 is a block diagram of a processing system in accordance with an embodiment;

[0019] Figure 17 is a block diagram of a processor in accordance with an embodiment;

[0020] Figure 18 is a block diagram of a graphics processor in accordance with an embodiment;

[0021] Figure 19 is a block diagram of a graphics processing engine of a graphics processor in accordance with some embodiments;

[0022] Figure 20 is a block diagram of a graphics processor provided by additional embodiments;

[0023] Figure 21 Thread execution logic, including an array of processing elements employed in some embodiments, is shown;

[0024] Figure 22 is a block diagram showing a graphics processor instruction format in accordance with some embodiments;

[0025] Figure 23 is a block diagram of a graphics processor in accordance with another embodiment;

[0026] Figures 24A-24B Graphics processor command formats and command sequences in accordance with some embodiments are shown;

[0027] Figure 25 An exemplary graphics software architecture of a data processing system in accordance with some embodiments is shown;

[0028] Figure 26 is a block diagram illustrating an IP core development system in accordance with an embodiment;

[0029] Figure 27 is a block diagram illustrating an exemplary system on a chip integrated circuit in accordance with an embodiment;

[0030] Figure 28 is a block diagram illustrating an additional exemplary graphics processor; and

[0031] Figure 29 is a block diagram illustrating an additional exemplary graphics processor of a system on a chip integrated circuit in accordance with an embodiment. DETAILED DESCRIPTION

[0032] In the following description, numerous specific details are set forth to provide a thorough understanding of various embodiments. However, embodiments can be practiced without the specific details. In other instances, well-known methods, procedures, components and circuits have not been described in detail so as not to obscure the embodiments. Also, embodiments can be practiced using various means, such as integrated semiconductor circuits ("hardware"), computer-readable instructions

[0033] Some embodiments discussed herein can be applied to any processor (such as a GPCPU, CPU, GPU, etc.), graphics controller, etc. Other embodiments are also disclosed and claimed.

[0034] Further, some embodiments can be applied in computing systems including one or more processors (e.g., having one or more processor cores), such as those discussed herein, including, for example, mobile computing devices, such as smartphones, tablets, UMPCs (Ultra Mobile Personal Computers), laptop computers, ultrabooks TM computing devices, wearable devices (such as smart watches or smart glasses), and so on.

[0035] In some embodiments, a graphics processing unit (GPU) is communicatively coupled to the host / processor cores to accelerate graphics operations, machine learning operations, pattern analysis operations, and various general purpose GPU (GPGPU) functions. The GPU can be communicatively coupled to the host processor / cores over a bus or another interconnect (e.g., a high-speed interconnect such as PCIe or NVLink). In other embodiments, the GPU can be integrated on the same package or chip as the cores and communicatively coupled to the cores through an internal processor bus / interconnect (i.e., internal to the package or chip). Regardless of the manner in which the GPU is connected, the processor cores can allocate work to the GPU in the form of sequences of commands / instructions contained in a work descriptor. The GPU then uses dedicated circuitry and / or logic for processing these commands / instructions in high speed / efficient manner.

[0036] In the following description, numerous specific details are set forth to provide a more thorough understanding. However, it will be apparent to one of skill in the art that the embodiments described herein can be practiced without one or more of these specific details. In other instances, well-known features have not been described in order to avoid obscuring the details of the present embodiments.

[0037] System Overview

[0038] Figure 1 is a block diagram illustrating a computer system 100 configured to implement one or more aspects of the embodiments described herein. The computing system 100 includes a processing subsystem 101 having one or more processor(s) 102 and system memory 104, which are in communication via an interconnect path 1 10. The interconnect path 1 10 can include a memory bus, a processor bus, and / or a local system bus, among others. The interconnect path 1 10 can be configured to implement various protocols, including, but not limited to, I2C, SPI, PCI, and / or USB. The interconnect path 1 10 can be configured to enable data to be communicated between one or more of the components of the computing system 100.

[0039] In one embodiment, processing subsystem 101 includes one or more parallel processor(s) 112 coupled to memory hub 105 via a bus or other communication link 113. Communication link 113 can be one of any number of standards-based communication links, such as a PCI Express, or the like. In one embodiment, one or more parallel processor(s) 112 form a computationally-intensive, parallel, or vector processing system that can include a number of processing cores and / or clusters of processing cores, such as a many integrated core (MIC) processor. In one embodiment, one or more parallel processor(s) 112 form a graphics processing subsystem that can output pixels to one or more display device(s) 110A coupled via I / O hub 107. One or more parallel processor(s) 112 can also include a display controller and display interface (not shown) to enable a direct connection to one or more display device(s) 110B.

[0040] Within I / O subsystem 111, system storage 114 can connect to I / O hub 107 to provide storage mechanisms for computing system 100. I / O switch 116 can be used to provide an interface mechanism to enable connections between I / O hub 107 and other components such as network adapter 118 and / or wireless network adapter 119 that can be integrated into the platform, as well as various other devices that can be added via one or more add-in devices 120. Network adapter 118 can be an Ethernet adapter or another wired network adapter. Wireless network adapter 119 can include one or more of a Wi-Fi, Bluetooth, near- field communication (NFC), or other network device that includes one or more wireless radios.

[0041] Computing system 100 can include other components not explicitly shown, including USB or other port connections, optical storage drives, video capture devices, and the like, which can also be connected to I / O hub 107. Figure 1 Communication paths interconnecting various components can use any suitable protocol, such as a PCI (Peripheral Component Interconnect) - based protocol, for example, PCI-Express, or the like, or any other bus or point-to-point communication interface and / or protocol, such as NV-Link high-speed interconnect, or the like, known in the art.

[0042] In one embodiment, one or more parallel processors 112 incorporate circuitry optimized for graphics and video processing, including, for example, video output circuitry, and the circuitry constitutes a graphics processing unit (GPU). In another embodiment, one or more parallel processors 112 incorporate circuitry optimized for general use applications

[0043] It should be appreciated that the computing system 100 illustrated herein is exemplary and variations and modifications are possible. The connection topology can be modified as desired, including the number and arrangement of bridges, the number of processors 102, and the number of parallel processors 112. For example, in some embodiments, system memory 104 is connected to the processor(s) 102 directly rather than via a bridge, while other devices communicate with system memory 104 via the memory hub 105 and the processor(s) 102. In other alternative topologies, the parallel processor(s) 112 are connected to the I / O hub 107 or directly to one of the processor(s) 102, rather than to the memory hub 105. In still other embodiments, the I / O hub 107 and memory hub 105 are integrated into a single chip. Some embodiments can include two or more groups of the processor(s) 102 coupled by one or more interconnects, which can be part of the

[0044] Some of the particular components shown herein are optional and can not be included in all implementations of the computing system 100. For example, any number of add-in cards or peripheral devices can be supported, or some components can be omitted. Additionally, some architectures can use different terminology to describe what is Figure 1 similar to the components shown. For example, in some architectures the memory hub 105 can be called a northbridge and the I / O hub 107 can be called a southbridge.

[0045] Figure 2AA parallel processor 200 according to an embodiment is shown. Various components of the parallel processor 200 can be implemented using one or more integrated circuit devices such as programmable processors, application specific integrated circuits (ASICs), or field programmable gate arrays (FPGAs). According to an embodiment, the illustrated parallel processor 200 is a graphics processor that has been simplified to focus on the overall parallel processing structure as it is used in various embodiments. Those skilled in the art will recognize that a typical graphics processor can include many more functional units and interconnects. Figure 1 Variations of the one or more parallel processors 112 shown are possible.

[0046] In one embodiment, the parallel processor 200 includes a parallel processing unit 202. The parallel processing unit includes an I / O unit 204 that enables communication with other devices, including other instances of the parallel processing unit 202. The I / O unit 204 can be directly connected to the other devices. In one embodiment, the I / O unit 204 communicates with other devices such as the memory hub 105 via the use of a hub or switch interface, such as the memory hub 105. The connections between the memory hub 105 and the I / O unit 204 form a communication link 113. Within the parallel processing unit 202, the I / O unit 204 is connected to a host interface 206 and a memory crossbar switch 216, where the host interface 206 receives commands required to carry out processing tasks and the memory crossbar switch 216 facilitates communication between the parallel processing unit 202 and the memory array 212.

[0047] When the host interface 206 receives a command buffer from the I / O unit 204, the host interface 206 can direct the work operations required to execute those commands to the front end 208. In one embodiment, the front end 208 is coupled with a scheduler 210 that is configured to distribute commands or other work items to the processing cluster array 212. In one embodiment, the scheduler 210 ensures that the processing cluster array 212 is correctly configured and that commands are issued to the processing clusters in the processing cluster array 212 in an even manner.

[0048] The processing cluster array 212 can include up to "N" processing clusters (e.g., cluster 214A, 214B, through 214N). Each cluster 214A-214N of the processing cluster array 212 can execute a large number of concurrent threads. The scheduler 210 can allocate work to the clusters 214A-214N of the processing cluster array 212 using a variety of scheduling and / or work distribution algorithms. The scheduling can be dynamic, with the scheduler 210 allocating work to the clusters 214A-214N with little or no human intervention. In other embodiments, the scheduling can be specified manually or as the result of a scheduling tool.

[0049] The processing cluster array 212 can be configured to perform a variety of types of parallel processing operations. In one embodiment, the processing cluster array 212 is configured to perform general-purpose parallel compute operations. For example, the processing cluster array 212 can include logic to perform processing tasks including filtering of video and / or audio data, performance of modeling operations including physics operations, and performance of data transformations.

[0050] In one embodiment, the processing cluster array 212 is configured to perform parallel graphics processing operations. In an embodiment in which the parallel processor 200 is configured to perform graphics processing operations, the processing cluster array 212 can include additional logic to support the execution of such graphics processing operations including without limitation texture mapping logic used to perform texture operations, and surface rendering logic used to perform surface rendering operations. In addition, the processing cluster array 212 can be configured to execute shader programs, including, but not limited to, vertex shaders, tessellation shaders, geometry shaders, and pixel

[0051] In one embodiment, when the parallel processing unit 202 is used to perform graphics processing, the scheduler 210 can be configured to divide the processing workload into approximately equal sized tasks, to better enable distribution of the graphics processing operations to multiple clusters 214A-214N of the processing cluster array 212. In some embodiments, portions of the processing cluster array 212 can be configured to perform different types of processing. For example a first portion can be configured to perform vertex shading and topology generation, a second portion can be configured to perform surface rendering and geometry shading, a third portion can be configured to perform pixel shading or other screen space operations, etc. Intermediate data produced by one or more of the clusters 214A-214N can be stored in buffers to allow the intermediate data to be transmitted between clusters 214A-214N for further processing.

[0052] During operation, the processing cluster array 212 can receive processing tasks to be performed via the scheduler 210, which receives commands defining the processing tasks from the front end 208. For graphics processing operations, the processing task can include data to be processed, such as surface (patch) data, geometric data, vertex data, and / or pixel data, and state parameters and commands defining how the data is to be processed (e.g., which program is to be executed). The scheduler 210 can be configured to fetch the indices of the tasks corresponding to the indices or can receive the indices from the front end 208. The front end 208 can be configured to ensure the processing cluster array 212 is configured in an effective state prior to a workload specified by an incoming command buffer (e.g., a batch buffer, an in-stall buffer, etc.) being launched.

[0053] Each of the one or more instances of parallel processing unit 202 can be coupled with a parallel processor memory 222 via a memory crossbar 216. The memory crossbar 216 can receive memory requests from the processing cluster array 212 as well as the I / O unit 204. The memory crossbar 216 can be configured to route memory requests from the processing clusters 210A-210N and / or the I / O unit 204 to the parallel processor memory 222 via a memory interface 218. The memory interface 218 can include a number of memory channels 212A-212N configured to provide parallel memory access via parallel processor memory 222. In one embodiment, the number of memory channels 212A-212N can be equal to the number of memory units 224A-224N. In other embodiments, the number of memory channels 212A-212N can be greater than or fewer than the number of memory units 224A-224N.

[0054] In various embodiments, memory units 224A-224N can include various types of memory devices including dynamic random access memory (DRAM) or graphics random access memory, such as synchronous graphics random access memory (SGRAM), including graphics double data rate (GDDR) memory. In one embodiment, memory units 224A-224N can also include 3D stacked memory, including but not limited to high bandwidth memory (HBM). As will be appreciated by one skilled in the art, the specific implementation of memory units 224A-224N can vary from one embodiment to the next and can be selected from one of various conventional designs. Render targets such as frame buffers or texture maps can be stored on memory units 224A-224N, allowing partition units 220A-220N to write portions of each render target in parallel for efficient use of the available bandwidth of parallel processor memory 222. In some embodiments, to support a unified memory design that utilizes system memory in conjunction with local cache memory, a local instance of parallel processor memory 222 can be excluded.

[0055] In one embodiment, any of clusters 214A-214N of processing cluster array 212 can process data to be written to any of memory units 224A-224N within parallel processor memory 222. Memory crossbar 216 can be configured to pass the output of each cluster 214A-214N to any partition unit 220A-220N or another cluster 214A-214N, which can perform additional processing operations on the output. Each cluster 214A-214N can communicate with memory interface 218 through memory crossbar 216 to perform read or write operations for various external memory devices. In one embodiment, memory crossbar 216 can be connected to memory interface 218 to communicate with I / O unit 204 and to a local instance of parallel processor memory 222, enabling processing units within the different processing clusters 214A-214N to communicate with system memory or other memory not local to parallel processor 202. In one embodiment, memory crossbar 216 can use virtual channels to separate traffic streams between clusters 214A-214N and partition units 220A-220N.

[0056] While a single instance of the parallel processing unit 202 is shown within the parallel processor 200, any number of instances of the parallel processing unit 202 can also be included. For example, multiple instances of the parallel processing unit 202 can be provided on a single plug-in card, or multiple plug-in cards can be interconnected. Different instances of the parallel processing unit 202 can be configured to interact even if the different instances have different numbers of processing cores, different amounts of local parallel processor storage, and / or other configuration differences. For example, and in one embodiment, some instances of the parallel processing unit 202 may include higher precision floating-point units relative to other instances. Systems incorporating one or more instances of the parallel processing unit 202 or the parallel processor 200 can be implemented in various configurations and form factors, including but not limited to desktop computers, laptop or handheld personal computers, servers, workstations, game consoles, and / or embedded systems.

[0057] Figure 2B This is a block diagram of partitioning unit 220 according to an embodiment. In one embodiment, partitioning unit 220 is... Figure 2A An example of one of partition units 220A to 220N. As shown, partition unit 220 includes an L2 cache 221, a frame buffer interface 225, and a ROP 226 (Raster Operation Unit). The L2 cache 221 is a read / write cache configured to perform load and store operations received from memory crossbar switch 216 and ROP 226. Read misses and urgent write-back requests are output by the L2 cache 221 to the frame buffer interface 225 for processing. Dirty updates can also be sent to the frame buffer via the frame buffer interface 225 for opportunistic processing. In one embodiment, the frame buffer interface 225 interacts with one of the memory cells in the parallel processor memory (e.g., memory cells 224A to 224N of FIG. 2 (e.g., within parallel processor memory 222)).

[0058] In graphics applications, ROP 226 is a processing unit that performs raster operations such as stencil printing, z-testing, and blending. ROP 226 then outputs processed graphics data, which is stored in graphics memory. In some embodiments, ROP 226 includes compression logic for compressing z- or color data written to memory and decompressing z- or color data read from memory. In some embodiments, ROP 226 is included within each processing cluster (e.g., clusters 214A to 214N of FIG. 2) rather than within partitioning unit 220. In such embodiments, read and write requests for pixel data are transmitted via memory crossbar 216 rather than pixel fragment data. The processed graphics data can be displayed on a display device such as...Figure 1 one of the one or more display devices 110, by the processor(s) 102 for further processing, or by one of the processing entities within the parallel processor 200 for further processing. Figure 2A one of the processing entities within the parallel processor 200 for further processing.

[0059] Figure 2C is a block diagram of a processing cluster 214 within a parallel processing unit of an embodiment. In one embodiment, the processing cluster is an instance of one of the processing clusters 214A-214N of FIG. 2. A processing cluster 214 can be configured to execute a plurality of threads in parallel, where the term "thread" refers to an instance of a particular program executing on a particular set of input data. In some embodiments, single-instruction, multiple-data (SIMD) instruction issue techniques are used to support parallel execution of a large number of threads on a single processor. In other embodiments, single-instruction, multiple-thread (SIMT) techniques are used to support parallel execution of multiple threads through the use of a common instruction unit configured to issue instructions to a set of processing engines within each of the processing clusters. Unlike a SIMD execution mechanism in which all processing engines typically execute identical instructions, a SIMT execution mechanism allows for divergent execution paths across threads following divergent branches in a given thread program. Those of skill in the art will understand that the SIMD processing mechanism represents a functional subset of the SIMT processing mechanism.

[0060] Operation of the processing cluster 214 can be controlled via a pipeline manager 232 that distributes processing tasks to SIMT parallel processors. The pipeline manager 232 receives instructions from the scheduler 210 of FIG. 2 and manages execution of those instructions via a graphics multiprocessor 234 and / or a texture unit 236. The graphics multiprocessor 234 shown is an exemplary instance of a SIMT parallel processor. However, different types of SIMT parallel processors of different architectures can be included within the processing cluster 214. One or more instances of the graphics multiprocessor 234 can be included within a processing cluster 214. The graphics multiprocessor 234 can process data and a data crossbar 240 can be used to distribute the processed data to one of multiple possible destinations. The pipeline manager 232 can facilitate distribution by specifying destinations for processed data to be distributed via the data crossbar 240.

[0061] Each graphics multiprocessor 234 within processing cluster 214 can include the same functional execution logic, which can be configured in a pipelined manner to process audio and video instructions. The functional execution logic can support a variety of operations including integer and floating-point arithmetic, comparison operations, Boolean operations, bit-shifting operations, and computation of various algebraic functions. In one embodiment, same functional -unit hardware can be leveraged to perform different operations using different software instructions or microcodes and / or any combination of hardware state and a software instruction.

[0062] Instructions transmitted to the processing cluster 214 form a thread. A set of threads executed on a group of parallel processing engines forms a warp. Threads in a warp are executed in parallel in a multiprocessor. Each thread can be assigned a unique thread identifier, thread ID, which can be used to determine the unique address of each thread's local data store. Each thread can also be assigned a unique lane ID. The lane ID can be used in situations where a thread handling an instruction branch can fork a thread to be executed, and each forked thread can be assigned a unique lane ID.

[0063] In one embodiment, the graphics multiprocessor 234 includes an internal cache memory to perform load and store operations. In one embodiment, the graphics multiprocessor 234 can discard the internal cache memory and use the cache memory within the processing cluster 214 (e.g., the LI cache 308). Each graphics multiprocessor 234 can also have access to the L2 cache within the partition unit (e.g., partition units 220A-220N of FIG. 2) shared between all processing clusters 214, and can be used to transfer data between threads. The graphics multiprocessor 234 can also have access to off-chip global memory, which can include one or more of a local parallel processor memory and a system memory. Any memory external to the parallel processor unit 202 can be accessed by the graphics multiprocessor 234 through the off-chip memory interface. Embodiments in which the processing cluster 214 includes multiple instances of graphics multiprocessor 234 can share common instructions and data stored within the LI cache 308.

[0064] Each processing cluster 214 can include an MMU 245 (memory management unit) configured to translate virtual addresses into physical addresses. In other embodiments, one or more instances of the MMU 245 can reside within the memory interface 218 of FIG. 2. The MMU 245 includes a set of page table entries (PTEs) used to translate virtual addresses into physical addresses and optionally cache line indices. The MMU 245 can include an address translation lookaside buffer (TLB) or cache to reduce the latency of address translations. The physical addresses are processed to distribute surface data access locality to achieve efficient request interleaving among partition units. The cache line indices can be used to determine whether requests for cache lines are a hit or miss.

[0065] In graphics and compute applications, the processing cluster 214 can be configured such that each graphics multiprocessor 234 is coupled to a texture unit 236 for performing texture mapping operations, e.g., determining texture sample positions, reading texture data, and filtering texture data. Texture data can be read from an internal texture Ll cache (not shown) or, in some embodiments, from an Ll cache within the graphics multiprocessor 234, and is fetched from an L2 cache, a local parallel processor memory, or system memory, as needed. Each graphics multiprocessor 234 outputs processed tasks to a data crossbar 240 that couples to another processing cluster 214 for further processing or to a memory crossbar 216 for processing via memory mapped I / O (MMIO) 272. A preROP 242 (pre-raster operations unit) is configured to receive data from the graphics multiprocessor 234, direct data to a ROP unit which can be located within the partition unit (e.g., partition unit 220A-220N of FIG. 2) as described herein. The preROP 242 unit can optimize any color blends, organize pixel color data, and perform address translations.

[0066] It should be understood that the core architecture described herein is illustrative and that variations and modifications are possible. For example, any number of processing units, such as the graphics multiprocessor 234, texture unit 236, preROP 242, etc., can be included within processing cluster 214. Further, while only one processing cluster 214 is shown, parallel processing units can include any number of instances of processing cluster 214 as described herein. In one embodiment, each processing cluster 214 can be configured to operate independently of the other processing clusters 214 using separate and different processing units, Ll caches, etc.

[0067] Figure 2DA graphics multiprocessor 234 according to one embodiment is shown. In such an embodiment, the graphics multiprocessor 234 is coupled with the pipeline manager 232 of the processing cluster 214. The graphics multiprocessor 234 has an execution pipeline that includes, without limitation, an instruction cache 252, an instruction unit 254, an address mapping unit 256, a register file 258, one or more general-purpose graphics processing unit (GPGPU) cores 262, and one or more load / store units 266. The GPGPU cores 262 and the load / store units 266 are coupled with a memory and cache interconnect 268 to cache memory 272 and shared memory 270.

[0068] In one embodiment, the instruction cache 252 receives a stream of instructions to execute from the pipeline manager 232. These instructions are cached in the instruction cache 252 and dispatched to the instruction unit 254 for execution. The instruction unit 254 can dispatch the instructions as threads to different ones of the GPGPU cores 262, each one of which can be allocated to different ones of the execution units within the GPGPU cores 262. The instructions can access the same local, shared, or global address space that is used by the GPGPU cores 262. The address mapping unit 256 can be used to convert addresses in the unified address space to different memory addresses available to the load / store units 266.

[0069] The register file 258 provides a set of registers to the functional units of the graphics multiprocessor 324. The register file 258 provides temporary storage for operands to the data paths connected to the functional units (e.g., GPGPU cores 262, load / store units 266) of the graphics multiprocessor 324. In one embodiment, the register file 258 is split into registers files for each of the functional units. In one embodiment, the register file 258 is split into registers files for each of the different threads executed by the graphics multiprocessor 324.

[0070] The GPGPU cores 262 can each include floating point units (FPUs) and / or integer arithmetic logic units (ALUs) that are capable of performing computational operations on received data. The GPGPU cores 262 can each maintain their own program counter, instruction pointer, and architectural state. In one embodiment, the GPGPU cores 262 maintain their program counters based on the program counter of the GPGPU cores 262 that they are replicated from. In one embodiment, the GPGPU cores 262 are capable of executing single instruction multiple data (SIMD) or single instruction multiple thread (SIMT) instructions. In one embodiment the GPGPU cores 262 execute an instruction in the single instruction multiple thread (SIMT) mode.

[0071] The memory and cache interconnect 268 is an interconnect network that connects each of the functional units of the graphics processing engine 320 to the register file 258 and shared memory 270. In one embodiment the memory and cache interconnect 268 is a crossbar interconnect that allows loads and stores from the load / store units 266 to be implemented between the shared memory 270 and the register file 258. The register file 258 can operate at the same frequency as the GPGPU cores 262, thus data transfers between the GPGPU cores 262 and the register file 258 have very low latency. The shared memory 270 can be used for communication between threads executing on the functional units within the graphics processing engine 234. For example, the cache memory 272 can be used as a data cache to cache texture data communicated between the functional units and the texture unit 236. The shared memory 270 can also be used to cache managed programs. In addition to the auto-cached data stored within the cache memory 272, threads executing on the GPGPU cores 262 can also programmatically store data within the shared memory.

[0072] Figures 3A-3B Additional graphics processing engines according to embodiments are shown. The illustrated graphics processing engines 325, 350 are variants of the graphics processing engine 234 of Figure 2C FIG. 2. The illustrated graphics processing engines 325, 350 can be configured to execute a large number of execution threads concurrently as stream multi-processors (SMs).

[0073] Figure 3A A graphics processing engine 325 according to an additional embodiment is shown. The graphics processing engine 325 includes a number of the graphics processing engines 320 relative to Figure 2DThe graphics processor 325 can include additional instances of execution resource units such as instruction units 332A-B, register file units 334A-B, and texture units 344A-B. The graphics processor 325 also includes groups of graphics or compute execution units (e.g., GPGPU cores 336A-B, GPGPU cores 337A-B, GPGPU cores 338A-B) and groups of load / store units 340A-B. In one embodiment, the execution resource units have a common instruction cache 330, texture and / or data cache memory 342, and shared memory 346. The various components can communicate via an interconnect fabric 327. In one embodiment, the interconnect fabric 327 includes one or more crossbars to facilitate communication between components of the graphics processor 325.

[0074] Figure 3B A graphics processor 350 is shown in accordance with an additional embodiment. As shown in Figure 2D and Figure 3A The graphics processor includes groups of execution resources 356A-D, where each group of execution resources includes multiple instruction units, register files, GPGPU cores, and load store units. The execution resources 356A-D can work with texture units 360A-D to perform texture operations, while sharing an instruction cache 354 and shared memory 362. In one embodiment, the execution resources 356A-D can share an instruction cache 354 and shared memory 362, as well as multiple instances of texture and / or data cache memory 358A-B. The various components can communicate via an interconnect fabric 352 similar to the interconnect fabric 327. Figure 3A

[0075] Those of skill in the art will appreciate that the architectures described in Figure 1 , Figures 2A-2D and Figures 3A-3B are descriptive, and do not limit the scope of embodiments of the present application. Thus, the techniques described herein can be implemented on any suitably configured processing unit, including but not limited to: one or more mobile application processors; one or more desktop or server central processing units (CPUs), including multi-core CPUs; one or more parallel processing units such as the parallel processing unit 202 of FIG. 2; and one or more graphics processors or specialized processing units, without departing from the scope of embodiments described herein.

[0076] ​In some embodiments, a parallel processor or GPGPU as described herein is communicatively coupled to a host / processor core to accelerate graphics operations, machine learning operations, pattern analysis operations, and various general purpose GPU (GPGPU) functions. The GPU can be communicatively coupled to the host processor / cores over a bus or other interconnect (e.g., a high-speed

[0077] Techniques for GPU to host processor interconnect

[0078] Figure 4A An exemplary architecture is shown in which a plurality of GPUs 410-413 are communicatively coupled to a plurality of multi-core processors 405-406 over high-speed links 440-443 (e.g., buses, point-to-point interconnects, etc.). In one embodiment, the high-speed links 440-443 support communications at a throughput level of 4GB / s, 30GB / s, 80GB / s or higher, depending on the implementation. Various interconnect protocols can be used including, but not limited to, PCIe 4.0 or 5.0 and NVLink 2.0. However, the underlying principles of the application are not limited to any particular communication protocol or throughput level.

[0079] In addition, in one embodiment, two or more of the GPUs 410-413 are interconnected over high-speed links 444-445, which can be implemented using the same or different protocol / links than used for high-speed links 440-443. Similarly, two or more of the multi-core processors 405-406 can be connected over high-speed link 433, which can be an SMP bus that runs at 20GB / s, 30GB / s, 120GB / s or higher. Alternatively, Figure 4A All communication between the various system components shown in FIG. 4 can be accomplished using the same protocol / s (e.g., over a common interconnect architecture). However, as mentioned, the underlying principles of the application are not limited to any particular type of interconnect technology.

[0080] In one embodiment, each multi-core processor 405-406 is communicatively coupled to processor memories 401-402 via memory interconnects 430-431, respectively, and each GPU 410-413 is communicatively coupled to GPU memories 420-423 through GPU memory interconnects 450-453, respectively. Memory interconnects 430-431 and 450-453 can utilize the same or different memory access technologies. By way of example and not limitation, processor memories 401-402 and GPU memories 420-423 can be volatile memory such as dynamic random access memory (DRAM) including stack DRAM, graphics DDR SDRAM (GDDR) such as GDDR5, GDDR6, or high bandwidth memory (HBM), and / or can be non-volatile memory such as 3D XPoint or Nano-Ram. In one embodiment, some portion of the memory can be volatile memory while another portion can be non-volatile memory (e.g., using a two-level memory (2LM) hierarchy).

[0081] As described below, although various processors 405-406 and GPUs 410-413 can each be physically coupled to a particular memory 401-402, 420-423, respectively, a unified memory architecture can be implemented in which the same virtual system address space (also referred to as "effective address" space) is distributed across all of the various physical memories. For example, processor memories 401-402 can each include 64 GB of system memory address space, and GPU memories 420-423 can each include 32 GB of system memory address space (resulting in a total of 256 GB of addressable memory space in the example described).

[0082] Figure 4B Additional details of the interconnect between multi-core processor 407 and graphics acceleration module 446 according to one embodiment are shown. Graphics acceleration module 446 can include one or more GPU chips integrated on a line card that is coupled to processor 407 via high-speed link 440. Alternatively, graphics acceleration module 446 can be integrated on the same package or chip as processor 407.

[0083] The illustrated processor 407 includes multiple cores 460A-D, each with a translation lookaside buffer 461A-D and one or more caches 462A-D. The cores can include various other components (e.g., instruction fetch unit, branch prediction unit, decoders, execution units, reorder buffer, etc.) to execute instructions and process data not shown to avoid obscuring the inventive subject matter. The caches 462A-D can include level one (Ll) and level two (L2) caches. In addition, one or more shared caches 426 can be included in the cache hierarchy and shared by the various groups of cores 460A-D. For example, one embodiment of the processor 407 includes 24 cores each with its own Ll cache, 12 shared L2 caches, and 12 shared L3 caches. In this embodiment, one of the L2 and L3 caches is shared by two adjacent cores. The processor 407 and graphics accelerator integrated module 446 are connected with system memory 441, which can include processor memories 401-402.

[0084] Data and instructions stored in various caches 462A-D, 456 and system memory 441 are kept coherent across the various caches and system memory by inter-core communications over coherency bus 464. For example, each cache can have cache coherency logic / circuitry associated therewith to communicate over coherency bus 464 in response to a detected read or write to a particular cache line. In one implementation, a cache snoop protocol is implemented over coherency bus 464 to snoop cache accesses. Cache snoop / coherency techniques are well understood by those skilled in the art to avoid obscuring the inventive subject matter and are not described in detail here.

[0085] In one embodiment, the proxy circuit 425 communicatively couples the graphics acceleration module 446 to the coherency bus 464, allowing the graphics acceleration module 446 to participate in the cache coherency protocol as a peer to the cores. Specifically, the interface 435 provides connectivity to the proxy circuit 425 over a high-speed link 440 (e.g., a PCIe bus, NVLink, etc.), and the interface 437 connects the graphics acceleration module 446 to the link 440.

[0086] In one implementation, the accelerator integration circuit 436 provides cache management, memory access, context management, and interrupt management services on behalf of multiple graphics processing engines 431, 432, 43N of the graphics acceleration module 446. The graphics processing engines 431, 432, 43N can each comprise a separate graphics processing unit (GPU). Alternatively, the graphics processing engines 431, 432, 43N can comprise different types of graphics processing engines such as graphics execution units, media processing engines (e.g., video encoders / decoders), samplers, and block image transfer engines within a GPU. In other words, the graphics acceleration module can be a GPU with a plurality of graphics processing engines 431, 432, 43N, or the graphics processing engines 431, 432, 43N can be separate GPUs integrated on a common package, line card, or chip.

[0087] In one embodiment, the accelerator integration circuit 436 includes a memory management unit (MMU) 439 to perform various memory management functions such as translation of virtual addresses into physical addresses (also known as virtual to physical translations or VaPTs), and to implement memory access protocols. The MMU 439 can also include a translation lookaside buffer (TLB) to cache recently translated addresses (not shown). In one implementation, the cache 438 stores commands and data for efficient processing by the graphics processing engines 431, 432, 43N. In one embodiment, the data stored in the cache 438 and the graphics memories 433, 434, 43N are kept coherent with the core caches 462A, 462D, 456 and the system memory 411. As noted, this can be accomplished via the agent circuit 425 participating in the cache coherence mechanism (e.g., sending updates to the cache 438 related to modifications / accesses to cache lines on the processor caches 462A, 462D, 456 and receiving updates from the cache 438).

[0088] A set of registers 445 store context data for threads executed by the graphics processing engines 431-432, 43N, and a context management circuit 448 manages thread contexts. For example, the context management circuit 448 can perform save and restore operations to save and restore the context of various threads during context switches (e.g., where a first thread is saved and a second thread is stored so that the second thread can be executed by the graphics processing engines). For example, upon context switch, the context management circuit 448 can store current register values to a designated area in memory (e.g., identified by a context pointer). The context management circuit can restore the register values upon return from context. In one embodiment, an interrupt management circuit 447 receives and processes interrupts received from system devices.

[0089] In one implementation, virtual / effective addresses from the graphics processing engines 431 are translated to real / physical addresses in system memory 411 by the MMU 439. One embodiment of the accelerator integration circuit 436 supports multiple (e.g., 4, 8, 16) graphics processor modules 446 and / or other accelerator devices. The graphics processor modules 446 can be dedicated to a single application executing on the processor 407 or shared between multiple applications. In one embodiment, a virtualized graphics execution environment is presented in which resources of the graphics processing engines 431-432, 43N are shared between multiple applications or virtual machines (VMs). The resources can be subdivided into "tiles" that are allocated to different VMs and / or applications based on their processing requirements and priorities.

[0090] Thus, the accelerator integration circuit functions as a bridge of the system for the graphics acceleration module 446 and provides address translation and system memory cache services. In addition, the accelerator integration circuit 436 can provide virtualization facilities for the host processor to manage virtualization of graphics processing engines, interrupts, and memory management.

[0091] Since the hardware resources of the graphics processing engines 431-432, 43N are explicitly mapped to the real address space seen by the host processor 407, any host processor can use effective address values to directly address these resources. In one embodiment, a function of the accelerator integration circuit 436 is to physically separate the graphics processing engines 431-432, 43N so that they appear as independent units on the system.

[0092] As mentioned, in the illustrated embodiment, one or more graphics memory 433-434, 43M is coupled to each of the graphics processing engines 431-432, 43N. The graphics memory 433-434, 43M stores instructions and data for processing by each of the graphics processing engines 431-432, 43N. The graphics memory 433-434, 43M can be volatile memory such as DRAM (including stacked DRAM), GDDR memory (e.g., GDDR5, GDDR6), or HBM, and / or can be non-volatile memory such as 3D XPoint or Nano-Ram.

[0093] In one embodiment, to reduce the data traffic on the link 440, a biasing technique is used to ensure that the data stored in the graphics memory 433-434, 43M is that which is most frequently used by the graphics processing engines 431-432, 43N and that which is preferably not used (at least not frequently) by the cores 460A-D. Similarly, the biasing mechanism attempts to keep data required by the cores (and preferably not the graphics processing engines 431-432, 43N) within the caches 462A-D, 456 of the cores and system memory 411.

[0094] Figure 4C Another embodiment is illustrated in which the accelerator integration circuit 436 is integrated within the processor 407. In this embodiment, the graphics processing engines 431-432, 43N communicate directly over high-speed link 440 to the accelerator integration circuit 436 via the interface 437 and the interface 435 (which can also utilize any form of bus or interface protocol). The accelerator integration circuit 436 can execute same operations as described with respect to the Figure 4B the same operations as described with respect to the

[0095] One embodiment supports different programming models, including a dedicated process programming model (without graphics acceleration module virtualization) and a shared programming model (with virtualization). The shared programming model can include a programming model controlled by the accelerator integration circuit 436 and a programming model controlled by the graphics acceleration module 446.

[0096] In one embodiment of the dedicated process model, the graphics processing engines 431-432, 43N are dedicated to a single application or process under a single operating system. The single application can centralize other application requests to the graphics engines 431-432, 43N, providing virtualization within the VM / partition.

[0097] In a dedicated process programming model, graphics processing engines 431-432, 43N can be shared by multiple VM / application partitions. The shared model requires a system hypervisor that is used to virtualize the graphics processing engines 431-432, 43N to allow access by each operating system. For single-partition systems without a hypervisor, the graphics processing engines 431-432, 43N are owned by the operating system. In both cases, the operating system can virtualize the graphics processing engines 431-432, 43N to provide access to each process or application.

[0098] For the shared programming model, graphics acceleration module 446 or individual graphics processing engines 431-432, 43N use a process handle to select a process element. In one embodiment, the process element is stored in system memory 411 and is addressable using the effective-to-real address translation techniques described herein. The process handle can be an implementation-specific value provided to the host process when it registers its context with the graphics processing engines 431-432, 43N (i.e., calls system software to add the process element to a process element linked list). The lower 16 bits of the process handle can be an offset into the process element linked list of the process element.

[0099] Figure 4D An exemplary accelerator integration tile 490 is shown. As used herein, a "tile" includes a specified portion of the processing resources of accelerator integration circuit 436. Application effective address space 482 within system memory 411 stores process elements 483. In one embodiment, process elements 483 are stored in response to GPU invocations 481 from an application 480 executing on processor 407. Process elements 483 contain processing state for the respective application 480. A work descriptor (WD) 484 contained in process element 483 can be a single job requested by an application or can contain a pointer to a job queue. In the latter case, WD 484 is a pointer to a job request queue in application address space 482.

[0100] Graphics acceleration module 446 and / or individual graphics processing engines 431-432, 43N can be shared by all or a subset of processes in a system. Embodiments of the present invention include infrastructure for setting up processing state and sending WDs 484 to graphics acceleration module 446 to start jobs in a virtual environment.

[0101] In one implementation, the dedicated process programming model is specific to the implementation. In this model, a single process owns the graphics acceleration module 446 or individual graphics processing engines 431. Since the graphics acceleration module 446 is owned by a single process, the hypervisor initializes the accelerator integration circuit 436 for the owning partition and the operating system initializes the accelerator integration circuit 436 for the owning process when the graphics acceleration module 446 is allocated.

[0102] In operation, the WD fetch unit 491 in the accelerator integration slice 490 fetches the next WD 484, which includes an indication of work to be done by one of the graphics processing engines of the graphics acceleration module 446. As shown, data from the WD 484 can be stored in the registers 445 and used by the MMU 439, interrupt management circuit 447, and / or context management circuit 446. For example, one embodiment of the MMU 439 includes segment / page walk circuitry to access segment / page tables 486 within the OS virtual address space 485. The interrupt management circuit 447 can handle interrupt events 492 received from the graphics acceleration module 446. When performing graphics operations, effective addresses 493 generated by the graphics processing engines 431-432, 43N are translated to real addresses by the MMU 439.

[0103] In one embodiment, a same set of registers 445 is replicated for each graphics processing engine 431-432, 43N and / or graphics acceleration module 446 and can be initialized by the hypervisor or operating system. Each of these replicated registers can be included in an accelerator integration slice 490. Exemplary registers that can be initialized by the hypervisor are shown in Table 1.

[0104] Table 1 - Hypervisor Initialized Registers

[0105]

[0106]

[0107] Exemplary registers that can be initialized by the operating system are shown in Table 2.

[0108] Table 2 - Operating System Initialized Registers

[0109] 1 Process and thread identification 2 Effective address (EA) context save / restore pointer 3 Virtual address (RA) accelerator utilization record pointer 4 Virtual address (RA) storage segment table pointer 5 Authorization mask 6 Work descriptor

[0110] In one embodiment, each WD 484 is specific to a particular graphics acceleration module 446 and / or graphics processing engines 431-432, 43N. The WD includes all information that the graphics processing engines 431-432, 43N need to do their work, or the WD can be a pointer to a memory location where the application has set up a command queue of work to be completed.

[0111] Figure 4E Additional details of one embodiment of a shared model are shown. The embodiment includes a hypervisor real address space 498 in which a list of process elements 499 is stored. The hypervisor real address space 498 is accessible via the hypervisor 496, which virtualizes the graphics acceleration module engine for the operating system 495.

[0112] The shared programming model allows all or a portion of processes from all or a portion of partitions in the system to use the graphics acceleration module 446. There are two programming models in which the graphics acceleration module 446 is shared by multiple processes and partitions: time-sliced sharing and graphics direct sharing.

[0113] In this model, the system hypervisor 496 owns the graphics acceleration module 446 and makes its functionality available to all operating systems 495. For the graphics acceleration module 446 to support virtualization by the hypervisor 496, the graphics acceleration module 446 can adhere to the following requirements: 1) Application job requests must be autonomous (i.e., no need to maintain state between jobs), or the graphics acceleration module 446 must provide a context save and restore mechanism. 2) The graphics acceleration module 446 guarantees that an application job request will complete in a specified amount of time, including any translation faults, or the graphics acceleration module 446 provides the ability to preempt processing of a job. 3) When operating in the direct shared programming model, fairness of the graphics acceleration module 446 among processes must be guaranteed.

[0114] In one embodiment, for a shared model, an application 480 is required to utilize a graphics acceleration module 446 type, a work descriptor (WD), an authorization mask register (AMR) value, and a context save / restore area pointer (CSRP) for an operating system 495 system call. The graphics acceleration module 446 type describes the target acceleration function for the system call. The graphics acceleration module 446 type can be a system specific value. The WD is formatted specifically for the graphics acceleration module 446 and can be in the form of a graphics acceleration module 446 command; a pointer to a user defined structure; a pointer to a command queue; or any other data structure used to describe work to be performed by the graphics acceleration module 446. In one embodiment, the AMR value is the AMR state for the current process. The value passed to the operating system is similar to how an application sets the AMR. If the accelerator integration circuit 436 and graphics acceleration module 446 implementation does not support a user authorization mask override register (UAMOR), the operating system can apply the current UAMOR value to the AMR value before passing the AMR in the hypervisor call. The hypervisor 496 can optionally apply the current authorization mask override register (AMOR) value before placing the AMR in the process element 483. In one embodiment, the CSRP is one of the registers 445 containing the effective address of an area in the application address space 482 for the graphics acceleration module 446 to save and restore context state. This pointer is optional if there is no need to save state between jobs or when a job is preempted. The context save / restore area can be paged system memory.

[0115] Upon receiving the system call, the operating system 495 can verify that the application 480 is registered and authorized to use the graphics acceleration module 446. The operating system 495 then calls the hypervisor 496 with the information shown in Table 3.

[0116] Table 3 - Operating System Call to Hypervisor Parameters

[0117]

[0118]

[0119] Upon receiving the hypervisor call, the hypervisor 496 can verify that the operating system 495 is registered and authorized to use the graphics acceleration module 446. The hypervisor 496 then places the process element 483 in a process element chain for the corresponding graphics acceleration module 446 type. The process element can contain the information shown in Table 4.

[0120] Table 4 - Process Element Information

[0121] 1 Work descriptor (WD) 2 Authorization mask register (AMR) value (possibly masked) 3 Effective address (EA) context save / restore area pointer (CSRP) 4 Process ID (PID) and optional thread ID (TID) 5 Virtual address (VA) accelerator utilization record pointer (AURP) 6 Virtual address of storage segment table pointer (SSTP) 7 Logical interrupt service number (LISN) 8 Interrupt vector table, derived from hypervisor call parameters 9 State register (SR) value 10 Logical partition ID (LPID) 11 Real address (RA) hypervisor accelerator utilization record pointer 12 Storage descriptor register (SDR)

[0122] In one embodiment, the hypervisor initializes the plurality of accelerator integration slices 490 of registers 445.

[0123] As Figure 4F embodiment of the application employs a unified memory that can be addressed via a common virtual memory address space for accessing physical processor memories 401-402 and GPU memories 420-423. In this implementation, operations performed on GPUs 410-413 utilize the same virtual / effective memory address space to access processor memories 401-402 and vice versa, thereby simplifying programmability. In one embodiment, a first portion of the virtual / effective address space is allocated to processor memory 401, a second portion is allocated to second processor memory 402, a third portion is allocated to GPU memory 420, and so on. The entire virtual / effective memory space (sometimes referred to as the effective address space) is thereby distributed across each of processor memories 401-402 and GPU memories 420-423, allowing any processor or GPU to access any physical memory with a virtual address mapped to said memory.

[0124] In one embodiment, bias / coherence management circuitry 494A-494E within one or more of MMUs 439A-439E ensures cache coherence between host processors (e.g., 405) and caches of GPUs 410-413 and performs bias techniques that indicate the physical memory where certain types of data should be stored. Although multiple instances of bias / coherence management circuitry 494A-494E are shown in Figure 4F Although multiple instances of bias / coherence management circuitry 494A-494E are shown in

[0125] One embodiment allows GPU-attached memory 420-423 to be mapped as part of system memory and accessed using shared virtual memory (SVM) technology, but without suffering the typical performance penalties associated with full system cache coherency. The ability for GPU-attached memory 420-423 to be accessed as system memory does not incur the heavy cache coherency overhead, which provides a favorable operating environment for GPU offload. This arrangement allows host processor 405 software to set operands and access computation results without the overhead of traditional I / O DMA data copies. These traditional copies involve driver calls, interrupts, and memory-mapped I / O (MMIO) accesses, which are inefficient relative to simple memory accesses. At the same time, the ability to access GPU-attached memory 420-423 without cache coherency overhead can be critical to the execution time of offloaded computations. For example, in the case of a large amount of streaming write memory traffic, cache coherency overhead can significantly reduce the effective write bandwidth seen by GPUs 410-413. The efficiency of operand setup, the efficiency of result access, and the efficiency of GPU computation all play a significant role in determining the effectiveness of GPU offload.

[0126] In one implementation, the selection between GPU bias and host processor bias is driven by a bias tracker data structure. For example, a bias table can be used, which can be a page-granularity structure (i.e., controlled at the granularity of a memory page) that includes 1 or 2 bits per GPU-attached memory page. The bias table can be implemented within the stolen memory range of one or more GPU-attached memories 420-423, with or without a bias cache in GPUs 410-413 (e.g., to cache frequently / recently used entries of the bias table). Alternatively, the entire bias table can be kept within the GPU.

[0127] In one implementation, the bias table entry associated with each access to GPU-attached memory 420-423 is accessed prior to the actual access to GPU memory, such that the following operations. First, local requests from GPUs 410-413 that find their page in the GPU bias are forwarded directly to the corresponding GPU memory 420-423. Local requests from the GPU that find their page in the host bias are forwarded to processor 405 (e.g., over the high-speed link as described above). In one embodiment, requests from processor 405 that find the requested page in the host processor bias complete the request as a normal memory read. Alternatively, requests for GPU-biased pages can be forwarded to GPUs 410-413. If the page is not currently in use by the GPU, the GPU can convert the page to the host processor bias.

[0128] The bias state of a page can be changed by a software-based mechanism, a hardware-assisted software-based mechanism, or for a limited set of cases, a hardware-only mechanism.

[0129] One mechanism for changing the bias state employs an API call (e.g., OpenCL) that in turn invokes a GPU device driver that in turn sends a message to the GPU (or enqueues a command descriptor) that directs the GPU to change the bias state and, for certain transitions, to perform a cache flush clear operation in the host. The cache flush clear operation is necessary for a transition from the host processor 405 bias to the GPU bias, but not for the reverse transition.

[0130] In one embodiment, cache coherency is maintained by temporarily presenting the GPU-biased pages that are not cacheable by the host processor 405. To access these pages, the processor 405 can request access from the GPU 410, which can or can not grant access depending on the implementation. Thus, to reduce the communication between the processor 405 and the GPU 410, it is advantageous to ensure that the GPU-biased pages are pages that are needed by the GPU but not by the host processor 405, and vice versa.

[0131] Graphics processing pipeline

[0132] Figure 5 A graphics processing pipeline 500 according to an embodiment is shown. In one embodiment, a graphics processor can implement the graphics processing pipeline 500 shown. The graphics processor can be included within a parallel processing subsystem such as the parallel processor 200 of Figure 2, in one embodiment the parallel processor is a GPGPU Figure 1FIG. 5 illustrates an example of a graphics processing pipeline 500 that can be implemented via a variant of the parallel processor(s) 112. As described herein, various parallel processing systems can implement the graphics processing pipeline 500 via one or more instances of a parallel processing unit (e.g., the parallel processing unit 202 of FIG. 2). For example, a shader unit (e.g., the graphics multiprocessor 234 of FIG. 3) can be configured to perform the functions of one or more of the vertex processing unit 504, the tessellation control processing unit 508, the tessellation evaluation processing unit 512, the geometry processing unit 516, and the fragment / pixel processing unit 524. The functionality of the data assembler 502, the primitive assemblers 506, 514, 518, the tessellation unit 510, the rasterizer 522, and the raster operations unit 526 can also be performed by other processing engines and corresponding partition units within a processing cluster (e.g., processing cluster 214 of FIG. 3) as described herein. The graphics processing pipeline 500 can be implemented using a number of dedicated processing units for one or more of the functions. In one embodiment, one or more parts of the graphics processing pipeline 500 can be executed by parallel processing logic within a general purpose processor (e.g., a CPU). In one embodiment, one or more parts of the graphics processing pipeline 500 can access on-chip memory (e.g., parallel processor memory 222 as shown in FIG. 2) via a memory interface 528, which can be an instance of the memory interface 218 of FIG. 2.

[0133] In one embodiment, the data assembler 502 is a processing unit that gathers vertex data for surfaces and primitives. The data assembler 502 then outputs the vertex data, including vertex attributes, to the vertex processing unit 504. The vertex processing unit 504 is a programmable execution unit that executes a vertex shader program to transform and illuminate vertices of a vertex shader program as specified. The vertex processing unit 504 reads data stored in cache, local or system memory that contains vertex data to process and can be programmed to transform the vertex data from an object-based coordinate representation to a world space coordinate space or a normalized device coordinate space.

[0134] A first instance of the primitive assembler 506 receives vertex attributes from the vertex processing unit 504. The primitive assembler 506 reads stored vertex attributes as needed, and constructs graphics primitives for processing by the tessellation control processing unit 508. The graphics primitives include triangles, lines, points, patches, and so forth, as supported by various graphics processing application programming interfaces (APIs).

[0135] The tessellation control processing unit 508 treats the input vertices as control points for the geometry patch. These control points are transformed from an input representation from the patch, such as a basis for the patch, to a representation suitable for surface evaluation by the tessellation evaluation processing unit 512. The tessellation control processing unit 508 can also compute tessellation factors for the edges of the geometry patch. The tessellation factors are applicable to individual edges and quantify a view-dependent level of detail associated with the edge. The tessellation unit 510 is configured to receive the tessellation factors for the edges of the patch and subdivide the patch into a plurality of geometric primitives such as line, triangle, or quadrilateral primitives that are transmitted to the tessellation evaluation processing unit 512. The tessellation evaluation processing unit 512 operates on the parametric coordinates of the subdivided patch to generate surface representations and vertex attributes for each vertex associated with the geometric primitives.

[0136] A second instance of the primitive assembler 514 receives the vertex attributes from the tessellation evaluation processing unit 512, reads the stored vertex attributes as needed, and constructs graphics primitives for processing by the geometry processing unit 516. The geometry processing unit 516 is a programmable execution unit that executes a geometry shader program to transform the graphics primitives received from the primitive assembler 514 as specified by the geometry shader program. In one embodiment, the geometry processing unit 516 is programmed to subdivide the graphics primitives into one or more new graphics primitives and compute parameters for rasterizing the new graphics primitives.

[0137] In some embodiments, the geometry processing unit 516 can add or delete elements in the geometry stream. The geometry processing unit 516 outputs parameters and vertices that specify new graphics primitives to the primitive assembler 518. The primitive assembler 518 receives the parameters and vertices from the geometry processing unit 516 and constructs graphics primitives for processing by the viewport scale, cull, and clip unit 520. The geometry processing unit 516 reads data stored in the parallel processor memory or system memory for processing the geometry data. The viewport scale, cull, and clip unit 520 performs clipping, culling, and viewport scaling and outputs the processed graphics primitives to the rasterizer 522.

[0138] The rasterizer 522 can perform depth culling and other depth-based optimizations. The rasterizer 522 also performs scan conversion on new graphics primitives to generate fragments and outputs these fragments and associated coverage data to the fragment / pixel processing units 524. The fragment / pixel processing units 524 are programmable execution units that are configured to perform fragment shader programs or pixel shader programs. The fragment / pixel processing units 524 transform the fragments or pixels received from the rasterizer 522 as specified by the fragment or pixel shader programs. For example, the fragment / pixel processing units 524 can be programmed to perform operations including, but not limited to, texture mapping, shading, blending, texture correction, and perspective correction to produce shaded fragments or pixels that are output to the raster operations unit 526. The fragment / pixel processing units 524 can read data stored in the parallel processor memory or system memory to use when processing the fragment data. The fragment or pixel shader programs can be configured to shade with samples, pixels, tiles, or other granularity, depending on how the processing units are configured.

[0139] The raster operations unit 526 is a processing unit that performs raster operations including, but not limited to, stencil operations, z-test, blending, and the like and outputs pixel data as processed graphics data to be stored in graphics memory (e.g., the parallel processor memory 222 as in FIG. 2, and / or the system memory 104 as in FIG. 1 to be displayed on one or more display devices 110 or further processed by one of the one or more processors 102 or parallel processor(s) 112. In some embodiments, the raster operations unit 526 is configured to compress z or color data that is written to memory and decompress z or color data that is read from memory. Figure 1

[0140] The foregoing description and accompanying drawings should be considered as illustrative rather than as restrictive on the principles of the application. Those skilled in the art will appreciate that various adaptations and modifications of the embodiments described herein can be configured without departing from the scope and spirit of the application as set forth in the following claims.

[0141] In some examples, different layers of a machine learning process can require different minimum levels of precision to achieve an overall rate with a desired degree of accuracy. Even though internal calculations can not be variable; rather, only the loading and / or storing can utilize variable precision data representations to achieve.

[0142] See Figure 6A ​In one aspect, it can be useful to expose a cast operation embedded in a load / store instruction(s) to support loading from variable integer precision (e.g., 2 bits, 3 bits, 7 bits, etc.). Internal computations can remain at a base precision (e.g., 8 bits or 16 bits). In some examples, this technique can be similar to full layer compression that is specific to deep learning neural network (DNN) data, where 12-bit vector inputs are loaded into 16-bit element vectors.

[0143] Referring to Figure 6B In some examples, the deployment tool can automatically determine a target (e.g., optimal) per-layer precision that maintains a target precision and matches a target hardware capability.

[0144] Referring to Figure 7 In some examples, different data types can be used to represent individual weights in different layers of a multi-layer deep learning neural network (DNN). High-precision floating point data can be used for the first few layers of a neural network and lower-precision floating point data and integers can be used for higher layers of the neural network to create a compressed representation of the network.

[0145] Further, the decision as to whether to perform addition and / or multiplication operations at a higher level of precision or at a lower level of precision can be made dynamically, at least in part depending on whether current operating conditions place greater stress on bandwidth or computational resources.

[0146] Machine learning overview

[0147] Machine learning algorithms are algorithms that can learn based on a set of data. Embodiments of machine learning algorithms can be designed to model high-order abstractions within a data set. For example, image recognition algorithms can be used to determine which of several categories a given input belongs to; regression algorithms can output a numerical value given an input; and pattern recognition algorithms can be used to generate translated text or perform text-to-speech and / or speech recognition.

[0148] One example type of machine learning algorithm is a neural network. There are many types of neural networks; one simple type of neural network is a feedforward network. A feedforward network can be implemented as a directed acyclic graph, with nodes arranged in layers. Typically, a feedforward network topology includes an input layer and an output layer, separated by at least one hidden layer. The hidden layer transforms the input received by the input layer into a representation useful for generating output in the output layer. Network nodes are fully connected to nodes in adjacent layers via edges, but there are no edges between nodes within each layer. Data received at the nodes of the input layer of a feedforward network is propagated (i.e., "fed forward") to the nodes of the output layer via an activation function that computes the state of the nodes of each successive layer in the network based on coefficients ("weights") respectively associated with each of the edges connecting these layers. The output from a neural network algorithm can take various forms, depending on the particular model represented by the algorithm being executed.

[0149] Before a machine learning algorithm can be used to model a particular problem, the algorithm is trained using a training data set. Training a neural network involves selecting a network topology, using a set of training data representing the problem being modeled by the network, and adjusting the weights until the network model performs with minimal error for all instances of the training data set. For example, during a supervised learning training process for a neural network, the output produced by the network in response to input representing an instance in the training data set is compared to the "correct" labeled output for the instance; an error signal representing the difference between the output and the labeled output is computed; and as the error signal is propagated backwards through the layers of the network, the weights associated with the connections are adjusted to minimize the error. When the error for each output generated from an instance of the training data set is minimized, the network is considered to have been "trained."

[0150] The accuracy of a machine learning algorithm can be greatly affected by the quality of the data set used to train the algorithm. The training process can be computationally intensive and can require a large amount of time on a conventional general-purpose processor. Therefore, parallel processing hardware is used to train many types of machine learning algorithms. This is particularly useful for optimizing the training of neural networks, as the computations performed in adjusting the coefficients in a neural network lend themselves naturally to a parallel implementation. In particular, many machine learning algorithms and software applications have been adapted to use parallel processing hardware within a general-purpose graphics processing device.

[0151] Figure 8is a generalized diagram of a machine learning software stack 800. Machine learning applications 802 can be configured to train neural networks using training data sets or implement machine intelligence using trained deep neural networks. Machine learning applications 802 can include training and inference functionality of neural networks and / or specialized software that can be used to train neural networks prior to deployment. Machine learning applications 802 can implement any type of machine intelligence, including but not limited to: image recognition, mapping and localization, autonomous navigation, speech synthesis, medical imaging, or language translation.

[0152] Hardware acceleration for machine learning applications 802 can be implemented via a machine learning framework 804. The machine learning framework 804 can provide a library of machine learning primitives. A machine learning primitive is a basic operation that is commonly performed by machine learning algorithms. Without the machine learning framework 804, a developer of a machine learning algorithm would be required to create and optimize the primary computational logic associated with the machine learning algorithm, then re-optimize the computational logic when new parallel processors are developed. Instead, machine learning applications can be configured to perform the necessary computations using primitives provided by the machine learning framework 804. Exemplary primitives include tensor convolutions, activation functions, and pooling, which are computational operations performed when training a convolutional neural network (CNN). The machine learning framework 804 can also provide primitives for implementing basic linear algebra subprograms, such as matrix and vector operations, that are performed by many machine learning algorithms.

[0153] The machine learning framework 804 can process input data received from the machine learning applications 802 and generate appropriate inputs to a compute framework 806. The compute framework 806 can abstract the underlying instructions provided to a GPGPU driver 808 to enable the machine learning framework 804 to leverage hardware acceleration via GPGPU hardware 810 without the machine learning framework 804 needing to be very familiar with the architecture of the GPGPU hardware 810. Additionally, the compute framework 806 can enable hardware acceleration for the machine learning framework 804 across multiple types and generations of GPGPU hardware 810.

[0154] GPGPU machine learning acceleration

[0155] Figure 9 A highly parallel general purpose graphics processing unit 900 according to an embodiment is shown. In one embodiment, the general purpose processing unit (GPGPU) 900 can be configured to be particularly efficient in processing this type of computational workload associated with training deep neural networks. Additionally, the GPGPU 900 can be directly linked to other instances of GPGPUs for creating a multi-GPU cluster to improve the speed of training particularly deep neural networks.

[0156] GPGPU 900 includes a host interface 902 for enabling connectivity to a host processor. In one embodiment, the host interface 902 is a PCI Express interface. However, the host interface can be a vendor specific communications interface or communications fabric in accordance with an embodiment. GPGPU 900 receives commands from the host processor and uses a global scheduler 904 to distribute execution threads associated with those commands to a set of compute clusters 906A-906H. The compute clusters 906A-906H share a cache memory 908. The cache memory 908 can act as an upper level cache for the cache memories within the compute clusters 906A-906H.

[0157] GPGPU 900 includes memory 914A-914B coupled with the compute clusters 906A-906H via a set of memory controllers 912A-912B. In various embodiments, the memory 914A-914B can include various types of memory devices including dynamic random access memory (DRAM) or graphics random access memory, such as synchronous graphics random access memory (SGRAM), including graphics double data rate (GDDR) memory. In one embodiment, the memory units 224A-224N can also include 3D stacked memory, including but not limited to high bandwidth memory (HBM).

[0158] In one embodiment, each compute cluster 906A-H includes a set of graphics processing units, such as graphics processing units 400 of FIG. 4. Figure 4A The graphics processing units of the compute clusters include multiple types of integer and floating point logic units that can perform compute operations at a range of precisions, including precisions suitable for machine learning computations. For example, and in one embodiment, at least a subset of the floating point units of each of the compute clusters 906A-906H can be configured to perform 16-bit or 32-bit floating point operations, while a different subset of the floating point units can be configured to perform 64-bit floating point operations.

[0159] Multiple instances of GPGPU 900 can be configured to operate as a compute cluster. The communication mechanism used by the compute cluster for synchronization and data exchange varies across embodiments. In one embodiment, multiple instances of GPGPU 900 communicate over host interface 902. In one embodiment, GPGPU 900 includes I / O hub 909 that couples GPGPU 900 to GPU links 910, which enable direct connections to other instances of GPGPU. In one embodiment, GPU links 910 are coupled to a specialized GPU-GPU bridge that enables communication and synchronization between multiple instances of GPGPU 900. In one embodiment, GPU links 910 are coupled with a high-speed interconnect for transmitting and receiving data to other GPGPUs or parallel processors. In one embodiment, multiple instances of GPGPU 900 are located in separate data processing systems and communicate via a network appliance that is accessible via host interface 902. In one embodiment, GPU links 910 can be configured to enable connection to a host processor in addition to or as an alternative to host interface 902.

[0160] While the illustrated configuration of GPGPU 900 can be configured to train a neural network, one embodiment provides an alternative configuration of GPGPU 900 that can be configured for deployment in a high-performance or low-power inferencing use platform. In an inferencing configuration, GPGPU 900 includes fewer compute clusters 906A-H relative to the training configuration. Additionally, the memory technology associated with memory 914A-B can differ between the inferencing and training configurations. In one embodiment, the inferencing configuration of GPGPU 900 can support inferencing-specific instructions. For example, the inferencing configuration can provide support for one or more 8-bit integer dot product instructions, which are commonly used during inferencing operations for a deployed neural network.

[0161] Figure 10 A multi-GPU computing system 1000 according to an embodiment is shown. Multi-GPU computing system 1000 can include a processor 1002 coupled to a plurality of GPGPUs 1006A-D via a host interface switch 1004. In one embodiment, host interface switch 1004 is a PCI Express switch device that couples processor 1002 to a PCI Express bus over which processor 1002 can communicate with the set of GPGPUs 1006A-D. Each of the plurality of GPGPUs 1006A-D can be Figure 9instances of GPGPU 900. GPGPUs 1006A-D can be interconnected via a set of high-speed point-to-point GPU-GPU links 1016. The high-speed GPU-GPU links can be connected to each of GPGPUs 1006A-D via a dedicated GPU link, such as GPU link 910 in Figure 9 P2P GPU links 1016 enable each of GPGPUs 1006A-D to communicate directly with one another without going through the host interface bus to which the processor 1002 is connected. In the case of GPU-GPU traffic, the host interface bus can still be used for system memory access or communication with other instances of the multi-GPU computing system 1000 (e.g., via one or more network devices). While in the illustrated embodiment GPGPUs 1006A-D are connected to processor 1002 via host interface switch 1004, in one embodiment, processor 1002 includes direct support for P2P GPU links 1016 and can connect directly to GPGPUs 1006A-D.

[0162] Machine learning neural network implementation

[0163] The computing architecture provided by the embodiments described herein can be configured to perform these types of parallel processing that are particularly well suited for training and deploying neural networks for machine learning. Neural networks can be generalized as networks of functions having graph relationships. As is well known in the art, there are multiple types of neural network implementations used in machine learning. One exemplary type of neural network is a feedforward network as previously described.

[0164] A second exemplary type of neural network is a convolutional neural network (CNN). CNNs are specialized feed-forward neural networks used to process data having a known, grid-like topology, such as image data. Thus, CNNs are commonly used in computer vision and image recognition applications, but they can also be used for other types of pattern recognition, such as speech and language processing. Nodes in the input layer of a CNN are organized into groups of "filters" (feature detectors inspired by the receptive fields found in the retina), and the output of each group of filters is propagated to nodes in successive layers of the network. The computations used for a CNN include applying a convolution mathematical operation to each filter to produce the output of the filter. Convolution is a specialized mathematical operation performed by two functions to produce a third function that is a modified version of one of the original functions. In convolution network terminology, the first function with respect to the convolution can be referred to as the input, and the second function can be referred to as the convolution kernel. The output can be referred to as a feature map. For example, the input to a convolutional layer can be a multidimensional data array that defines various color components of an input image. The convolution kernel can be a multidimensional array of parameters, where the parameters are adapted through a training process for the neural network.

[0165] A recurrent neural network (RNN) is a type of feed-forward neural network that includes feedback connections between layers. RNNs enable modeling of sequential data by sharing parameter data across different parts of the neural network. The architecture of an RNN includes loops. These loops represent the influence of a current value of a variable on its own value at a future time, as at least a portion of the output data from the RNN is used as feedback for processing subsequent input in the sequence. This feature makes RNNs particularly useful for language processing due to the variable nature in which language data can be composed.

[0166] The diagrams described below present exemplary feed-forward, CNN, and RNN networks, and describe general processes for training and deploying each of those types of networks, respectively. It will be understood that these descriptions are exemplary and non-limiting with respect to any particular embodiment described herein, and that the concepts illustrated can generally be applied to deep neural networks and machine learning techniques in general.

[0167] The exemplary neural networks described above can be used to perform deep learning. Deep learning is machine learning using deep neural networks. In contrast to shallow neural networks that include only a single hidden layer, deep neural networks used in deep learning are artificial neural networks composed of multiple hidden layers. More deeply neural networks are generally more computationally intensive to train. However, the additional hidden layers of the network enable multi-step pattern recognition that results in reduced output error relative to shallow machine learning techniques.

[0168] Deep neural networks used in deep learning typically include a front-end network to perform feature recognition coupled to a back-end network representing a mathematical model that can perform operations (e.g., object classification, speech recognition, etc.) based on feature representations provided to the model. Deep learning enables machine learning to be performed without hand-engineering features for the model. Instead, a deep neural network can learn features based on statistical structure or correlations within input data. The learned features can be provided to a mathematical model that can map the detected features to an output. The mathematical model used by the network is typically specific to the particular task to be performed, and different models will be used to perform different tasks.

[0169] Once a neural network is structured, a learning model can be applied to the network to train the network to perform a particular task. The learning model describes how to adjust weights within the model to reduce the output error of the network. Backpropagation of errors is one commonly used method for training neural networks. An input vector is presented to the network for processing. The output of the network is compared to a desired output using a loss function, and an error value is computed for each neuron in the output layer. These error values are then propagated backwards until each neuron has an associated error value that roughly represents its contribution to the original output. The network can then learn from those errors using an algorithm, such as a stochastic gradient descent algorithm, to update the weights of the neural network.

[0170] Figures 11A-11B An example convolutional neural network is shown. Figure 11A Various layers within a CNN are shown. As shown in Figure 11A An example CNN used to model processing of an image can receive an input 1102 that describes the red, green, and blue (RGB) components of an input image, as shown in FIG. 11. The input 1102 can be processed by a plurality of convolutional layers (e.g., convolutional layer 1104, convolutional layer 1106). Optionally, the output from the plurality of convolutional layers can be processed by a set of fully connected layers 1108. Neurons in the fully connected layers have full connections to all activation functions in the previous layer, as previously described for feedforward networks. The output from the fully connected layers 1108 can be used to generate an output result from the network. The activation functions within the fully connected layers 1108 can be computed using matrix multiplication rather than convolution. Not all CNN implementations use fully connected layers 1108. For example, in some implementations, the convolutional layer 1106 can generate the output of the CNN.

[0171] Convolutional layers are sparsely connected, unlike the traditional neural network configuration found in fully connected layers 1108. Traditional neural network layers are fully connected, such that each output unit interacts with every input unit. However, convolutional layers are sparsely connected, in that the output of the convolution of the receptive field (rather than the respective state value of each node in the receptive field) is input to the nodes of the subsequent layer, as illustrated. The kernel associated with the convolutional layer performs a convolution operation, the output of which is sent to the next layer. The dimensionality reduction performed within the convolutional layer is one aspect that enables the CNN to scale to handle large images.

[0172] Figure 11B An exemplary computation stage within a convolutional layer of a CNN is illustrated. The input 1112 to the convolutional layer of a CNN can be processed in three stages of the convolutional layer 1114. The three stages can include a convolution stage 1116, a detector stage 1118, and a pooling stage 1120. The convolutional layer 1114 can then output data to a successive convolutional layer. The last convolutional layer of the network can generate output feature map data or provide input to a fully connected layer, for example, to generate classification values to the input to the CNN.

[0173] Several convolutions are performed in parallel in the convolution stage 1116 to produce a set of linear activation functions. The convolution stage 1116 can include an affine transformation, which is any transformation that can be specified as a linear transformation plus a translation. Affine transformations include rotation, translation, scaling, and combinations of these transformations. The convolution stage computes the output (e.g., a neuron) of a function connected to a particular region in the input, which can be determined as a local region associated with the neuron. The neuron computes a dot product between the weights of the neuron and the region in the local input to which the neuron is connected. The output from the convolution stage 1116 defines a set of linear activation functions that are processed by successive stages of the convolutional layer 1114.

[0174] The linear activation functions can be processed by the detector stage 1118. In the detector stage 1118, each linear activation function is processed by a non-linear activation function. The non-linear activation function increases the non-linear properties of the overall network without affecting the receptive field of the convolutional layer. Several types of non-linear activation functions can be used. One particular type is a rectified linear unit (ReLU), which uses an activation function defined as f(x) = max(0, x) such that the activation function is thresholded to zero.

[0175] The pooling stage 1120 uses a pooling function that replaces the output of the convolutional layer 1106 with summary statistics of nearby outputs. Pooling functions can be used to introduce translational invariance into a neural network, such that slight translations to the input do not change the pooling output. Local translational invariance can be useful in cases where the presence of a feature in the input data is more important than the precise location of the feature. Various types of pooling functions can be used during the pooling stage 1120, including max pooling, average pooling, and L2 norm pooling. Additionally, some CNN implementations do not include a pooling stage. Instead, such implementations substitute an additional convolutional stage with an increased stride relative to the previous convolutional stage.

[0176] The output from the convolutional layer 1114 can then be processed by a next layer 1122. The next layer 1122 can be an additional convolutional layer or one of the fully connected layers 1108. For example, Figure 11A The first convolutional layer 1104 of the convolutional neural network 1100 can output to a second convolutional layer 1106, which can output to a first layer of the fully connected layers 1108.

[0177] Figure 12 An example recurrent neural network 1200 is shown. In a recurrent neural network (RNN), the previous state of the network influences the output of the current state of the network. A wide variety of functions can be used in a wide variety of ways to establish an RNN. The use of RNNs often revolves around using a mathematical model to make predictions of the future based on a sequence of previous inputs. For example, an RNN can be used to perform statistical language modeling to predict an upcoming word given a sequence of previous words. The RNN 1200 shown can be described as having an input layer 1202 that receives an input vector, a hidden layer 1204 that implements a recurrent function, a feedback mechanism 1205 that implements a'memory' of previous states, and an output layer 1206 that outputs a result. The RNN 1200 operates based on time steps. The state of the RNN at a given time step is influenced by previous time steps via the feedback mechanism 1205. The state of the hidden layer 1204 is defined for a given time step by the previous state and the input at the current time step. An initial input (x1) at a first time step can be processed by the hidden layer 1204. A second input (x2) can be processed by the hidden layer 1204 using state information determined during processing of the initial input (x1). The given state can be computed as s t = f(Ux t + Ws t-1), where U and W are parameter matrices. The function f is typically non-linear, such as the hyperbolic tangent function (Tanh) or a variant of the rectified function f(x) = max(0, x). However, the particular mathematical function used in the hidden layer 1204 can vary depending on the particular implementation details of the RNN 1200.

[0178] In addition to the basic CNN and RNN networks described, variations of those networks can be implemented. One example RNN variant is a long short-term memory (LSTM) RNN. LSTM RNNs are capable of learning long-term dependencies that can be necessary for processing longer language sequences. A variant of a CNN is a convolutional deep belief network, which has a structure similar to a CNN and is trained in a manner similar to a deep belief network. A deep belief network (DBN) is a generative neural network composed of multiple layers of stochastic (random) variables. A DBN can be trained layer by layer using greedy unsupervised learning. The learned weights of a DBN can then be used to provide a pre-trained neural network by determining a set of optimal initial weights for a neural network.

[0179] Figure 13 Training and deployment of deep neural networks is shown. Once a given network has been structured for a task, the neural network is trained using a training dataset 1302. Various training frameworks have been developed for implementing hardware acceleration of the training process. For example, Figure 8 The machine learning framework 804 of FIG. 8 can be configured as a training framework 1304. The training framework 604 can hook into an untrained neural network 1306 and enable the untrained neural network to be trained using the parallel processing resources described herein to generate a trained neural network 1308.

[0180] To begin the training process, the initial weights can be selected randomly or by pre-training using a deep belief network. The training loop is then performed in a supervised or unsupervised manner.

[0181] Supervised learning is a method of learning in which training is performed as an arbitration operation, such as when the training dataset 1302 includes inputs paired with expected outputs for the inputs, or in cases where the training dataset includes inputs with known outputs and the output of the neural network is manually graded. The network processes the inputs, and the resulting outputs are compared to a set of expected or desired outputs. Errors are then backpropagated through the system. The training framework 1304 can make adjustments to adjust the weights that control the untrained neural network 1306. The training framework 1304 can provide tools for monitoring how well the untrained neural network 1306 is converging to a model that is suitable for generating correct answers based on known input data. The training process occurs repeatedly as the weights of the network are adjusted to improve the output generated by the neural network. The training process can continue until the neural network reaches a statistically expected level of accuracy associated with a trained neural network 1308. The trained neural network 1308 can then be deployed to implement any number of machine learning operations.

[0182] Unsupervised learning is a method of learning in which the network attempts to train itself using unlabeled data. Thus, for unsupervised learning, the training dataset 1302 will include input data without any associated output data. The untrained neural network 1306 can learn groupings within the unlabeled inputs, and can determine how individual inputs relate to the overall dataset. Unsupervised training can be used to generate self-organizing maps, which are a type of trained neural network 1307 that can perform operations useful in data reduction. Unsupervised training can also be used to perform anomaly detection, which allows for the identification of data points in an input dataset that deviate from the normal patterns of the data.

[0183] Variations of supervised and unsupervised training can also be employed. Semi-supervised learning is a technique in which the training dataset 1302 includes a mix of labeled and unlabeled data of the same distribution. Incremental learning is a variation of supervised learning in which input data is used continuously for further training of the model. Incremental learning enables the trained neural network 1308 to adapt to new data 1312 without forgetting the knowledge rooted within the network during initial training.

[0184] Regardless of whether supervised or unsupervised, the training process for particularly deep neural networks can be too computationally intensive for a single computing node. Rather than using a single computing node, a distributed network of computing nodes can be used to speed up the training process.

[0185] Figure 14is a block diagram illustrating distributed learning. Distributed learning is training a model that uses multiple distributed computing nodes to perform supervised or unsupervised training of a neural network. The distributed computing nodes can each include one or more host processors and one or more of general purpose processing nodes, such as general purpose central processing units 902 in FIG. 1. Figure 9 As illustrated, distributed learning can perform model parallelization 1402, data parallelization 1404, or a combination of model and data parallelization 1404.

[0186] In model parallelization 1402, different computing nodes in a distributed system can perform training computations for different parts of a single network. For example, each layer of a neural network can be trained by different processing nodes of a distributed system. Benefits of model parallelization include the ability to scale to particularly large models. Splitting computations associated with different layers of a neural network enables training of super large neural networks where the weights of all layers would not fit into the memory of a single computing node. In some instances, model parallelization can be particularly useful in performing unsupervised training of large neural networks.

[0187] In data parallelization 1404, different nodes of a distributed network have a complete instance of a model, and each node receives a different portion of data. Results from different nodes are then combined. While different approaches for data parallelization are possible, data parallel training approaches all require a technique to combine results and synchronize model parameters between each node. Exemplary approaches for combining data include parameter averaging and update-based data parallelization. Parameter averaging trains each node on a subset of training data and sets global parameters (e.g., weights, biases) to the average of the parameters from each node. Parameter averaging uses a central parameter server that holds parameter data. Update-based data parallelization is similar to parameter averaging except that updates to the model are passed instead of passing parameters from nodes to a parameter server. Additionally, update-based data parallelization can be performed in a decentralized manner where updates are compressed and passed between nodes.

[0188] For example, combined model and data parallelization 1406 can be implemented in a distributed system where each computing node includes multiple GPUs. Each node can have a complete instance of a model, with individual GPUs within each node used to train different parts of the model.

[0189] Distributed training has increased overhead relative to training on a single machine. However, the parallel processors and GPGPUs described herein can each implement techniques for reducing the overhead of distributed training, including techniques for implementing high-bandwidth GPU-GPU data transfers and accelerated remote data synchronization.

[0190] Exemplary machine learning applications

[0191] Machine learning can be applied to solve a number of technical problems, including but not limited to computer vision, autonomous driving and navigation, speech recognition, and language processing. Computer vision has traditionally been one of the most active areas of research for machine learning applications. Applications of computer vision range from replicating human vision capabilities (e.g., recognizing human faces) to creating new classes of vision capabilities. For example, a computer vision application can be configured to recognize sound waves from vibrations induced in objects visible in a video. Parallel processor-accelerated machine learning enables training of computer vision applications using training data sets significantly larger than previously feasible, and enables deployment of inference- use systems using low-power parallel processors.

[0192] Parallel processor-accelerated machine learning has applications in autonomous driving, including lane and road sign recognition, obstacle avoidance, navigation, and driving control. Accelerated machine learning techniques can be used to train a driving model based on a data set that defines an appropriate response to a particular training input. The parallel processors described herein can enable fast training of increasingly complex neural networks for autonomous driving solutions, and enable deployment of low-power inference-use processors in mobile platforms suitable for integration into autonomous vehicles.

[0193] Parallel processor-accelerated deep neural networks have implemented machine learning methods for automatic speech recognition (ASR). ASR includes creating a function that computes the most likely sequence of words given an input acoustic sequence. Accelerated machine learning using deep neural networks has implemented in place of Hidden Markov Models (HMMs) and Gaussian Mixture Models (GMMs) previously used for ASR.

[0194] Parallel processor-accelerated machine learning can also be used to accelerate natural language processing. Automated learning programs can use statistical inference algorithms to produce models that are robust to erroneous or unfamiliar input. An exemplary natural language processor application includes automatic machine translation between human languages.

[0195] Parallel processing platforms for machine learning can be divided into training platforms and deployment platforms. Training platforms are typically highly parallel and include optimizations for accelerating multi-GPU single-node training and multi-node multi-GPU training. Exemplary parallel processors suitable for training include Figure 9highly parallel general-purpose graphics processing units 900 and Figure 10 multi-GPU computing systems 1000. In contrast, deployed machine learning platforms typically include low-power parallel processors suitable for use in products such as cameras, autonomous robots, and autonomous vehicles.

[0196] Figure 15 An exemplary inference chip on system (SOC) 1500 suitable for performing inference using trained models is shown. The SOC 1500 can integrate multiple processing components, including a media processor 1502, a vision processor 1504, a GPGPU 1506, and a multi-core processor 1508. The SOC 1500 can additionally include on-chip memory 1505, which can implement a shared on-chip data pool accessible by each of the processing components. The processing components can be optimized for low-power operation for enabling deployment to a wide variety of machine learning platforms, including autonomous vehicles and autonomous robots. For example, one implementation of the SOC 1500 can be used as part of a main control system for an autonomous vehicle. In the case where the SOC 1500 is configured for use in an autonomous vehicle, the SOC is designed and configured to comply with relevant functional safety standards for the deployment jurisdiction.

[0197] During operation, the media processor 1502 and the vision processor 1504 can work in concert to accelerate computer vision operations. The media processor 1502 can enable low-latency decoding of multiple high-resolution (e.g., 4K, 8K) video streams. The decoded video streams can be written to a buffer in the on-chip memory 1505. The vision processor 1504 can then parse the decoded video and perform preliminary processing operations on frames of the decoded video in preparation for processing the frames using trained image recognition models. For example, the vision processor 1504 can accelerate convolution operations for CNNs used to perform image recognition on high-resolution video data, while back-end model computations are performed by the GPGPU 1506.

[0198] The multi-core processor 1508 can include control logic to facilitate ordering and synchronization of data transfers and shared memory operations performed by the media processor 1502 and the vision processor 1504. The multi-core processor 1508 can also act as an application processor for executing software applications that can use the inference computing capabilities of the GPGPU 1506. For example, at least a portion of navigation and driving logic can be implemented in software executing on the multi-core processor 1508. Such software can directly issue computing workloads to the GPGPU 1506, or can issue computing workloads to the multi-core processor 1508, which can offload at least a portion of those operations to the GPGPU 1506.

[0199] GPGPU 1506 can include compute clusters, such as a low power configuration of compute clusters 906A-906H within highly parallel general purpose graphics processing unit 700. The compute clusters within GPGPU 1506 can support instructions that are explicitly optimized for performing inferencing computations on trained neural networks. For example, GPGPU 1506 can support instructions for performing low-precision computations, such as 8-bit and 4-bit integer vector operations.

[0200] Additional exemplary graphics processing systems

[0201] Details of the embodiments described above can be included within graphics processing systems and devices described below. Figures 16-29 The graphics processing systems and devices of FIG. 1 illustrate alternative systems and graphics processing hardware that can implement any and all of the techniques described above.

[0202] Additional exemplary graphics processing system overview

[0203] Figure 16 is a block diagram of a processing system 1600 according to an embodiment. In embodiments, system 1600 includes one or more processors 1602 and one or more graphics processors 1608, and can be a single processor desktop system, a multiprocessor workstation system, or a server system that includes a large number of processors 1602 or processor cores 1607. In one embodiment, system 1600 is a processing platform incorporated within a server or a mobile device such as a mobile telephone, a handheld computer, or a set-top box.

[0204] Embodiments of system 1600 can include or incorporate a server-based gaming platform, a game console, including a games and media console, a mobile gaming console, a handheld game console, or an online game console. In some embodiments, system 1600 is a mobile telephone, a smart phone, a tablet computing device, or a mobile internet device. Data processing system 1600 can also include, be coupled to, or integrated within a wearable device such as a smart watch wearable device, smart glasses device, augmented reality device, or virtual reality device. In some embodiments, data processing system 1600 is a television or set-top box device having one or more processors 1602 and a graphical interface generated by one or more graphics processors 1608.

[0205] In some embodiments, one or more processors 1602 each include one or more processor cores 1607 to process instructions which, when executed, implement the operations of system and user software. In some embodiments, each of the one or more processor cores 1607 is configured to process a specific instruction set 1609. In some embodiments, instruction set 1609 can facilitate complex instruction set computing (CISC), reduced instruction set computing (RISC), or computing via a very long instruction word (VLIW). Multiple processor cores 1607 can each process a different instruction set 1609, which can include instructions to facilitate the emulation of other instruction sets. Processor core 1607 can also include other processing devices, such as a digital signal processor (DSP).

[0206] In some embodiments, processor 1602 includes cache memory 1604. Depending upon the configuration, processor 1602 can have a single internal cache or multiple levels of internal cache. In some embodiments, cache memory is shared among multiple components of processor 1602. In some embodiments, processor 1602 also uses an external cache (e.g., a 3- level (L3) cache, or a last level cache (LLC)), which can be shared among processor cores 1607 using known cache coherence techniques. Additionally, register file 1606 is included in processor 1602, which can include different types of registers for storing different kinds of data (e.g., integer registers, floating point registers, status registers, and instruction pointer registers). Some registers can be general- purpose registers, while other registers can be specific to processor 1602 design.

[0207] In some embodiments, processor 1602 is coupled to a processor bus 1610 that is used to transmit communication signals between processor 1602 and other components in system 1600, such as address, data, or control signals. In one embodiment, system 1600 uses an exemplary 'hub' system architecture, including a memory controller hub 1616 and an input / output (I / O) controller hub 1630. The memory controller hub 1616 facilitates communication between memory devices and other components in system 1600, while the I / O controller hub 1630 provides connections between the I / O devices and the local I / O bus. In one embodiment, the logic of memory controller hub 1616 is integrated within processor.

[0208] The memory device 1620 can be a dynamic random access memory (DRAM) device, a static random access memory (SRAM) device, a flash memory device, a phase- change memory device, or some other memory device having suitable performance to serve as a processing memory. In one embodiment, the memory device 1620 can operate as system memory for the system 1600, to store data 1622 and instructions 1621 for use when the one or more processors 1602 execute an application or process. The memory controller hub 1616 also couples with an optional external graphics processor 1612, which can communicate with the one or more graphics processors 1608 within the processors 1602 to perform graphics and media operations.

[0209] In some embodiments, the ICH 1630 enables peripherals to connect to the memory device 1620 and the processors 1602 via a high-speed I / O bus. The I / O peripherals include, but are not limited to, an audio controller 1646, a firmware interface 1628, a wireless transceiver 1626 (e.g., Wi-Fi, Bluetooth), a data storage device 1624 (e.g., hard disk drive, flash memory, etc.), and a

[0210] Figure 17 is a block diagram of an embodiment of a processor 1700 having one or more processor cores 1702A to 1702N, an integrated memory controller 1714, and an integrated graphics processor 1708. Figure 17Those elements of the FIGS. having the same reference number (or name) as the elements of any other figure herein can operate or function in any manner similar to that described elsewhere herein, but are not limited to such. The processor 1700 can include additional cores up to and including the additional core 1702N represented by the dashed box. Each of the processor cores 1702A to 1702N includes one or more internal cache units 1704A to 1704N. In some embodiments, each processor core can also access one or more shared cache units 1706.

[0211] The internal cache units 1704A to 1704N and shared cache units 1706 represent a cache memory hierarchy internal to the processor 1700. The cache memory hierarchy can include at least one level of instruction and data caches within each processor core and one or more levels of shared mid-level caches, such as level 2 (L2), level 3 (L3), level 4 (L4), or other levels of caches, with the highest level cache being classified as an LLC before external memory. In some embodiments, cache coherency logic maintains coherency among the cache units 1706 and 1704A to 1704N.

[0212] In some embodiments, the processor 1700 can also include a set of one or more bus controller units 1716 and a system agent core 1710. The one or more bus controller units 1716 manage a set of peripheral buses, such as one or more of the Peripheral Component Interconnect buses (e.g., PCI, PCI Express). The system agent core 1710 provides management functionality for the various processor components. In some embodiments, the system agent core 1710 includes one or more integrated memory controllers 1714 to manage access to various external memory devices (not shown).

[0213] In some embodiments, one or more of the processor cores 1702A to 1702N include support for simultaneous multi-threading. In such embodiments, the system agent core 1710 includes components to coordinate and operate the processor cores 1702A to 1702N during multi-threaded processing. Additionally, the system agent core 1710 can also include a power control unit (PCU), including logic and components to adjust the power state of the processor cores 1702A to 1702N, as well as the graphics processor 1708.

[0214] In some embodiments, in addition, processor 1700 includes a graphics processor 1708 for performing graphics processing operations. In some embodiments, graphics processor 1708 is coupled to shared cache unit 1706 set and system agent core 1710, which includes one or more integrated memory controllers 1714. In some embodiments, a display controller 1711 is coupled with graphics processor 1708 in order to drive graphics processor output to one or more coupled displays. In some embodiments, display controller 1711 can be a separate module coupled with the graphics processor via at least one interconnect, or can be integrated within graphics processor 1708 or system agent core 1710.

[0215] In some embodiments, ring-based interconnect unit 1712 is used to couple internal components of processor 1700. However, alternative interconnect units can be used, such as point-to-point interconnects, switched interconnects, or other technologies, including those well known in the art. In some embodiments, graphics processor 1708 is coupled with ring interconnect 1712 via I / O link 1713.

[0216] Exemplary I / O link 1713 represents at least one of a variety of I / O interconnects in a plurality of I / O interconnects, including a package I / O interconnect facilitating communication between various processor components and a high performance embedded memory module 1718, such as an eDRAM module. In some embodiments, each of processor cores 1702A-1702N and graphics processor 1708 use embedded memory module 1718 as a shared last level cache.

[0217] In some embodiments, processor cores 1702A-1702N are homogeneous cores executing the same instruction set architecture. In another embodiment, processor cores 1702A-1702N are heterogeneous in terms of instruction set architecture (ISA), where one or more of processor cores 1702A-1702N execute a first instruction set and at least one of the other cores executes a subset of the first instruction set or a different instruction set. In one embodiment, processor cores 1702A-1702N are homogeneous in terms of microarchitecture where one or more cores have a relatively high power consumption and one or more power cores have a lower power consumption. In addition, processor 1700 can be implemented on one or more chips or as a SoC integrated circuit with the illustrated components, among other components.

[0218] Figure 18is a block diagram of a graphics processor 1800 that can be a discrete graphics processing unit, or can be graphics processor integrated with a multiple core processor. In some embodiments, the graphics processor communicates with the processor through a bus or through a point-to-point interface. In some embodiments, graphics processor 1800 includes a memory interface 1814 to access a memory. The memory interface 1814 can be to local memory, one or more internal caches, one or more shared external caches, and / or to system memory.

[0219] In some embodiments, graphics processor 1800 also includes a display controller 1802 to drive display output data to a display device 1820. Display controller 1802 includes hardware for one or more overlay planes for compositing and for plane blending of multiple video or user interface elements. In some embodiments, graphics processor 1800 includes a video codec engine 1806 to

[0220] In some embodiments, graphics processor 1800 includes a block image transfer (BLIT) engine 1804 to perform two-dimensional (2D) rasterizer operations including, for example, bit-boundary block transfers. However, in one embodiment, 2D graphics operations are performed using one or more components of graphics-processing engine (GPE) 1810. In some embodiments, GPE 1810 is a compute engine to perform graphics and media operations, including three-dimensional (3D) graphics operations and media operations.

[0221] In some embodiments, GPE 1810 includes a 3D pipeline 1812 for processing 3D graphics operations. 3D pipeline 1812 includes programmable and fixed function elements that perform various tasks on geometry shapes or objects represented in data streams. In some embodiments, the 3D pipeline 1812 includes stages for a 3D pipeline 1812, such as an application program interface (API) fetch stage, a program call interface (PUI) fetch stage, a vertex fetch stage, a vertex shader stage, a geometry fetch stage, a geometry shader stage, a primitive assembly stage, a rasterizer stage, a fragment fetch stage, a fragment shader stage, a sample coverage stage, and an output merger stage. However, not all graphics pipeline stages are necessary at all times. In some embodiments, 3D pipeline 1812 also includes fixed function and programmable logic to perform media operations, such as video post-processing and image enhancements.

[0222] In some embodiments, media pipeline 1816 includes fixed function or programmable logic for accelerating media operations, such as video decode acceleration, video de-interlacing, and video encode acceleration in place of, or on behalf of, video codec engine 1806. In some embodiments, media pipeline 1816 additionally includes thread spawning units to spawn threads for execution on 3D / Media sub-system 1815. The spawned threads perform computations for the media operations in place of, or on behalf of the one or more graphics execution units included in 3D / Media sub-system 1815.

[0223] In some embodiments, 3D / Media sub-system 1815 includes logic to execute threads generated by 3D pipeline 1812 and media pipeline 1816. In one embodiment, the pipeline sends thread execution requests to 3D / Media sub-system 1815, which includes thread dispatch logic to arbitrate the requests and dispatch the requests to available thread execution resources. The execution resources include an array of graphics execution units to process 3D and media threads. In some embodiments, 3D / Media sub-system 1815 includes one or more internal caches to cache it thread instructions and data. In some embodiments, the sub-system also includes shared memory (including registers and

[0224] Graphics processing engine

[0225] Figure 19 FIG. 19 is a block diagram of a graphics processing engine 1910 of a graphics processor of some embodiments in accordance with some embodiments. In one embodiment, the graphics processing engine (GPE) 1910 is a version of the GPE 1810 shown in FIG. 18. Figure 18 FIG. 19 is a block diagram of a graphics processing engine 1910 of a graphics processor of some embodiments in accordance with some embodiments. In one embodiment, the graphics processing engine (GPE) 1910 is a version of the GPE 1810 shown in FIG. 18. Figure 19 Those elements of FIG. 19 having the same reference numbers (or names) as the elements of any other figure herein can operate or function in any manner similar to that described elsewhere herein, but are not limited to such. For example, the 3D pipeline 1812 and media pipeline 1816 of FIG. 18 are illustrated as separate from, and Figure 18

[0226] ​In some embodiments, GPE 1910 is coupled with or includes a command streamer 1903 that provides a command stream to 3D pipeline 1812 and / or media pipeline 1816. In some embodiments, command streamer 1903 is a separate component coupled with GPE 1910, while in other embodiments, command streamer 1903 is integrated within the processor or GPE 1910. In some embodiments, command streamer 1903 receives commands, which can be stored in memory, such as a buffer, and provides these commands to the 3D pipeline 1812 and / or media pipeline 1816. In one embodiment, the commands are fetched from the ring buffer that stores the batch of commands for the 3D and media pipelines 1812 and 1816. In another embodiment, the commands can be fetched from a batch command buffer that stores batches of multiple commands. The commands for the 3D pipeline 1812 can include instructions for handling graphics data such as vertex and geometry data, and / or the commands for the media pipeline 1816 can include instructions for handling media data such as image and / or video data. The 3D pipeline 1812 and media pipeline 1816 execute the commands by processing the graphics and media data, respectively, and by dispatching execution threads to the execution unit array 1914 for processing.

[0227] In various embodiments, 3D pipeline 1812 can execute one or more shader programs, such as a vertex shader, a geometry shader, a pixel (fragment) shader, a compute shader, or other shader programs, by processing the instructions and dispatching execution threads to the graphics core array 1914. The graphics core array 1914 provides unified execution resources for

[0228] In some embodiments, graphics core array 1914 also includes execution logic to perform media functions such as video and / or image processing. In one embodiment, the execution units are Figure 16 programmed to perform parallel general-purpose computing operations, in addition to graphics processing operations. The general-purpose logic can be used to perform any of a variety of other processing, including but not limited to storage Figure 17 operations, or in conjunction with the general-purpose logic within the processor core(s) 1607 of

[0229] Output data generated by threads executing on graphics core array 1914 can be output to memory in a unified return buffer (URB) 1918. URB 1918 can store data from multiple threads. In some embodiments, URB 1918 can be used to transmit data between different threads executing on graphics core array 1914. In some embodiments, URB 1918 can additionally be used for synchronization between threads on graphics core array and fixed function logic within shared function logic 1920.

[0230] In some embodiments, graphics core array 1914 is scalable, such that the array includes a variable number of graphics cores each having a variable number of execution units based on target performance and power levels of the GPE 1910. In one embodiment, the execution resources are dynamically scalable such that execution resources can be enabled or disabled as needed.

[0231] Graphics core array 1914 is coupled with shared function logic 1920, which includes resources shared between graphics cores in the graphics core array. Shared functions within shared function logic 1920 are hardware logic units that provide specialized supplemental functionality to graphics core array 1914. In various embodiments, shared function logic 1920 includes, without limitation, a sampler 1921, math 1922, and inter-thread communication (ITC) 1923 logic. Additionally, some embodiments implement one or more caches 1925 within shared function logic 1920. Shared functions are implemented in cases where demand for a given specialized function is insufficient to include multiple instances of the specialized function within graphics core array 1914. Instead, a single instance of the specialized function is implemented as a standalone entity within shared function logic 1920 and shared between execution resources within graphics core array 1914. The exact set of functions shared between graphics core array 1914 and included within graphics core array 1914 varies between embodiments.

[0232] Figure 20 is a block diagram of another embodiment of a graphics processor 2000. Figure 20 Those elements of having the same reference number (or name) in the figures herein as elements in any other figure herein can operate or function in any manner similar to that described elsewhere herein, but are not limited to such.

[0233] In some embodiments, graphics processor 2000 includes a ring interconnect 2002, a front-end 2004, a media engine 2037, and graphics cores 2080A-2080N. In some embodiments, ring interconnect 2002 couples the graphics processor to other processing units including other graphics processors or one or more general-purpose processor cores. In some embodiments, the graphics processor is one of a plurality of processors integrated within a multi-core processing system.

[0234] In some embodiments, graphics processor 2000 receives batches of commands via ring interconnect 2002. An incoming batch of commands is interpreted by a command streamer 2003 in pipeline front-end 2004. In some embodiments, graphics processor 2000 includes scalable execution logic to perform 3D geometry processing and media processing via the graphics core(s) 2080A-2080N. For 3D geometry processing commands, command streamer 2003 supplies commands to geometry pipeline 2036. For at least some media processing commands, command streamer 2003 supplies commands to video front end 2034, which couples with a media engine 2037. In some embodiments, media engine 2037 includes a video quality engine (VQE) 2030 for video and image post-processing, and a multi-format encode / decode (MFX) 2033 engine to

[0235] In some embodiments, graphics processor 2000 includes a number of graphics core(s) 2080A-2080N (sometimes referred to as core slices) each having a number of sub-cores 2050A-550N, 2060A-2060N (sometimes referred to as core sub-slices). In some embodiments, graphics processor 2000 can have any number of graphics core(s) 2080A-2080N. In some embodiments, graphics processor 2000 includes a graphics core 2080A having at least a first sub-core 2050A and a second sub-core 2060A. In other embodiments, graphics processor is a low power processor with a single sub-core (e.g., 2050A). In some embodiments, graphics processor 2000 includes a number of graphics core(s) 2080A-2080N each including a set of first sub-cores 2050A-2050N and a set of second sub-cores 2060A-2060N. Each sub-core in the set of first sub-cores 2050A-2050N includes at least a first set of execution units 2052A-2052N and a media / texture sampler 2054A-2054N. Each sub-core in the set of second sub-cores 2060A-2060N includes at least a second set of execution units 2062A-2062N and a sampler 2064A-2064N. In some embodiments, each sub-core 2050A-2050N, 2060A-2060N shares a set of shared resources 2070A-2070N. In some embodiments, the shared resources include shared cache memory and pixel operation logic. Other shared resources can also be included in graphics processor embodiments.

[0236] Execution unit

[0237] Figure 21 Thread execution logic 2100 is shown including an array of processing elements employed in some embodiments of GPEs. Figure 21 Those elements of 2036 having the same reference number (or name) as an element herein, but having different letter suffixes, represent similar objects that can be different instances of the same object type. For instance, elements having the same reference number as an element having a "A" suffix, but having a "B" suffix, represent a similar object. In some embodiments, the "similar" object can be an identical object (e.g., an identical hardware element). In other embodiments, the "similar" object can be a similar but different object (e.g., a different hardware element providing a similar functionality). In some embodiments, elements having the same reference number as an element having a "A" suffix, but having a "B" suffix, represent an identical object.

[0238] In some embodiments, thread execution logic 2100 includes a shader processor 2102, a thread dispatcher 2104, an instruction cache 2106, a scalable execution unit array 2108 including a number of execution units 2108A-2108N, a sampler 2110, a data cache 2112, and a data port 2114. In one embodiment, the scalable execution unit array can dynamically scale by enabling or disabling one or more execution units (e.g., any of execution units 2108A, 2108B, 2108C, 2108D, up to 2108N-1, and 2108N) based on the computational requirements of the applications running on the GPE.

[0239] In some embodiments, execution units 2108A-2108N are primarily used for executing shader programs. Shader processor 2102 can process various shader programs and dispatch execution threads associated with the shader programs to execution units 2108A-2108N via thread dispatcher 2104. In one embodiment, thread dispatcher 2104 includes logic to Figure 20 Figure 21 ​) for processing. In some embodiments, thread dispatcher 2104 can also handle run-time thread generation requests from executing shader programs.

[0240] In some embodiments, execution units 2108A-2108N support single program multiple instruction (SPMD) closely aligned with SIMD and parallel (SIMT) execution models. In some embodiments, execution units 2108A-2108N include logic to handle SIMD9 16- and SIMD8 operations. In some embodiments, execution units 2108A-2108N also support integer and floating-point data types, including:

[0241] Each of execution units 2108A-2108N is capable of multi-issue single instruction multiple data (SIMD) and multi-threaded operation. The number of threads executed on the processor each clock is dependent on the thread dispatch rate and the SIMD width of the execution units. The thread dispatch rate is controlled by the operating system (OS) process scheduler and is therefore application or OS controlled. The SIMD width of the execution units is application dependent. Single instruction multiple thread (SIMT) operation is used on a thread by thread basis where a different instruction is executed on a different thread. Hence, one instruction can be executed on a plurality of threads or a plurality of instructions can be executed on a single thread. Threads executing on the same execution lane cannot access data in each other's thread local storage (TLS). Threads that execute on different execution lanes can access data in each other's TLS but there can be latency and inter-core cache coherency constraints to the memory writes observed to the thread's own TLS.

[0242] The instruction set of the execution unit includes SIMD instructions. Various data elements can be stored as a packed data type within the registers and the execution unit will process the data elements according to their data type. For example, when processing SIMD single-precision floating-point instructions, the execution unit processes a 32-bit floating-point data element in each column of the register on a column-by-column basis. In another example, when processing SIMD single-precision vector instructions, the execution unit processes 4 single-precision floating-point data elements or 8 packed data elements in parallel. Other numbers of data elements may

[0243] One or more internal instruction caches (e.g., 2106) are included in the thread execution logic 2100 to cache thread instructions for the execution units. In some embodiments, one or more data caches (e.g., 2112) are included to cache thread data during thread execution. In some embodiments, a sampler 2110 is included to provide texture sampling for 3D operations and media sampling for media operations. In some embodiments, the sampler 2110 includes specialized texture or media sampling functionality to process texture or media data during sampling before the data is provided to the execution units.

[0244] During execution, the graphics and media pipeline sends thread initiation requests to the thread execution logic 2100 via the thread generation and dispatch logic. Once a set of geometric objects has been processed and rasterized into pixel data, pixel processor logic (e.g., pixel shader logic, fragment shader logic, etc.) within the shader processor 2102 is invoked to further calculate output values and cause the results to be written to an output surface (e.g., color buffer, depth buffer, stencil buffer, etc.). In some embodiments, pixel or fragment shader programs calculate values of vertex attributes that are interpolated across rasterized objects. In some embodiments, the pixel processor logic within the shader processor 2102 then executes an application programming interface (API) supplied pixel or fragment shader program. To execute the shader program, the shader processor 2102 dispatches threads to the execution units (e.g., 2108A) via the thread dispatcher 2104. In some embodiments, the pixel shader 2102 uses texture sampling logic in the sampler 2110 to access texture data stored in memory for textures in the texture maps stored in memory. Arithmetic operations on the texture data and input geometry data calculate pixel color values of each geometric fragment, or discard one or more pixels without further processing.

[0245] In some embodiments, data ports 2114 provide a memory access mechanism for thread execution logic 2100 to output processed data to memory for processing on a graphics processor output pipeline. In some embodiments, data ports 2114 include or couple to one or more cache memories (e.g., data cache 2112) to cache data for memory access via data ports.

[0246] Figure 22 is a block diagram illustrating a graphics processor instruction format 2200 according to some embodiments. In one or more embodiments, a graphics processor execution unit supports an instruction set that includes a number of instructions in multiple formats. The solid lined boxes illustrate the format of a typical instruction used by an execution unit in one embodiment and provided for

[0247] In some embodiments, a graphics processor execution unit natively supports the 128-bit instruction format 2210. Some instructions are based on the 64-bit compact instruction format 2230. An instruction is specified using an opcode as well as a number of operands. Some instructions are compacted into a single 64-bit instruction and others are sufficiently long that multiple instructions (such as, for example, micro-ops) are packed into a 128-bit instruction slot.

[0248] For each format, the instruction opcode 2212 defines the operation that the execution unit is to perform. The execution unit processes one or more data elements included in a packet. The packet can be a single data element up to a vector register length worth of data elements (such as, for example, 128-data elements in one embodiment). The execution unit is instructed to perform a certain operation on the packet. In some embodiments, the instruction control field 2214 enables control over certain execution options, such as cross-lane operation, 32-bit instruction compression format and pipelined instruction operation. In some embodiments, data controls field 2216 enable control over data flow operation, such as lane selection.

[0249] Some execution unit instructions have up to three operands, including two source operands (src0 2220, src1 2222) and a destination 2218. In some embodiments, an execution unit supports dual destination instructions, where one of the destinations is implied. Data operation instructions can have a third source operand (e.g., SRC2 2224), where the instruction opcode 2212 determines the number of source operands. The last source operand of an instruction can be an immediate (e.g., hard coded) value passed with the instruction.

[0250] In some embodiments, the 128-bit instruction format 2210 includes a access / address mode field 2226 that, for example, defines whether a direct register addressing mode or an indirect register addressing mode is used. When using direct register addressing mode, the register address for one or more operands is provided directly by bits in the instruction.

[0251] In some embodiments, the 128-bit instruction format 2210 includes an access / address mode field 2226 that specifies the address mode and / or access mode of the instruction. In one embodiment, the access mode is used to define the data access alignment for the instruction. Some embodiments support access modes including a 16-byte aligned access mode and a 1-byte aligned access mode, where the byte alignment of the access mode determines the access alignment of the instruction operands. For example, when in a first mode, the instruction can use byte-aligned addressing for source and destination operands, and when in a second mode, the instruction can use 16-byte aligned addressing for all source and destination operands.

[0252] In one embodiment, the address mode portion of the access / address mode field 2226 determines whether the instruction uses direct addressing or indirect addressing. When using direct register addressing mode, the register address for one or more operands is provided directly by bits in the instruction. When using indirect register addressing mode, the register address for one or more operands can be calculated based on an address register value and an address immediate field in the instruction.

[0253] In some embodiments, instructions are grouped based on the opcode 2212 bit field to simplify opcode decoding 2240. For 8-bit opcodes, bits 4, 5, and 6 allow the execution unit to determine the type of opcode. The precise opcode grouping shown is exemplary only. In some embodiments, move and logic opcode group 2242 includes data movement and logic instructions (e.g., move (mov), compare (cmp)). In some embodiments, move and logic opcode group 2242 shares five most-significant bits (MSB), with move (mov) instructions taking the form 0000xxxxb and logic instructions taking the form 0001xxxxb. Flow control instruction group 2244 (e.g., call, jmp) includes instructions that take the form 0010xxxxb (e.g., Ox20). Hybrid instruction group 2246 includes a mix of instructions, including synchronization instructions (e.g., wait, send) that take the form 0011xxxxb (e.g., Ox30). Parallel math instruction group 2248 includes per-component arithmetic instructions (e.g., add, mul) that take the form 0100xxxxb (e.g., Ox40). Parallel math group 2248 performs arithmetic operations across data lanes in parallel. Vector math group 2250 includes arithmetic instructions (e.g., dp4) that take the form 0101xxxxb (e.g., Ox50). Vector math group performs arithmetic operations on vector operands, such as a dot product operation.

[0254] Graphics pipeline

[0255] Figure 23 is a block diagram of another embodiment of a graphics processor 2300. Figure 23 Those elements of having the same reference number (or name) in the figures of the present disclosure as elements in any other figure herein can operate or function in any manner similar to that described elsewhere herein, but are not limited to such.

[0256] In some embodiments, graphics processor 2300 includes a graphics pipeline 2320, a media pipeline 2330, a display engine 2340, thread execution logic 2350, and a render output pipeline 2370. In some embodiments, graphics processor 2300 is a graphics processor included in a multi-core processing system that includes one or more general-purpose processing cores. The graphics processor is controlled by register writes to one or more control registers (not shown) or via commands issued to the graphics processor 2300 via the ring interconnect 2302. In some embodiments, ring interconnect 2302 couples graphics processor 2300 to other processing components such as other graphics processors or general-purpose processors. Commands from the ring interconnect 2302 are interpreted by a command streamer 2303 which supplies instructions to individual components of the graphics pipeline 2320 or media pipeline 2330.

[0257] In some embodiments, command streamer 2303 directs the operation of a vertex fetcher 2305 that reads vertex data from memory and executes vertex processing commands provided by command streamer 2303. In some embodiments, vertex fetcher 2305 provides vertex data to a vertex shader 2307 that performs coordinate space transformation and lighting operations on each vertex. In some embodiments, vertex fetcher 2305 and vertex shader 2307 execute vertex processing instructions via dispatch of execution threads to execution units 2352A-2352B by thread dispatcher 2331.

[0258] In some embodiments, execution units 2352A-2352B are vector processors that are configured to execute a set of instructions for performing graphics and media operations. In some embodiments, execution units 2352A-2352B have an attached Ll cache 2351 that is dedicated to each array or shared between arrays. The cache can be configured as a data cache, an instruction cache, or both a data cache and an instruction cache. In some embodiments, execution units 2352A-2352B are configured to execute instructions in parallel.

[0259] In some embodiments, graphics pipeline 2320 includes a tessellation component for hardware-accelerated tessellation of 3D objects. In some embodiments, programmable hull shader 811 configures the tessellation operation. Programmable domain shader 817 provides post-evaluation of the tessellation output. Tessellator 2313 operates in the direction of hull shader 2311 and includes specialized logic for generating a detailed set of geometric objects based on a coarse geometric model provided as input to graphics pipeline 2320. In some embodiments, the tessellation components (e.g., hull shader 2311, tessellator 2313, domain shader 2317) can be bypassed if tessellation is not used.

[0260] In some embodiments, complete geometric objects can be processed by geometry shader 2319 via one or more threads dispatched to execution units 2352A-B, or can proceed directly to clipper 2329. In some embodiments, the geometry shader operates on entire geometric objects (rather than vertices or vertex patches as in previous stages of the graphics pipeline). If tessellation is disabled, geometry shader 2319 receives input from vertex shader 2307. In some embodiments, geometry shader 2319 can be programmed by a geometry shader program to perform geometric tessellation when the tessellation unit is disabled.

[0261] Clipper 2329 processes vertex data prior to rasterization. Clipper 2329 can be a fixed- function clipper or a programmable clipper with clip and geometry shader functionality. In some embodiments, rasterizer and depth test component 2373 in render output pipeline 2370 dispatches pixel shaders to convert a geometric object into its per-pixel representation. In some embodiments, pixel shader logic is included in thread execution logic 2350. In some embodiments, an application can bypass rasterizer and depth test component 2373 and access un-rasterized vertex data via egress unit 2323.

[0262] Graphics processor 2300 has an interconnect bus, interconnect fabric, or some other interconnect mechanism to allow data and messages to be passed between components of the graphics processor, including the main components. In some embodiments, execution units 2352A-B and associated cache(s) 2351, texture and media sampler 2354, and texture / sampler caches 2358 are interconnected via a data port 2356 to perform memory accesses and to communicate with each other and with other components of the processor. In some embodiments, sampler 2354, caches 2351, 2358, and execution units 2352A-B each have separate memory access ports to the data port 2356.

[0263] In some embodiments, the render output pipeline 2370 includes a rasterizer and depth test unit 2373 that converts vertex-based objects into associated pixel-based representations. In some embodiments, the rasterizer logic includes a windower / masker unit for performing fixed function triangle and line rasterization. An associated render cache 2378 and depth cache 2379 are also available in some embodiments. Pixel operation logic 2377 performs pixel-based operations, although in some instances pixel operations associated with 2D operations (e.g., bit block image transfers utilizing blitting) are performed by 2D engine 2341 or replaced at display time by display controller 2343 using overlay display planes. In some embodiments, a shared L3 cache 2375 is available for all graphics components, allowing sharing of data without use of main system memory.

[0264] In some embodiments, the graphics processor media pipeline 2330 includes a media engine 2337 and a video front-end 2334. In some embodiments, the video front-end 2334 receives pipeline commands from the command streamer 2303. In some embodiments, the media pipeline 2330 includes a separate command streamer. In some embodiments, the video front-end 2334 processes media commands before sending the commands to the media engine 2337. In some embodiments, the media engine 2337 includes thread spawning functionality to process media

[0265] In some embodiments, the graphics processor 2300 includes a display engine 2340. In some embodiments, the display engine 2340 is external to the processor 2300 and couples with the graphics processor via the ring interconnect 2302, or some other interconnect bus or fabric. In some embodiments, the display engine 2340 includes a 2D engine 2341 and a display controller 2343. In some embodiments, the display engine 2340 contains special purpose logic that is, in some embodiments, capable of independent operation with a 3D pipeline 2302. In some embodiments, the display controller 2343 couples with a display device (not illustrated), which is, in some embodiments, a system integrated display device (such as in a laptop computer) or an external display device attached via an display device connector.

[0266] In some embodiments, graphics pipeline 2320 and media pipeline 2330 can be configured to perform operations based on a number of graphics and media programming interfaces and not specific to any one application programming interface (API). In some embodiments, driver software for a graphics processor will translate API calls or calls that include operations using API commands into commands that can be processed by the graphics processor. In some embodiments, support is provided for APIs that include, but are not limited to, Open Graphics Library (OpenGL), Open Computing Language (OpenCL), and / or Vulkan graphics and compute API by the Khronos Group. In some embodiments, support can also be provided for Microsoft Corporation's Direct3D library. In some embodiments, a combination of these APIs can be supported. Support can also be provided for the Open Source Computer Vision Library (OpenCV). Future APIs with compatible 3D pipeline architectures can also be supported as long as mapping of

[0267] Graphics pipeline programming

[0268] Figure 24A FIG. 24 is a block diagram illustrating a graphics processor command format 2400 according to some embodiments. Figure 24B FIG. 25 is a block diagram illustrating a graphics processor command sequence 2410 according to an embodiment. Figure 24A The solid lined boxes in FIG. 25 illustrate a typical graphics processor command sequence that can be generated by an application for a graphics processing unit. The command sequence can include a number of graphics processor commands. These graphics processor commands can be, for example, in a graphics processing commands stream. The individual graphics processor commands can be ordered in the stream, for example, based upon the graphics processing operations that they are associated with. In some embodiments, the commands can be ordered in the graphics processing commands stream based on a variety of other criteria such as the order in which the operations can be performed on the graphics processor, the order in which the operations can complete, or a combination of these and / or other criteria. Figure 24A The exemplary graphics processor command format 2400 of FIG. 24 includes data fields to identify a target client 2402 for the command, a command operation code (opcode) 2404, and data fields for associated data 2406 for the command. Some commands can include a sub-opcode 2405 and a command size 2408.

[0269] In some embodiments, the client 2402 defines a client unit of the graphics device that processes the command data. In some embodiments, the graphics processor command parser examines a client field of each command to determine the further processing to be performed on the command and routes the command data to the appropriate client unit. In some embodiments, a graphics processor client unit includes a memory interface unit, render units, a 2D unit, a 3D unit, and a media unit. Each client unit has a respective processing pipeline to process the commands. Once a command is received by a client unit, the client unit reads the operation code 2404 and sub-op code 2405 (if present) to determine the operation to be performed on the command. The client unit uses information in the data field 2406 to perform the command. For some commands, an explicit command size 2408 is expected to define the size of the command. In some embodiments, the command parser automatically determines the size of at least some of the commands based on the command operation code. In some embodiments, commands are aligned via multiples of double-words.

[0270] Figure 24B The flow diagram in FIG. 24 illustrates an exemplary graphics processor command sequence 2410. In some embodiments, software or firmware of a data processing system featuring an embodiment of a graphics processor uses a version of the command sequence shown to set up, execute, and terminate a set of graphics operations. A sample command sequence is shown and described for illustrative purposes only. The embodiments are not limited to these particular commands or to this command sequence. Moreover, the described commands can be issued as batch of commands in a command sequence, such that the graphics processor will process the sequence of commands in at least partially concurrence.

[0271] In some embodiments, the graphics processor command sequence 2410 can begin with a pipeline flush command 2412 to cause any active graphics pipelines to complete any current pending commands in the pipeline. In some embodiments, the 3D pipeline 2422 and media pipeline 2424 are not operating simultaneously. The pipeline flush is performed to ensure that active graphics pipelines complete any pending commands. In response to the pipeline flush, the command parser for the graphics processor will stop processing commands until the active draw engine completes the pending operations and the related read cache is invalidated. Optionally, any data in the render caches that is marked dirty can be flushed to memory. In some embodiments, the pipeline flush command 2412 can be used in conjunction with a pipeline synchronization or before bringing the graphics processor to a low power state.

[0272] In some embodiments, a pipeline selection command 2413 is used when the command sequence requires the graphics processor to explicitly switch between pipelines. In some embodiments, only one pipeline selection command 2413 is needed in the execution context before issuing the pipeline commands, unless the context is issuing commands to two pipelines. In some embodiments, a pipeline flush clear command 2412 is needed just before the pipeline switch via the pipeline selection command 2413.

[0273] In some embodiments, pipeline control commands 2414 configure the graphics pipeline for operation and to program the 3D pipeline 2422 and media pipeline 2424. In some embodiments, the pipeline control commands 2414 configure the pipeline state for the active pipeline. In one embodiment, the pipeline control commands 2414 are used for pipeline synchronization and to clear data from one or more cache memories within the active pipeline before processing a batch of commands.

[0274] In some embodiments, a return buffer state command 2416 is used to configure a set of return buffers for a respective pipeline to write data. Some pipeline operations require allocation, selection, or configuration of one or more return buffers in which intermediate data is written during processing. In some embodiments, the graphics processor also uses one or more return buffers to store output data and to perform cross-thread communication. In some embodiments, the return buffer state 2416 includes selecting the size and number of return buffers for a set of pipeline operations.

[0275] The remaining commands in the command sequence differ based on the active pipeline for operation. Based on the pipeline determination 2420, the command sequence is tailored for either the 3D pipeline 2422 starting with 3D pipeline state 2430, or the media pipeline 2424 starting at media pipeline state 2440.

[0276] The commands for 3D pipeline state 2430 include 3D state setting commands for vertex buffer state, vertex element state, constant color state, depth buffer state, and other state variables to be configured prior to processing 3D primitive commands. The values for these commands are determined at least in part on the particular 3D API in use. In some embodiments, the 3D pipeline state 2430 commands can also selectively disable or bypass certain pipeline elements if those elements will not be used.

[0277] In some embodiments, 3D primitive 2432 commands are used to submit 3D primitives to be processed by the 3D pipeline. Commands and associated parameters passed to the graphics processor via a 3D primitive 2432 command are forwarded to a vertex fetch function in the graphics pipeline. The vertex fetch function uses the 3D primitive 2432 command data to generate a number of vertex data structures. The vertex data structures are stored in one or more vertex buffers. In some embodiments, 3D primitive 2432 commands are used to perform vertex operations on 3D primitives via a vertex shader. To process the vertex shader, the 3D pipeline 2422 dispatches shader execution threads to the graphics processor execution units.

[0278] In some embodiments, 3D pipeline 2422 is triggered via an execute 2434 command or an event. In some embodiments, a register write triggers command execution. In some embodiments, execution is triggered via a 'go' or 'kick' command in the command sequence. In one embodiment, a pipeline synchronization command is used to trigger command execution in order to flush the command sequence with no-ops through the graphics pipeline. The 3D pipeline will perform geometry processing for 3D primitives. Once the processing is completed, the resulting geometry is rasterized and the pixel engine is used to shade the resulting pixels. For these operations, additional commands can also be included to control the pixel shading and pixel back end operations.

[0279] In some embodiments, when performing media operations, graphics processor command sequence 2410 follows media pipeline 2424 path. In general, the specific use and manner of programming for media pipeline 2424 depends on the media or compute operations to be performed. In media decode processes, specific media decode operations can be offloaded to the media pipeline. In some embodiments, the media pipeline can also be bypassed and media decode can be performed in whole or in part using resources provided by one or more general purpose processing cores. In one embodiment, the media pipeline also includes elements for general purpose graphics processor unit (GPGPU) operations, where the graphics processor is used to perform SIMD vector operations using a compute shader program that is not explicitly related to the rendering of a graphics primitive.

[0280] In some embodiments, media pipeline 2424 is configured in a similar manner as 3D pipeline 2422. A set of commands to configure media pipeline state 2440 is dispatched or placed into a command queue, prior to media object command 2442. In some embodiments, commands 2440 for media pipeline state include data to configure media pipeline elements that will be used to process the media object. This includes data to configure video decode and video encode logic within the media pipeline, such as encoding or decoding formats. In some embodiments, commands 2440 for media pipeline state implement support for using one or more pointers for "indirect" state elements that contain a batch of state settings.

[0281] In some embodiments, media object command 2442 supplies a pointer to a media object for processing by the media pipeline. The media object includes a memory buffer containing video data to be processed. In some embodiments, all media pipeline state must be valid prior to issuing media object command 2442. Once the pipeline state is configured and media object command 2442 is queued, the media pipeline 2424 is triggered via an execute 2444 command or equivalent execution event (e.g., register write). Output from the media pipeline 2424 can then be post-processed by operations provided by 3D pipeline 2422 or media pipeline 2424. In some embodiments, GPGPU operations are configured and executed in a similar manner as media operations.

[0282] Graphics software architecture

[0283] Figure 25 An exemplary graphics software architecture of a data processing system 2500 is shown in accordance with some embodiments. In some embodiments, the software architecture includes a 3D graphics application 2510, an operating system 2520, and at least one processor 2530. In some embodiments, processor 2530 includes a graphics processor 2532 and one or more general-purpose processor cores 2534. Graphics application 2510 and operating system 2520 each execute in system memory 2550 of the data processing system.

[0284] In some embodiments, 3D graphics application 2510 contains one or more shader programs including shader instructions 2512. The shader language instructions can be in a high-level shader language, such as the High-Level Shader Language (HLSL) or the OpenGL Shader Language (GLSL). The application also includes executable instructions 2514 in a machine language suitable for execution by the general- purpose processor core(s) 2534. The application also includes graphics objects 2516 defined by vertex data.

[0285] In some embodiments, the operating system 2520 is from Microsoft Corporation. The operating system 2520 may be a dedicated UNIX-like operating system or an open-source UNIX-like operating system using a variant of the Linux kernel. The operating system 2520 may support graphics APIs 2522, such as the Direct3D API, OpenGL API, or Vulkan API. When the Direct3D API is in use, the operating system 2520 uses a front-end shader compiler 2524 to compile any shader instructions 2512 in HLSL into a lower-level shader language. This compilation may be just-in-time (JIT) compilation or pre-compilation of the application-executable shaders. In some embodiments, high-level shaders are compiled into low-level shaders during the compilation of the 3D graphics application 2510. In some embodiments, the shader instructions 2512 are provided in an intermediate form, such as a version of the standard Portable Intermediate Representation (SPIR) used by the Vulkan API.

[0286] In some embodiments, the user-mode graphics driver 2526 includes a back-end shader compiler 2527 for translating shader instructions 2512 into a hardware-specific representation. When using the OpenGL API, shader instructions 2512 in the GLSL high-level language are passed to the user-mode graphics driver 2526 for compilation. In some embodiments, the user-mode graphics driver 2526 uses an operating system kernel-mode feature 2528 to communicate with a kernel-mode graphics driver 2529. In some embodiments, the kernel-mode graphics driver 2529 communicates with a graphics processor 2532 to dispatch commands and instructions.

[0287] IP core implementation

[0288] One or more aspects of at least one embodiment can be implemented by representative code stored on a machine-readable medium that represents and / or defines logic within an integrated circuit, such as a processor. For example, the machine-readable medium may include instructions representing various logic within a processor. When read by a machine, these instructions can cause the machine to manufacture logic for performing the techniques described herein. Such representations (referred to as “IP cores”) are reusable units of logic for an integrated circuit, which can be stored on a tangible, machine-readable medium as a hardware model describing the structure of the integrated circuit. The hardware model can be supplied to various consumers or manufacturing facilities that load the hardware model onto manufacturing machines that manufacture integrated circuits. Integrated circuits can be manufactured such that the circuits perform the operations described in association with any of the embodiments described herein.

[0289] Figure 26 is a block diagram showing an IP core development system 2600 that can be used to manufacture integrated circuits to perform operations in accordance with embodiments. The IP core development system 2600 can be used to generate modular, re-usable designs that can be incorporated into larger designs or used to construct an entire integrated circuit (e.g., an SOC integrated circuit). A design facility 2630 can employ a high-level programming language (e.g., C / C++) to generate a software simulation 2610 of the IP core design. The software simulation 2610 can be used to design, test, and verify the behavior of the IP core using a simulation model 2612. The simulation model 2612 can include functional, behavioral, and / or timing simulations. The simulation model 2612 can then be used to create or synthesize a register transfer level (RTL) design 2615. The RTL design 2615 is an abstraction of the behavior of the integrated circuit (including associated logic that executes using modeled digital signals) that models the flow of digital signals between hardware registers, including the associated logic performed by the hardware registers. In addition to an RTL design 2615, a lower- level design, such as a logic level or transistor level design can also be created, designed, or synthesized.

[0290] The RTL design 2615, or equivalent, can be further synthesized, formatted, or prepared to produce a hardware model 2620, which can be in a hardware description language (HDL) or some other representation of the design. The HDL can be further simulated or tested to verify the IP core design. The IP core design can be stored using non-volatile memory 2640, such as a hard disk, flash memory, or any non-transitory storage medium, for delivery to a third party fabrication facility 2665. Alternatively, the IP core design can be transmitted (e.g., via the Internet) over a wired connection 2650 or wireless connection 2660. The fabrication facility 2665 can then manufacture an integrated circuit that is based at least in part on the IP core design. The manufactured integrated circuit can be configured to perform operations in accordance with at least one embodiment described herein.

[0291] Exemplary system on a chip integrated circuit

[0292] Figures 27-29 Exemplary integrated circuits and related graphics processors that can be manufactured using one or more IP cores in accordance with various embodiments described herein are shown. Other logic and circuitry can also be included, including additional graphics processors / cores, peripheral interface controllers, or general-purpose processor cores.

[0293] Figure 27is a block diagram that illustrates an exemplary system on a chip integrated circuit 2700 that can be fabricated using one or more IP cores in accordance with embodiments. The exemplary integrated circuit 2700 includes one or more application processors 2705 (e.g., CPUs), at least one graphics processor 2710, and can additionally include an image processor 2715 and / or a video processor 2720, any of which can be a modular IP core from the same or multiple different design facilities. The integrated circuit 2700 includes peripheral or bus logic including USB controllers 2725, UART controllers 2730, SPI / SDIO controllers 2735, and I2C controllers 2740. In addition, the integrated circuit can also include a display device 2745 coupled to one or more of a high-definition multimedia interface (HDMI) controller 2750 and a mobile industry processor interface (MIPI) display interface 2755. Storage can be provided by a flash memory subsystem 2760 including flash memory and a flash memory controller. Memory interfaces can be provided via a memory controller 2765 to access SDRAM or SRAM memory devices. In addition, some integrated circuits include an embedded security engine 2770. 2 S / I 2 C controller 2740. In addition, the integrated circuit can also include a display device 2745 coupled to one or more of a high-definition multimedia interface (HDMI) controller 2750 and a mobile industry processor interface (MIPI) display interface 2755. Storage can be provided by a flash memory subsystem 2760 including flash memory and a flash memory controller. Memory interfaces can be provided via a memory controller 2765 to access SDRAM or SRAM memory devices. In addition, some integrated circuits include an embedded security engine 2770.

[0294] Figure 28 is a block diagram that illustrates an exemplary graphics processor 2810 that can be fabricated using one or more IP cores in accordance with embodiments. Graphics processor 2810 can be a variant of the graphics processor 2710 of FIG. 27. Figure 27 Graphics processor 2810 includes a vertex processor 2805 and one or more fragment processor(s) 2815A-2815N (e.g., 2815A, 2815B, 2815C, 2815D, up to 2815N-1, and 2815N). Graphics processor 2810 can execute different shader programs via separate logic for vertex processing and / or for fragment / luminance processing, where the shade programs employed for the processing can be in different domains and / or address different processing functionality. Vertex processor(s) 2805 is optimized to execute operations for vertex processing, such as rendering, decompression, and the like, which use one or more graphics processing cores to perform the processing in a complex data domain with operations that can have a variable number of latency cycles. In one embodiment, vertex processing is optimized for a high number of threads at the expense of single thread performance for smooth and seamless multi-threaded graphics processing. Fragment processor(s) 2815A-2815N receive processed vertex data from vertex processor(s) 2805 for processing or generating a rendered image. In one embodiment, a number of threads for processing is optimized for high single-thread performance to maximize the use of a single core for the execution of instructions processed for a fixed number of threads, as opposed to the high multi-threading for a variable number of threads as described in the context of the vertex processing. Thus, both the vertex and the fragment processor(s) have different kernel profiles that are optimized for their different processing needs.

[0295] In addition, the graphics processor 2810 also includes one or more memory management units (MMUs) 2820A-2820B, cache(s) 2825A-2825B, and circuit interconnects 2830A-2830B. One or more MMU(s) 2820A-2820B provide for virtual to physical address mapping for vertex processor 2805, and / or the one or more fragment processor(s) 2815A-2815N, in addition to other processing and / or graphics processors coupled to the circuit interconnects 2830A-2830B. The one or more MMU(s) 2820A-2820B couple with other MMUs within the system, such as the one or more MMUs associated with one or more application processor(s) 2705, a graphics processor 2710, and / or a display processor 2715, as described herein, such that each processor 2705-2720 can participate in a shared or unified virtual memory system. Figure 27 The one or more circuit interconnects 2830A-2830B enable the graphics processor 2810 to interface with other IP cores within the SoC, either via an internal bus, as shown, or via a direct connection. In either implementation, the circuit interconnects 2830A-2830B include circuit hardware that operates to send and receive communication signals between the graphics processor 2810 and other IP cores within the SoC.

[0296] Figure 29 is a block diagram of an additional exemplary graphics processor 2910 that can be fabricated using one or more IP cores, according to an embodiment. The graphics processor 2910 can be a variant of the graphics processor 2710 of Figure 27 FIG. 27. The graphics processor 2910 includes one or more shader engines 2915A-2915N, which implement a Figure 28 The graphics processor 2910 also includes one or more memory management units (MMUs) 2820A-2820B, cache(s) 2825A-2825B, and circuit interconnects 2830A-2830B, which are optional components when the graphics processor 2910 is used in scenarios

[0297] The graphics processor 2910 includes one or more shader cores 2915A-2915N (e.g., 2915A, 2915B, 2915C, 2915D, 2915E, 2915F, up to 2915N-1, and 2915N) that provide a unified shader core architecture, in which single instruction multiple thread (SIMT) instruction processing can be performed on each shader core. The exact numbers of shader cores present can vary in embodiments and implementations. Additionally, the graphics processor 2910 also includes an inter-core task manager 2905 that acts as a thread dispatcher and scheduler, and a block unit 2918 that can operate as a lightweight GPGPU processor that runs a task scheduler and handles tasks assigning tasks to shader cores.

[0298] The following relates to further examples.

[0299] Example 1 can optionally include an apparatus comprising: a plurality of execution units; and logic, at least partially including hardware logic, to expose embedded cast operations in at least one of a load instruction or a store instruction; determine a target precision level for the cast operations; and load the cast operations at the target precision level.

[0300] Example 2 can optionally include the apparatus of example 1, wherein the target precision level represents a best precision level.

[0301] Example 3 can optionally include the apparatus of any of examples 1-2, wherein the target precision level is determined to match a hardware capability.

[0302] Example 4 can optionally include the apparatus of any of examples 1-3, wherein different data types are used to represent individual weights in different layers of a multi-layer deep neural network (DNN).

[0303] Example 5 can optionally include the apparatus of any of examples 1-4, wherein high precision floating point data is used for one or more lower layers of a neural network.

[0304] Example 6 can optionally include the apparatus of any of examples 1-5, wherein lower precision floating point data and integers are used for one or more higher layers of a neural network.

[0305] Example 7 can optionally include an electronic device comprising a processor having a plurality of execution units, and logic, at least partially including hardware logic, to expose an embedded predicate operation in at least one of a load instruction or a store instruction, determine a target precision level for the predicate operation, and load the predicate operation at the target precision level.

[0306] Example 8 can optionally include the apparatus of example 7, wherein the target precision level represents an optimal precision level.

[0307] Example 9 can optionally include the apparatus of any of examples 7-8, wherein the target precision level is determined to match a hardware capability.

[0308] Example 10 can optionally include the apparatus of any of examples 7-9, wherein different data types are used to represent individual weights in different layers of a multi-layer deep learning neural network (DNN).

[0309] Example 11 can optionally include the apparatus of any of examples 7-10, wherein high precision floating point data is used for one or more lower layers of a neural network.

[0310] Example 12 can optionally include the apparatus of any of examples 7-11, wherein lower precision floating point data and integers are used for one or more higher layers of a neural network.

[0311] In various embodiments, the operations discussed herein can be implemented as hardware (e.g., logic circuitry), software, firmware, or a combination thereof, which can be provided as a computer program product, e.g., including a tangible (e.g., non-volatile) machine-readable or computer-readable medium having stored thereon instructions (or software procedures) for programming a computer to perform the processes discussed herein. A machine-readable medium can include a machine information storage device.

[0312] Further, such computer readable media can be downloaded as a computer program product, wherein the program can be transferred from a remote computer (e.g., a server) to a requesting computer (e.g., a client) by way of a communication link (e.g., a bus, a modem, or a network connection) as a data signal embodied in a carrier wave or other propagation medium.

[0313] Reference throughout this specification to "one embodiment" or "an embodiment" means that a particular feature, structure, or characteristic described in connection with the embodiment is included in at least one implementation. The appearances of the phrase "in one embodiment" in various places in the specification are not necessarily all referring to the same embodiment.

[0314] Also, the terms "coupled" and "connected," along with their derivatives, can be used in this specification and claims. In some embodiments, "connected" can be used to indicate that two or more elements are in direct physical and / or electrical contact with each other. "Coupled" can mean that two or more elements are in direct physical or electrical contact with each other. However, "coupled" can also mean that two or more elements are not in direct contact with each other, but yet still co-operate and / or interact with each other.

[0315] Thus, although embodiments have been described with regard to particular structural features and / or method actions, it is to be understood that the claimed subject matter can not be limited to the particular features or actions described. Rather, the particular features and actions are disclosed as sample forms of implementing the claimed subject matter.

Claims

1. An apparatus comprising: Multiple execution units; as well as Logic that includes at least part of hardware logic, said logic being used for: An embedded projection operation is revealed in at least one of the load or store instructions; For each layer in a multilayer deep learning neural network (DNN), a target precision level for the projection operation is determined, wherein first precision data is used to represent the weights in a first group of layers of the DNN, and second precision data is used to represent the weights in a second group of layers of the DNN, the first group comprising the first layer of the DNN, and the second group comprising layers of the DNN above the first layer, wherein the first precision is higher than the second precision; and The projection operation is loaded with the target precision level; in: The target accuracy level is determined to match the hardware capabilities.

2. An electronic device, comprising: A processor having a plurality of execution units; and Logic that includes at least part of hardware logic, said logic being used for: An embedded projection operation is revealed in at least one of the load or store instructions; as well as For each layer in a multilayer deep learning neural network (DNN), a target precision level for the projection operation is determined, wherein first precision data is used to represent the weights in a first group of layers of the DNN, and second precision data is used to represent the weights in a second group of layers of the DNN, the first group comprising the first layer of the DNN, and the second group comprising layers of the DNN above the first layer, wherein the first precision is higher than the second precision; and The projection operation is loaded with the target precision level; in: The target accuracy level is determined to match the hardware capabilities.

Images