Decomposition of SoC architecture

Abstract

This application discloses decomposition of SoC architectures. Embodiments described herein provide techniques for decomposing the architecture of a system-on-a-chip integrated circuit into a plurality of different dies that can be packaged onto a common chassis. In one embodiment, a graphics processing unit or parallel processor is composed of a variety of separately manufactured silicon dies. A die is an at least partially packaged integrated circuit that includes different units of logic that can be assembled with other dies into a larger package. A diverse set of dies with different IP core logic can be assembled into a single device.

Description

[0001] This patent application is a continuation-in-part of International Application No. PCT / US2020 / 014766, International Filing Date January 23, 2020, which entered the National Stage in the United States as Application No. 202080014312.1, entitled “Decomposition of SoC Architecture,” filed on January 23, 2020.

[0002] Related Applications

[0003] This application claims the benefit of U.S. Application No. 16 / 355,377, filed March 15, 2019, which is hereby incorporated by reference in its entirety. TECHNICAL FIELD

[0004] Embodiments generally relate to the design and manufacture of general purpose graphics and parallel processing units. BACKGROUND

[0005] Current parallel graphics data processing includes systems and methods developed to perform specific operations on graphics data, such as, for example, linear interpolation, tessellation, rasterization, texture mapping, depth testing, etc. Traditionally, graphics processors have used fixed-function compute units to process graphics data; however, recently, some graphics processors have been made programmable, enabling such processors to support a wider variety of operations for processing vertex and fragment data.

[0006] To further improve performance, graphics processors typically implement processing techniques such as pipelining, which attempt to process as much graphics data in parallel throughout different parts of the graphics pipeline as possible. Parallel graphics processors with a single instruction, multiple thread (SIMT) architecture are designed to maximize the amount of parallel processing in the graphics pipeline. In a SIMT architecture, groups of parallel threads attempt to execute program instructions together as often as possible to improve processing efficiency. A general overview of software and hardware for SIMT architectures can be found in Chapter 3, pages 37-51 of CUDA Programming by Shane Cook (2013). BRIEF DESCRIPTION OF DRAWINGS

[0007] For a more detailed understanding of the above-recited features of the present embodiments, reference is made to the following written description and accompanying drawings in which some embodiments are illustrated and described, and in which:

[0008] Figure 1 is a block diagram illustrating a computer system configured to implement one or more aspects of the embodiments described herein;

[0009] Figures 2A-2D illustrates a parallel processor component according to an embodiment;

[0010] Figures 3A-3C is a block diagram of a graphics multiprocessor and a multiprocessor-based GPU according to embodiments;

[0011] Figures 4A-4F shows an exemplary architecture in which multiple GPUs are communicatively coupled to multiple multi-core processors;

[0012] Figure 5 shows a graphics processing pipeline according to embodiments;

[0013] Figure 6 shows a machine learning software stack according to embodiments;

[0014] Figure 7 shows a general purpose graphics processing unit according to embodiments;

[0015] Figure 8 shows a multi-GPU computing system according to embodiments;

[0016] Figures 9A-9B shows a layer of an exemplary deep neural network;

[0017] Figure 10 shows an exemplary recurrent neural network;

[0018] Figure 11 shows training and deployment of a deep neural network;

[0019] Figure 12 is a block diagram showing distributed learning;

[0020] Figure 13 shows an exemplary inference system-on-a-chip (SOC) suitable for performing inference using a trained model;

[0021] Figure 14 is a block diagram of a processing system according to embodiments;

[0022] Figure 15 is a block diagram of a processor according to embodiments;

[0023] Figure 16 is a block diagram of a graphics processor according to embodiments;

[0024] Figure 17 is a block diagram of a graphics processing engine of a graphics processor according to some embodiments;

[0025] Figure 18 is a block diagram of hardware logic of a graphics processor core according to some embodiments described herein;

[0026] Figures 19A-19BThread execution logic including an array of processing elements employed in a graphics processor core is shown in accordance with embodiments described herein;

[0027] Figure 20 is a block diagram illustrating a graphics processor instruction format in accordance with some embodiments;

[0028] Figure 21 is a block diagram of a graphics processor in accordance with another embodiment;

[0029] Figures 22A-22B Graphics processor command formats and command sequences are shown in accordance with some embodiments;

[0030] Figure 23 An exemplary graphics software architecture for a data processing system is shown in accordance with some embodiments;

[0031] Figure 24A is a block diagram illustrating an IP core development system for use in designing IP cores in accordance with embodiments;

[0032] Figure 24B A cross-sectional side view of an integrated circuit package assembly is shown in accordance with some embodiments described herein;

[0033] Figure 25 is a block diagram illustrating an exemplary system on a chip integrated circuit in accordance with embodiments;

[0034] Figures 26A-26B is a block diagram illustrating an exemplary graphics processor for use within a SoC in accordance with embodiments described herein;

[0035] Figure 27 A parallel computing system is shown in accordance with embodiments;

[0036] Figures 28A-28B A mixed logical / physical view of a disaggregated parallel processor is shown in accordance with embodiments described herein;

[0037] Figures 29A-29B A package view of a disaggregated parallel processor is shown in accordance with embodiments;

[0038] Figure 30 An information transfer system for an interconnect fabric is shown in accordance with embodiments;

[0039] Figure 31 Transmitting messages or signals between functional units across multiple physical links of an interconnect fabric is shown;

[0040] Figure 32 Transmitting messages or signals for multiple functional units across a single physical link of an interconnect fabric is shown;

[0041] Figure 33Methods of showing functional unit configuration fabric connections within a disaggregated parallel processor;

[0042] Figure 34 Methods of relaying messages and / or signals across an interconnect fabric within a disaggregated parallel processor are shown;

[0043] Figure 35 Methods of power gating chiplets on a per work load basis are shown;

[0044] Figure 36 Parallel processor components including interchangeable chiplets are shown;

[0045] Figure 37 Interchangeable chiplet systems according to embodiments are shown;

[0046] Figure 38 An illustration of multiple traffic classes carried over a virtual lane according to embodiments;

[0047] Figure 39 Methods of agnostic data transfer between sockets of interchangeable chiplets according to embodiments are shown;

[0048] Figure 40 Modular architecture of interchangeable chiplets according to embodiments is shown;

[0049] Figure 41 Using standardized chassis interfaces for enabling chiplet testing, validation, and integration is shown;

[0050] Figure 42 Using individually boxed chiplets to create various product tiers is shown; and

[0051] Figure 43 Methods of enabling different product tiers based on chiplet configurations are shown. DETAILED DESCRIPTION

[0052] In some embodiments, a graphics processing unit (GPU) is communicatively coupled to a host / processor core to accelerate graphics operations, machine learning operations, pattern analysis operations, and various general purpose GPU (GPGPU) functions. The GPU can be communicatively coupled to the host processor / cores over a bus or another interconnect (e.g., a high-speed interconnect such as PCIe or NVLink). In other embodiments, the GPU can be integrated on the same die as the cores and communicatively coupled to the cores through an internal processor bus / interconnect (i.e., internal to the package or chip). Regardless how the GPU connects to the processor cores, the processor cores can allocate work to the GPU in the form of sequences of commands / instructions contained in work descriptors. The GPU then uses dedicated circuitry / logic for efficiently processing these commands / instructions in sequence.

[0053] In the following description, numerous specific details are set forth to provide a more thorough understanding. One skilled in the relevant art, however, will appreciate the embodiments described herein can be practiced without one or more of the specific details, or with other methods, components, materials, etc. In other instances, well-known features, such as features of a computer system, are not described in detail to avoid obscuring the description of the present embodiments.

[0054] System overview

[0055] Figure 1 FIG. 1 is a block diagram illustrating a computing system 100 configured to implement one or more aspects of the embodiments described herein. The computing system 100 includes a processing subsystem 101 having one or more processor(s) 102 and system memory 104, which are in communication via an interconnect path 105 that can include a memory hub 105. The memory hub 105 can be a separate component coupled with or integrated within the one or more processor(s) 102. The memory hub 105 communicates with an I / O subsystem 111 via an interconnect path 106. The I / O subsystem 111 includes an I / O hub 107 that can enable the computing system 100 to receive input from one or more input devices 108. Additionally, the I / O hub 107 can enable a display controller to provide outputs to one or more display devices 110A that can be coupled via the I / O hub 107. In one embodiment, the one or more display devices 110A coupled with the I / O hub 107 can include a local, internal, or embedded display device.

[0056] In one embodiment, the processing subsystem 101 includes one or more parallel processor(s) 112 coupled to memory hub 105 via a bus or other communication link 113. The one or more parallel processor(s) 112 can be one of the many processors available from Intel Corporation of Santa Clara, California and AMD Corporation of the Sunnyvale, California. Alternatively, the parallel processor(s) 112 can be graphics processing units (GPUs), physics processing units (PPUs), general purpose graphics processing units (GPGPUs), digital signal processors (DSPs), tensor processing units (TPUs), or any other processor suitable for implementation in a computing system. In one embodiment, the one or more parallel processor(s) 112 form a computationally-intensive, parallel, or vector processing system that can include a large number of processing cores and / or processing clusters (such as integrated numerous core (MIC) processors). In one embodiment, the one or more parallel processor(s) 112 form a graphics processing subsystem that can output pixels to one or more of the display device(s) 110A coupled via the I / O hub 107. The one or more parallel processor(s) 112 can also include a display controller and display interface (not shown) to enable a direct connection to one or more display device(s) 110B.

[0057] Within I / O subsystem 111, system storage 114 can be connected to I / O hub 107 to provide storage mechanisms for computing system 100. I / O switch 116 can be used to provide an interface mechanism to enable connections between I / O hub 107 and other components such as network adapter 118 and / or wireless network adapter 119, which can be integrated into the platform, as well as various other devices that can be added via one or more add-in devices 120. Network adapter 118 can be an Ethernet adapter or another wired network adapter. Wireless network adapter 119 can include one or more of Wi-Fi, Bluetooth, Near Field Communication (NFC), or other network devices including one or more wireless radios.

[0058] Computing system 100 can include other components not explicitly shown, including USB or other port connections, optical storage drives, video capture devices, and the like, which can also be connected to I / O hub 107. The communication paths for which various components in Figure 1 The communication paths interconnecting the various components in computing system 100 can use any suitable protocol, such as a PCI (Peripheral Component Interconnect) - based protocol (e.g., PCI Express), or any other bus or point-to-point communication interface and / or protocol, such as NV-Link high-speed interconnect, or interconnect protocols known in the art.

[0059] In one embodiment, the one or more parallel processors 112 incorporate circuitry optimized for graphics and video processing, including, for example, video output circuitry. In another embodiment, the one or more parallel processors 112 incorporate circuitry optimized for general purpose processing. In yet another embodiment, the components of computing system 100 can be integrated with one or more other system elements on a single integrated circuit. For example, the one or more parallel processors 112, memory hub 105, processor(s) 102, and I / O hub 107 can be integrated together into a system on a chip (SoC) integrated circuit. Alternatively, components of computing system 100 can be integrated into a single package to form a system in a package (SIP) configuration. In one embodiment, at least a portion of computing system 100 can be integrated into a multi-chip module (MCM), which can be interconnected with other multi-chip modules to form a modular computing platform.

[0060] It will be recognized that the computing system 100 shown in here is illustrative and that variations and modifications are possible. The connection topology, including the number and arrangement of bridges, the number of processors 102 and the number of parallel processors 112, can be modified as desired. For example, in some embodiments, system memory 104 is connected to the processor(s) 102 directly rather than through a bridge, and other devices communicate with system memory 104 and the processor(s) 102 via a memory hub 105. In other embodiments, the parallel processor(s) 112 are connected to an I / O hub 107 or directly to one of the one or more processors 102, rather than to the memory hub 105. In still other embodiments, I / O hub 107 and memory hub 105 are integrated into a single chip. Some embodiments can include two or more sets of processors 102 attached via multiple sockets, which can be coupled with two or more instances of the parallel processor(s) 112.

[0061] Some of the specific components shown here are optional and can not be included in all implementations of the computing system 100. For example, any number of add-in cards or peripherals can be supported, or some components can be eliminated. Additionally, some architectures can use different terminology for components that are analogous to those described here. For example, in some architectures a memory hub 105 can be called a northbridge and an I / O hub 107 can be called a southbridge. Figure 1

[0062] Figure 2A A parallel processor 200 according to an embodiment is shown. The various components of the parallel processor 200 can be implemented using one or more integrated circuit devices, such as programmable processor(s), application specific integrated circuits (ASICs), or field programmable gate arrays (FPGAs). The parallel processor 200 shown is an embodiment of one or more parallel processors 112 shown in FIG. 1. Figure 1

[0063] ​​In one embodiment, parallel processor 200 includes a parallel processing unit 202. Parallel processing unit 202 includes an I / O unit 204 that enables communication with other devices (including other instances of parallel processing unit 202). The I / O unit 204 can be directly connected to the other devices. In one embodiment, the I / O unit 204 communicates with other devices via the use of a hub or switch interface, such as memory hub 105. The connection between the memory hub 105 and the I / O unit 204 form a communication link 113. Within the parallel processing unit 202, the I / O unit 204 is connected to a host interface 206 and a memory crossbar 216, where the host interface 206 receives commands required to perform processing tasks and the memory crossbar 216 receives commands required to

[0064] When host interface 206 receives a command buffer via I / O unit 204, host interface 206 can direct the work operations for performing those commands to front end 208. In one embodiment, front end 208 is coupled with scheduler 210, which is configured to distribute the commands or other work items to the processing cluster array 212. In one embodiment, scheduler 210 ensures that the processing clusters array 212 are properly configured and in an active state before tasks are distributed to the processing clusters of processing cluster array 212. In one embodiment, scheduler 210 is implemented via firmware logic executed on a microcontroller. The microcontroller implemented scheduler 210 can be configured to perform complex scheduling and work distribution operations with rough and fine grain granularity, enabling context switching and rapid preemption between threads of execution on the processing array 212. In one embodiment, host software can inspect the work load scheduled on the processing array 212 via one of a number of graphics processing doorbells. The work load can then be automatically distributed across the processing array 212 by the scheduler 210 logic within the scheduler microcontroller.

[0065] The processing cluster array 212 can include up to "N" processing clusters (e.g., cluster 214A, 214B, through 214N). Each cluster 214A-214N of the processing cluster array 212 can execute a large number of concurrent threads. The scheduler 210 can allocate work to the clusters 214A-214N of the processing cluster array 212 using various scheduling and / or work distribution algorithms. The scheduling can be dynamic, made dependent upon the work load associated with each type of program or computation, or can be assisted in part by compiler logic during compilation of program logic configured for execution by the processing cluster array 212. In one embodiment, different clusters 214A-214N of the processing cluster array 212 can be allocated for processing different types of programs or for performing different types of computations.

[0066] The processing cluster array 212 can be configured to perform a wide variety of parallel processing operations. In one embodiment, the processing cluster array 212 is configured to perform general-purpose parallel compute operations. For example, the processing cluster array 212 can include logic to perform processing tasks comprising filtering video and / or audio data, performing modeling operations (including physics operations), and performing data transformations.

[0067] In one embodiment, the processing cluster array 212 is configured to perform parallel graphics processing operations. In an embodiment in which the parallel processor 200 is configured to perform graphics processing operations, the processing cluster array 212 can include additional logic to support the execution of such graphics processing operations including without limitation texture sampling logic to perform texture parameter estimation for substituting texture cube map information into 3D wire cube map primitives; a surface tessellation unit to flatten surfaces for 3D geometry;

[0068] In one embodiment, when parallel processing unit 202 is used to perform graphics processing, scheduler 210 can be configured to divide processing workload into approximately equal sized tasks to better enable distribution of graphics processing operations to multiple clusters 214A-214N of processing cluster array 212. In some embodiments, portions of processing cluster array 212 can be configured to perform different types of processing. For example, a first portion can be configured to perform vertex shading and topology generation, a second portion can be configured to perform tessellation and geometry shading, and a third portion can be configured to perform pixel shading or other screen space operations to produce a rendered image for display. Intermediate data produced by one or more of clusters 214A-214N can be stored in buffers to allow the intermediate data to be transmitted between clusters 214A-214N for further processing.

[0069] During operation, processing cluster array 212 can receive processing tasks to be executed via scheduler 210, which receives commands defining processing tasks from front end 208. For graphics processing operations, the processing tasks can include commands and state parameters that define how data is to be processed (e.g., what programs are to be executed) and an index of data to be processed, e.g., an index of surface (patch) data, primitive data, vertex data, and / or pixel data. Scheduler 210 can be configured to fetch the index corresponding to a task, or the index can be received from front end 208. Front end 208 can be configured to ensure that processing cluster array 212 is configured in an effective state prior to initiating a workload specified by an incoming command buffer (e.g., a batch buffer, a push buffer, etc.).

[0070] Each of the one or more instances of parallel processing unit 202 can be coupled with a parallel processor memory 222. Parallel processor memory 222 can be accessed via a memory crossbar 216, which can receive memory requests from processing cluster array 212 as well as I / O unit 204. Memory crossbar 216 can access parallel processor memory 222 via a memory interface 218. Memory interface 218 can include a number of partition units (e.g., partition unit 220A, partition unit 220B, up to partition unit 220N), which can each be coupled to a portion (e.g., memory unit) of parallel processor memory 222. In one implementation, the number of partition units 220A-220N is configured to be equal to the number of memory units from memory units 224A-224N, such that first partition unit 220A has a corresponding first memory unit 224A, second partition unit 220B has a corresponding memory unit 224B, and Nth partition unit 220N has a corresponding Nth memory unit 224N. In other embodiments, the number of partition units 220A-220N can not be equal to the number of memory devices.

[0071] In various embodiments, memory units 224A-224N can include various types of memory devices including dynamic random access memory (DRAM) or graphics random access memory, such as synchronous graphics random access memory (SGRAM), including graphics double data rate (GDDR) memory. In one embodiment, memory units 224A-224N can also include 3D stacked memory, including but not limited to high bandwidth memory (HBM). Those skilled in the art will recognize that the particular implementation of memory units 224A-224N can vary and can be selected from one of various conventional designs. Render targets, such as frame buffers or texture maps, can be stored across memory units 224A-224N, allowing partition units 220A-220N to write portions of each render target in parallel to efficiently use the available bandwidth of parallel processor memory 222. In some embodiments, local instances of parallel processor memory 222 can be excluded in favor of a unified memory design utilizing system memory in conjunction with local cache memory.

[0072] In one embodiment, any of clusters 214A-214N of processing cluster array 212 can process data that is to be written into any of memory units 224A-224N within parallel processor memory 222. Memory crossbar 216 can be configured to transfer output of each cluster 214A-214N to any partition unit 220A-220N or another cluster 214A-214N, which can perform additional processing operations on the output. Each cluster 214A-214N can communicate with memory interface 218 via memory crossbar 216 to read from or write to various external memory devices. In one embodiment, memory crossbar 216 has a connection to memory interface 218 to communicate with I / O unit 204, as well as a connection to a local instance of parallel processor memory 222, enabling processing units within the different processing clusters 214A-214N to communicate with system memory or other memory not local to the parallel processor units 202. In one embodiment, memory crossbar 216 can use virtual channels to separate traffic streams between clusters 214A-214N and partition units 220A-220N.

[0073] Although a single instance of parallel processing unit 202 is shown within parallel processor 200, any number of instances of parallel processing unit 202 can be included. For example, a multiplicity of parallel processing units 202 can be provided on a single add-in card, or multiple add-in cards can be interconnected. Different instances of parallel processing unit 202 can be configured to operate interdependently, or can be configured to operate independently. Collective parallel processing can be used to perform processing on the different instances of parallel processing unit 202. For example, one or more instances of parallel processing unit 202 can be configured to perform a processing task, or tasks, while one or more other instances of parallel processing unit 202 can be configured to perform concurrent processing tasks. For example, different instances of parallel processing unit 202 can be configured to perform different tasks. In one embodiment, parallel processing unit 202 is configured to perform a graphics processing task, while one or more other instances of parallel processing unit 202 are configured to perform a general processing task. In one embodiment, parallel processing unit 202 is configured to perform a graphics processing task, while one or more other instances of parallel processing unit 202 are configured to perform a physics processing task.

[0074] Figure 2B is a block diagram of a partition unit 220 according to an embodiment. In one embodiment, partition unit 220 is an instance of one of partition units 220A-220N of FIG. 2A. As shown, partition unit 220 includes an L2 cache 221, a frame buffer interface 225, and an ROP 226 (raster operations unit). L2 cache 221 is a read / write cache that is configured to perform load and store operations received from memory crossbar 216 and ROP 226. Read misses and urgent write-back requests are output to frame buffer interface 225 for processing by L2 cache 221. Updates can also be sent to a frame buffer via frame buffer interface 225 for processing. In one embodiment, frame buffer interface 225 interfaces with one of memory units 224A-224N (e.g., within parallel processor memory 222) in parallel processor memory to perform load and store memory operations. Figure 2A

[0075] In graphics applications, ROP 226 is a processing unit that performs raster operations, such as stencil, z-test, blending, and the like. ROP 226 then outputs processed graphics data that is stored in graphics memory. In some embodiments, ROP 226 includes compression logic to compress depth or color data that is written to memory, and decompress depth or color data that is read from memory. The compression logic can be lossless compression logic that utilizes one or more of a variety of compression algorithms. The type of compression that is performed by ROP 226 can vary based on the statistical characteristics of the data to be compressed. For example, in one embodiment, delta color compression is performed on depth and color data on a per-tile basis.

[0076] ​In some embodiments, in each processing cluster (e.g., Figure 2A The clusters 214A-214N (roughly translated as clusters) instead of partition units 220 include ROP 226. In such embodiments, read and write requests for pixel data, rather than pixel fragment data, are transmitted via memory crossbar switches 216. The processed graphics data can be displayed on a display device (e.g., ...). Figure 1 Displayed on one or more display devices 110), routed for further processing by one or more processors 102, or routed for... Figure 2A One of the processing entities within the parallel processor 200 is used for further processing.

[0077] Figure 2C This is a block diagram of a processing cluster 214 within a parallel processing unit according to an embodiment. In one embodiment, the processing cluster is... Figure 2A An example of one of the processing clusters 214A-214N. Processing cluster 214 can be configured to execute many threads in parallel, wherein the term "thread" refers to an instance of a specific program executing on a particular set of input data. In some embodiments, a Single Instruction Multiple Data (SIMD) instruction dispatch technique is used to support the parallel execution of a large number of threads without providing multiple independent instruction units. In other embodiments, a Single Instruction Multiple Thread (SIMT) technique is used to support the parallel execution of a large number of generally synchronous threads using a common instruction unit configured to dispatch instructions to a set of processing engines within each of the processing cluster. Unlike the SIMD execution regime (where all processing engines typically execute the same instructions), SIMT execution allows different threads to more easily follow divergent execution paths through a given thread program. Those skilled in the art will understand that the SIMD processing regime represents a subset of the functionality of the SIMT processing regime.

[0078] The operation of the processing cluster 214 can be controlled via a pipeline manager 232, which distributes processing tasks to the SIMT parallel processors. The pipeline manager 232... Figure 2A The scheduler 210 receives instructions and manages the execution of those instructions via the graphics multiprocessor 234 and / or texture unit 236. The graphics multiprocessor 234 shown is an exemplary instance of a SIMT parallel processor. However, various types of SIMT parallel processors with different architectures may be included within the processing cluster 214. One or more instances of the graphics multiprocessor 234 may be included within the processing cluster 214. The graphics multiprocessor 234 can process data, and the data crossover switch 240 can be used to distribute the processed data to one of several possible destinations (including other shader units). The pipeline manager 232 can facilitate the distribution of processed data by specifying the destination for the processed data to be distributed via the data crossover switch 240.

[0079] Each graphics multiprocessor 234 within the processing cluster 214 may include the same set of functional execution logic (e.g., arithmetic logic units, load-memory units, etc.). The functional execution logic can be configured in a pipelined manner, allowing new instructions to be issued before previous instructions complete. The functional execution logic supports a variety of operations, including integer and floating-point arithmetic, comparison operations, Boolean operations, bit shifting, and computation of various algebraic functions. In one embodiment, the same functional unit hardware can be used to perform different operations, and any combination of functional units can exist.

[0080] Instructions sent to processing cluster 214 constitute threads. A group of threads executing across a set of parallel processing engines is a thread group. Thread groups execute the same program on different input data. Each thread within a thread group can be assigned to a different processing engine within graphics multiprocessor 234. A thread group may include fewer threads than the number of processing engines within graphics multiprocessor 234. When a thread group includes fewer threads than the number of processing engines, one or more of the processing engines may be idle during a loop in which the thread group is being processed. A thread group may also include more threads than the number of processing engines within graphics multiprocessor 234. When a thread group includes more threads than the number of processing engines within graphics multiprocessor 234, processing can be performed in a continuous clock cycle. In one embodiment, multiple thread groups can be executed concurrently on graphics multiprocessor 234.

[0081] In one embodiment, the graphics multiprocessor 234 includes an internal cache memory for performing load and store operations. In one embodiment, the graphics multiprocessor 234 may forgo the internal cache and use a cache memory within the processing cluster 214 (e.g., L1 cache 248). Each graphics multiprocessor 234 also has access to partition units that can be used to transfer data between threads and are shared across all processing clusters 214 (e.g., ...). Figure 2A The graphics multiprocessor 234 has an L2 cache within partition units 220A-220N. The graphics multiprocessor 234 can also access off-chip global memory, which may include one or more of local parallel processor memory and / or system memory. Any memory outside of the parallel processing unit 202 can be used as global memory. Embodiments (where processing cluster 214 includes multiple instances of the graphics multiprocessor 234) can share common instructions and data, which may be stored in L1 cache 248.

[0082] Each processing cluster 214 may include an MMU 245 (Memory Management Unit), which is configured to map virtual addresses to physical addresses. In other embodiments, one or more instances of the MMU 245 may reside on Figure 2A The memory interface 218 is located within the MMU 245. The MMU 245 includes: a set of page table entries (PTEs) for mapping virtual addresses to physical addresses of tiles; and optionally, a cache line index. The MMU 245 may include an address translation lookahead buffer (TLB) or cache that may reside within the graphics multiprocessor 234 or the L1 cache or processing cluster 214. Physical addresses are processed to distribute surface data access locality, thereby allowing efficient request interleaving within partitioned units. The cache line index can be used to determine whether a request for a cache line is a hit or a miss.

[0083] In graphics and computing applications, processing cluster 214 can be configured such that each graphics multiprocessor 234 is coupled to a texture unit 236 for performing texture mapping operations, such as determining texture sample locations, reading texture data, and filtering texture data. Texture data is read from an internal texture L1 cache (not shown) or, in some embodiments, from an L1 cache within the graphics multiprocessor 234, and fetched from an L2 cache, local parallel processor memory, or system memory, as needed. Each graphics multiprocessor 234 outputs a processed task to a data crossbar switch 240 to provide the processed task to another processing cluster 214 for further processing or to store the processed task in an L2 cache, local parallel processor memory, or system memory via a memory crossbar switch 216. A preROP (pre-raster operation unit) 242 is configured to receive data from the graphics multiprocessor 234 and direct the data to a ROP unit, which may be associated with partitioning units as described herein (e.g., Figure 2A The partition units 220A-220N are located together. The preROP 242 unit can perform optimizations for color blending, organize pixel color data, and perform address translation.

[0084] It will be appreciated that the core architecture described herein is illustrative, and variations and modifications are possible. Any number of processing units (e.g., graphics multiprocessor 234, texture unit 236, preROP 242, etc.) may be included within processing cluster 214. Furthermore, although only one processing cluster 214 is shown, the parallel processing units as described herein may include any number of instances of processing cluster 214. In one embodiment, each processing cluster 214 may be configured to operate independently of other processing clusters 214, using separate and distinct processing units, L1 caches, etc.

[0085] Figure 2DA graphics processor 234 according to one embodiment is shown. In such embodiments, the graphics processor 234 is coupled with the pipeline manager 232 of the processing cluster 214. The graphics processor 234 has an execution pipeline that includes, without limitation: an instruction cache 252; an instruction unit 254; an address mapping unit 256; a register file 258; one or more general-purpose graphics processing unit (GPGPU) cores 262; and one or more load / store units 266. The GPGPU cores 262 and load / store units 266 are coupled with a cache memory 272 and shared memory 270 via a memory and cache interconnect 268. In one embodiment, the graphics processor 234 additionally includes a tensor and / or ray tracing core 263 that includes hardware logic to accelerate matrix and / or ray tracing operations.

[0086] In one embodiment, the instruction cache 252 receives a stream of instructions to be executed from the pipeline manager 232. The instructions are cached in the instruction cache 252 and, once prepared for execution, are dispatched to the instruction unit 254 for execution. The instruction unit 254 can dispatch the instructions as thread groups (e.g., warps) of instructions, where each thread of the thread group is assigned to a different execution unit within the GPGPU cores 262. The instructions can access one of the local, shared, or global address spaces by specifying an address in the unified address space. The address mapping unit 256 can be used to translate the address in the unified address space to different memory addresses that can be accessed by the load / store units 266.

[0087] The register file 258 provides a set of registers to the functional units of the graphics processor 234. The register file 258 provides temporary storage for the data operands used by the data paths connected to the functional units (e.g., GPGPU cores 262, load / store units 266) of the graphics processor 234. In one embodiment, the register file 258 is divided between each of the functional units such that each functional unit is allocated a dedicated portion of the register file 258. In one embodiment, the register file 258 is divided between different thread warps executed by the graphics processor 234.

[0088] The GPGPU cores 262 can each include floating point, integer, and / or single instruction multiple data (SIMD) logic to perform operations in single precision, double precision, and / or integer precision, according to one embodiment. The SIMD logic can be used to perform a single or multiple arithmetic and / or logical operations on a set of data that is passed to the GPGPU cores 262. The GPGPU cores 262 can process the data at the same time with the SIMD logic operating in parallel across each piece of the data. The GPGPU cores 262 can be configured for a variety of implementations, including but not limited to applications in the field of graphics, physics, and computational science. In one embodiment, the GPGPU cores 262 include 64 single precision floating point units (FPUs) and 64 double precision floating point units. In one embodiment, FPU requests are handled by a posted switch architecture, which provides low latency to the memory. The GPGPU cores 262 are low latency to a cache / memory hierarchy.

[0089] In one embodiment, the GPGPU cores 262 include SIMD logic, additional SIMD logic can be included in the graphics processor 230. In one embodiment, the GPGPU cores 262 can physically execute SIMD4, SIMD8, and SIMD16 instructions and logically execute a SIMD1, SIMD2, and SIMD32 instructions via microcoding. The SIMD logic allows the GPGPU cores 262 to efficiently process the computation for multiple

[0090] The memory and cache interconnect 268 is an interconnection network that connects each of the functional units of the graphics multiprocessor 234 to the register file 258 and to the shared memory 270. In one embodiment, the memory and cache interconnect 268 is a crossbar switch interconnect that allows the load / store units 266 to implement load and store operations between the shared memory 270 and the register file 258. The register file 258 can operate at the same frequency as the GPGPU cores 262, such that data transfers between the GPGPU cores 262 and the register file 258 are very low latency. The shared memory 270 can be used to enable communication between threads executing on the functional units within the graphics multiprocessor 234. The cache memory 272 can be used, for example, as a data cache to cache texture data transferred between the functional units and the texture units 236. The shared memory 270 can also be used as a program managed cache. Threads executing on the GPGPU cores 262 can store data in the shared memory in a programmatic manner, in addition to data stored in the cache memory 272 that is automatically cached.

[0091] Figures 3A-3C An additional graphics multiprocessor according to an embodiment is shown. Figures 3A-3B Graphics multiprocessors 325, 350 are shown that are variations of the graphics multiprocessor 234 of Figure 2C FIG. 1. Figure 3C A graphics processing unit (GPU) 380 is shown that includes a dedicated set of graphics processing resources arranged into multi-core groups 365A-365N. The graphics multiprocessors 325, 350 and multi-core groups 365A-365N shown can be streaming multiprocessors (SMs) capable of executing a large number of execution threads in parallel.

[0092] Figure 3A A graphics multiprocessor 325 according to an additional embodiment is shown. The graphics multiprocessor 325 includes a number of additional instances of execution resource units involved in Figure 2D the graphics multiprocessor 234 of FIG. 1. For example, the graphics multiprocessor 325 can include multiple instances of instruction units 332A-332B, register files 334A-334B, and texture unit(s) 344A-344B. The graphics multiprocessor 325 also includes multiple groups of graphics or compute execution units (e.g., GPGPU cores 336A-336B, tensor cores 337A-337B, ray-tracing cores 338A-338B) and multiple groups of load / store units 340A-340B. In one embodiment, the execution resource units have a common instruction cache 330, texture and / or data cache memory 342, and shared memory 346.

[0093] The various components can communicate via an interconnect fabric 327. In one embodiment, the interconnect fabric 327 includes one or more crossbar switches to enable communication between the various components of the graphics processing unit 325. In one embodiment, the interconnect fabric 327 is a separate high-speed network fabric layer on which each component of the graphics processing unit 325 is stacked. The components of the graphics processing unit 325 communicate with remote components via the interconnect fabric 327. For example, the GPGPU cores 336A-336B, 337A-337B, and 338A-338B can each communicate with the shared memory 346 via the interconnect fabric 327. The interconnect fabric 327 can arbitrate communications between the components within the graphics processing unit 325 to ensure fair bandwidth allocation.

[0094] Figure 3B A graphics processing unit 350 is shown in accordance with an additional embodiment. The graphics processor includes multiple sets of execution resources 356A-356D, where each set of execution resources includes multiple instruction units, register file, GPGPU cores, and load store units as shown in Figure 2D and Figure 3A The execution resources 356A-356D can work in unison with the texture units 360A-360D to perform texture operations in conjunction with the shared instruction cache 354 and shared memory 353. In one embodiment, the execution resources 356A-356D can share the instruction cache 354 and shared memory 353, as well as multiple instances of the texture and / or data cache memory 358A-358B. The various components can communicate via an interconnect fabric 352 that is similar to the interconnect fabric 327 of Figure 3A .

[0095] Those of skill in the art will appreciate that the architectures described in Figure 1 , 2A 2D, and 3A-3B are descriptive and not limiting in the context of the present embodiments. Thus, the technology described herein can be implemented on any properly configured processing unit, including but not limited to one or more mobile application processors, one or more desktop computer or server central processing units (CPUs) (including multi-core CPUs), one or more parallel processing units such as the parallel processing unit 202 of Figure 2A , and one or more graphics processors or specialized processing units.

[0096] In some embodiments, a parallel processor or GPGPU as described herein is communicatively coupled to a host / processor core to accelerate graphics operations, machine-learning operations, pattern analysis operations, and various general purpose GPU (GPGPU) functions. The GPU can be communicatively coupled to the host processor / cores over a bus or other interconnect (e.g., a high-speed

[0097] Figure 3C A graphics processing unit (GPU) 380 is shown, which includes a set of specialized graphics processing resources arranged into multiple core groups 365A-N. While only the details of a single multiple core group 365A are provided, it will be recognized that the other multiple core groups 365B-365N can be equipped with the same or similar sets of graphics processing resources.

[0098] As shown, the multiple core group 365A can include a group of graphics cores 370, a group of tensor cores 371, and a group of ray-tracing cores 372. A scheduler / dispatcher 368 schedules and dispatches graphics threads for execution on the various cores 370, 371, 372. A set of register files 369 stores operand values for use by the cores 370, 371, 372 when executing graphics threads. These can include, for example, integer registers for storing integer values, floating point registers for storing floating point values, vector registers for storing packed data elements (integer and / or floating point data elements), and tile registers for storing tensor / matrix values. In one embodiment, the tile registers are implemented as a combined set of vector registers.

[0099] One or more combined Level 1 (LI) cache and shared memory units 373 locally store graphics data such as texture data, vertex data, pixel data, ray data, bounding volume data, etc. within each multi-core group 365A. One or more texture units 374 can also be used to perform texture operations, such as texture mapping and sampling. A Level 2 (L2) cache 375, which is shared by all or a subset of the multi-core groups 365A-365N, stores graphics data and / or instructions for a number of concurrent graphics threads. As shown, the L2 cache 375 can be shared across multiple multi-core groups 365A-365N. One or more memory controllers 367 couple the GPU 380 to memory 366, which can be system memory (e.g., DRAM) and / or dedicated graphics memory (e.g., GDDR6 memory).

[0100] Input / output (I / O) circuitry 363 couples the GPU 380 to one or more I / O devices 362, such as a digital signal processor (DSP), a network controller, or user input devices. An on-chip interconnect can be used to couple the I / O devices 362 to the GPU 380 and the memory 366. One or more I / O memory management units (IOMMU) 364 of the I / O circuitry 3195 directly couple the I / O devices 362 to the system memory 366. In one embodiment, the IOMMU 364 manages multiple sets of page tables to map virtual addresses to physical addresses in the system memory 366. In that embodiment, the I / O devices 362, the CPU(s) 361, and the GPU(s) 380 can share the same virtual address space.

[0101] In one implementation, the IOMMU 364 supports virtualization. In that case, it can manage a first set of page tables to map guest / graphics virtual addresses to guest / graphics physical addresses, and a second set of page tables to map guest / graphics physical addresses to system / host physical addresses (e.g., in the system memory 366). The base addresses of each of the first and second sets of page tables can be stored in control registers and swapped out on context switch (e.g., to provide a new context with access to the relevant set of page tables). Although not shown in FIG. 3B, the cores 370, 371, 372, and / or each of the multi-core groups 365A-365N can include translation lookaside buffers (TLBs) for caching guest virtual to guest physical translations, guest physical to host physical translations, and guest virtual to host physical translations. Figure 3C

[0102] ​In one embodiment, CPU 361, GPU 380, and I / O devices 362 are integrated on a single semiconductor chip and / or chip package. The illustrated memory 366 can be integrated on the same chip, or can be coupled to memory controller 367 via an off-chip interface. In one implementation, memory 366 comprises GDDR6 memory that shares the same virtual address space as other physical system level memory, although the underlying principles of the application are not limited to this particular implementation.

[0103] In one embodiment, tensor core 371 comprises a plurality of execution units specifically designed to perform matrix operations that are the basic computational operations for performing deep learning operations. For example, simultaneous matrix multiplication operations can be used for neural network training and inference. Tensor core 371 can perform matrix processing using a variety of operand precisions, including single precision floating point (e.g., 32 bits), half precision floating point (e.g., 16 bits), integer (16 bits), byte (8 bits), and half byte (4 bits). In one embodiment, a neural network implementation extracts features of each rendered scene, potentially combining details from multiple frames to construct a high quality final image.

[0104] In deep learning implementations, parallel matrix multiplication work can be scheduled for execution on tensor core 371. Training of neural networks in particular requires a large number of matrix dot product operations. To handle the inner product formula for N x N x N matrix multiplication, tensor core 371 can contain at least N dot product processing elements. Before matrix multiplication begins, a full matrix is loaded into a tile register, and for each of N cycles, at least one column of a second matrix is loaded. For each cycle, N dot products are processed.

[0105] Depending on the particular implementation, matrix elements can be stored with different precisions, including 16-bit words, 8-bit bytes (e.g., INT8), and 4-bit half-bytes (e.g., INT4). Different precision modes can be specified for tensor core 371 to ensure that the most efficient precision is used for different workloads (e.g., such as inference workloads that can tolerate quantization to bytes and half-bytes).

[0106] In one embodiment, the ray tracing core 372 accelerates ray tracing operations for both real-time ray tracing and non-real-time ray tracing implementations. In particular, the ray tracing core 372 includes ray traversal / intersection circuitry to perform ray traversal using a bounding volume hierarchy (BVH) and to identify intersections between a ray and primitives enclosed within the BVH volumes. The ray tracing core 372 can also include circuitry to perform depth testing and culling (e.g., using a Z-buffer or similar arrangement). In one implementation, the ray tracing core 372 cooperates with the image denoising techniques described herein to perform traversal and intersection operations, at least a portion of which can be performed on the tensor core 371. For example, in one embodiment, the tensor core 371 implements a deep learning neural network to perform denoising on frames generated by the ray tracing core 372. However, the CPU(s) 361, graphics core 370, and / or ray tracing core 372 can also implement all or a portion of the denoising and / or deep learning algorithms.

[0107] Additionally, as described above, a distributed approach can be employed for denoising in which the GPU 380 is located in a computing device that is coupled to other computing devices over a network or high-speed interconnect. In this embodiment, the interconnected computing devices share neural network learning / training data to improve the speed at which the overall system learns to perform denoising on different types of image frames and / or different graphics applications.

[0108] In one embodiment, the ray tracing core 372 handles all BVH traversal and ray primitive intersection, avoiding overloading the graphics core 370 with thousands of instructions per ray. In one embodiment, each ray tracing core 372 includes a first set of specialized circuitry to perform bounding box tests (e.g., for traversal operations) and a second set of specialized circuitry to perform ray-triangle intersection tests (e.g., for intersected rays that have been traversed). Thus, in one embodiment, the multi-core group 365A can simply launch the ray probes, and the ray tracing core 372 independently performs the ray traversal and intersection, returning hit data (e.g., hit, no hit, multiple hits, etc.) to the thread context. The other cores 370, 371 are freed up to perform other graphics or compute work while the ray tracing core 372 performs the traversal and intersection operations.

[0109] In one embodiment, each ray tracing core 372 includes a traversal unit to perform BVH test operations and an intersection unit to perform ray primitive intersection tests. The intersection unit generates a “hit,” “no hit,” or “multiple hit” response and provides it to the appropriate thread. During the traversal and intersection operations, the execution resources of the other cores (e.g., graphics core 370 and tensor core 371) are freed up to perform other forms of graphics work.

[0110] In one particular embodiment described below, a hybrid rasterization / ray tracing approach is used, with work distributed between graphics cores 370 and ray tracing cores 372.

[0111] In one embodiment, ray tracing cores 372 (and / or other cores 370, 371) include hardware support for a ray tracing instruction set, such as Microsoft’s DirectX Raytracing (DXR), which includes a DispatchRays command as well as ray generation, closest hit, any hit, and miss shaders that enable assigning a unique set of shaders and textures to each object. Another ray tracing platform that ray tracing cores 372, graphics cores 370, and tensor cores 371 can support is Vulkan 1.1.85. Note, however, that the underlying principles of the present application are not limited to any particular ray tracing ISA.

[0112] In general, various cores 372, 371, 370 can support a ray tracing instruction set that includes instructions / functions for ray generation, closest hit, any hit, ray-primitive intersection, per-primitive and level bound box construction, miss, access, and exceptions. More specifically, one embodiment includes ray tracing instructions for performing the following functions:

[0113] Ray generation — Ray generation instructions can be executed for each pixel, sample, or other user-defined work assignment.

[0114] Recent hits — Closest hit instructions can be executed to locate the closest intersection of a ray and a primitive within a scene.

[0115] Any hits — Any hit instructions identify multiple intersections between a ray and primitives within a scene to potentially identify a new closest intersection.

[0116] Intersections — Intersection instructions perform ray-primitive intersection tests and output results.

[0117] Primitive bounding box construction — This instruction constructs a bounding box around a given primitive or group of primitives (e.g., when constructing a new BVH or other acceleration data structure).

[0118] Misses — Indicates that a ray missed all geometry within a scene or a specified region of a scene.

[0119] Accesses — Indicates a sub-volume that a ray will traverse.

[0120] Techniques for GPU interconnection with host processors — Includes various types of exception handlers (e.g., invoked for various error conditions).

[0121] Figure 4A

[0122] Figure 4A An exemplary architecture is shown in which multiple GPUs 410-413 are communicatively coupled to multiple multi-core processors 405-406 over high-speed links 440A-440D (e.g., buses, point-to-point interconnects, etc.). In one embodiment, high-speed links 440A-440D support communication throughput of 4GB / s, 30GB / s, 80GB / s or higher, depending on implementation. Various interconnect protocols can be used including, but not limited to, PCIe 4.0 or 5.0 and NVLink 2.0. However, the underlying principles of the application are not limited to any particular communication protocol or throughput.

[0123] In addition, in one embodiment, two or more of GPUs 410-413 are interconnected over high-speed links 442A-442B, which can be implemented using the same or different protocols / links than those used for high-speed links 440A-440D. Similarly, two or more of multi-core processors 405-406 can be connected by high-speed link 443, which can be a Symmetric Multi-Processor (SMP) bus operating at 20GB / s, 30GB / s, 120GB / s or higher. Alternatively, Figure 4B All communication between the various system components shown in FIG. 4A can be achieved using the same protocol / link (e.g., over a common interconnect fabric). However, as mentioned, the underlying principles of the application are not limited to any particular type of interconnect technology.

[0124] In one embodiment, each multi-core processor 405-406 is communicatively coupled with processor memory 401-402 via memory interconnects 430A-430B, respectively, and each GPU 410-413 is communicatively coupled with GPU memory 420-423 via GPU memory interconnects 450A-450D, respectively. Memory interconnects 430A-430B and 450A-450D can utilize the same or different memory access technologies. By way of non-limiting example, processor memory 401-402 and GPU memory 420-423 can be volatile memories such as dynamic random access memory (DRAM) including stacked DRAM, graphics DDR SDRAM (GDDR) (e.g., GDDR5, GDDR6), or high-bandwidth memory (HBM), and / or can be non-volatile memories such as 3D XPoint or Nano-Ram. In one embodiment, some portion of the memory can be volatile memory and another portion can be non-volatile memory (e.g., using a two-level memory (2LM) hierarchy).

[0125] As described below, while the various processors 405-406 and GPUs 410-413 can be physically coupled to specific memories 401-402, 420-423, respectively, a unified memory architecture can be implemented in which the same virtual system address space (also referred to as "effective address" space) is distributed across all of the various physical memories. For example, the processor memories 401-402 can each include 64 GB of system memory address space, and the GPU memories 420-423 can each include 32 GB of system memory address space (resulting in a total of 256 GB of addressable memory in this example).

[0126] Figure 4C Additional details of the interconnection between the multi-core processor 407 and the graphics acceleration module 446 are shown. The graphics acceleration module 446 can include one or more GPU chips integrated on a line card that is coupled with the processor 407 via the high-speed link 440. Alternatively, the graphics acceleration module 446 can be integrated on the same package or chip as the processor 407.

[0127] The processor 407 shown includes multiple cores 460A-460D, each with a translation lookaside buffer 461A-461D and one or more caches 462A-462D. The cores can include various other components for executing instructions and processing data not shown, such as instruction fetch units, branch prediction units, decoders, execution units, reorder buffers, etc., in order to avoid obscuring the essential elements of the application. The caches 462A-462D can include level one (LI) and level two (L2) caches. Additionally, one or more shared caches 456 can be included in the cache hierarchy and shared by the sets of cores 460A-460D. For example, one embodiment of the processor 407 includes 24 cores each with its own LI cache, twelve shared L2 caches, and a twelve shared L3 cache. In this embodiment, one of the L2 and L3 caches is shared by two adjacent cores. The processor 407 and graphics accelerator integration module 446 are connected with system memory 441, which can include the processor memories 401-402.

[0128] Consistency for data and instructions stored in the various caches 462A-462D, 456, and system memory 441 is maintained via inter-core communication over the coherence bus 464. For example, each cache can have cache coherency logic / circuitry associated therewith to communicate over the coherence bus 464 in response to a detected read or write for a particular cache line. In one implementation, a snoop protocol is implemented over the coherence bus 464 to snoop cache accesses. Cache snoop / coherency techniques are well understood by those skilled in the art, and will not be described in detail here to avoid obscuring the underlying principles of the application.

[0129] In one embodiment, the agent circuit 425 communicatively couples the graphics acceleration module 446 to the coherence bus 464, allowing the graphics acceleration module 446 to participate in a cache coherence protocol as a peer to the cores. Specifically, the interface 435 provides connectivity to the agent circuit 425 over a high-speed link 440 (e.g., a PCIe bus, NVLink, etc.), and the interface 437 connects the graphics acceleration module 446 to the high-speed link 440.

[0130] In one implementation, the accelerator integration circuit 436 provides the cache management, memory access, context management, and interrupt management services on behalf of multiple graphics processing engines 431, 432, N of the graphics acceleration module 446. The graphics processing engines 431, 432, N can each comprise a separate graphics processing unit (GPU). Alternatively, the graphics processing engines 431, 432, N can comprise different types of graphics processing engines within a GPU such as graphics execution units, media processing engines (e.g., video encoders / decoders), samplers, and bitblitting engines. In other words, the graphics acceleration module can be a GPU with a plurality of graphics processing engines 431-432, N or the graphics processing engines 431-432, N can be individual GPUs integrated on a common package, line card, or chip.

[0131] In one embodiment, the accelerator integration circuit 436 includes a memory management unit (MMU) 439 to provide translation of virtual addresses into physical addresses and / or to provide memory protection services. In one embodiment, the MMU 439 includes address translation lookaside buffers (TLBs) to improve translation speed. In one embodiment, the MMU 439 includes memory protection circuits to control access to various memory structures.

[0132] A set of registers 445 store context data for threads executed by the graphics processing engines 431-432, N, and a context management circuit 448 manages thread contexts. For example, the context management circuit 448 can perform save and restore operations to save and restore the context of various threads during context switches (e.g., where a first thread is saved and a second thread is stored so that the second thread can be executed by the graphics processing engines). For example, upon a context switch, the context management circuit 448 can store current register values into an assigned region of memory (e.g., identified by a context pointer). It can then restore those register values upon a return to the context. In one embodiment, an interrupt management circuit 447 receives and processes interrupts received from system devices.

[0133] In one implementation, virtual / effective addresses from the graphics processing engines 431 are translated to real / physical addresses in system memory 411 by the MMU 439. One embodiment of the accelerator integration circuit 436 supports multiple (e.g., 4, 8, 16) graphics accelerator modules 446 and / or other accelerator devices. The graphics accelerator modules 446 can be dedicated to a single application executing on the processor 407, or can be shared between multiple applications. In one embodiment, a virtualized graphics execution environment is presented where resources of the graphics processing engines 431-432, N are shared with multiple applications or virtual machines (VMs). The resources can be subdivided into “slices” that are allocated to different VMs and / or applications based on processing requirements and priorities associated with the VMs and / or applications.

[0134] Thus, the accelerator integration circuit functions as a bridge to the system for the graphics acceleration module 446 and provides address translation and system memory cache services. In addition, the accelerator integration circuit 436 can provide virtualization facilities for the host processor to manage virtualization of the graphics processing engines, interrupts, and memory management.

[0135] Because the hardware resources of the graphics processing engines 431-432, N are explicitly mapped to the real address space seen by the host processor 407, any host processor can use effective address values to directly address these resources. In one embodiment, one function of the accelerator integration circuit 436 is the physical separation of the graphics processing engines 431-432, N so that they appear as independent units to the system.

[0136] As noted, in the illustrated embodiment, one or more graphics memories 433-434, M are coupled to each of the graphics processing engines 431-432, N, respectively. The graphics memories 433-434, M store instructions and data being processed by each of the graphics processing engines 431-432, N. The graphics memories 433-434, M can be volatile memory such as DRAM (including stacked DRAM), GDDR memory (e.g., GDDR5, GDDR6), or HBM, and / or can be non-volatile memory such as 3D XPoint or Nano-Ram.

[0137] In one embodiment, to reduce data traffic on the high-speed link 440, a biasing technique is used to ensure that data stored in the graphics memories 433-434, M is that which will be most frequently used by the graphics processing engines 431-432, N and preferably not used by the cores 460A-460D (at least not frequently). Similarly, the biasing mechanism attempts to keep data required by the cores (and preferably not the graphics processing engines 431-432, N) within the system memory 411 and the caches 462A-462D, 456 of the cores.

[0138] Figure 4B Another embodiment is shown in which the accelerator integration circuit 436 is integrated within the processor 407. In this embodiment, the graphics processing engines 431-432, N communicate directly over the high-speed link 440 to the accelerator integration circuit 436 via the interface 437 and the interface 435 (which again can utilize any form of bus or interface protocol). The accelerator integration circuit 436 can execute same operations as those described with respect to the accelerator integration circuit 436, but potentially at higher throughput given its close proximity to the coherence bus 464 and the caches 462A-462D, 456. Figure 4D The described operations of those operations, but potentially at higher throughput given its close proximity to the coherence bus 464 and the caches 462A-462D, 456.

[0139] One embodiment supports different programming models, including a dedicated process programming model (without graphics acceleration module virtualization) and a shared programming model (with virtualization). The latter can include a programming model controlled by the accelerator integration circuit 436 and a programming model controlled by the graphics acceleration module 446.

[0140] In one embodiment of the dedicated process model, the graphics processing engines 431-432, N are dedicated to a single application or process under a single operating system. The single application can funnel other application requests to the graphics engines 431-432, N, providing virtualization within the VM / partition.

[0141] In the dedicated process programming model, the graphics processing engines 431-432, N can be shared by multiple VM / application partitions. The shared model requires a hypervisor to virtualize the graphics processing engines 431-432, N to allow access by each operating system. For hypervisor-less, single-partition systems, the graphics processing engines 431-432, N are owned by the operating system. In both cases, the operating system can virtualize the graphics processing engines 431-432, N to provide access to each process or application.

[0142] For the shared programming model, the graphics acceleration module 446 or individual graphics processing engines 431-432, N use a process handle to select a process element. In one embodiment, the process element is stored in system memory 441 and is addressable using the effective-to-real address translation techniques described herein. The process handle can be an implementation-specific value provided to the host process when it registers its context with the graphics processing engines 431-432, N (that is, calls system software to add the process element to a process element linked list). The lower 16 bits of the process handle can be an offset into the process element linked list of the process element.

[0143] Process and thread identification An exemplary accelerator integration slice 490 is shown. As used herein, a "slice" includes a specified portion of the processing resources of the accelerator integration circuit 436. An application effective address space 482 within system memory 441 stores process elements 483. In one embodiment, the process elements 483 are stored in response to GPU invocations 481 from an application 480 executing on the processor 407. The process elements 483 contain process state for the corresponding application 480. A work descriptor (WD) 484 contained in the process element 483 can be a single job requested by the application, or can contain a pointer to a queue of jobs. In the latter case, the WD 484 is a pointer to the application's address space 482 request queue of jobs.

[0144] Graphics acceleration module 446 and / or individual graphics processing engines 431-432, N can be shared by all or a subset of the processes in the system. Embodiments of the application include infrastructure for setting up process state and sending a WD 484 to graphics acceleration module 446 to start a job in a virtualized environment.

[0145] In one implementation, the dedicated process programming model is implementation specific. In this model, a single process owns graphics acceleration module 446 or an individual graphics processing engine 431. As graphics acceleration module 446 is owned by a single process, the hypervisor initializes accelerator integration circuit 436 for the owning partition and the operating system initializes accelerator integration circuit 436 for the owning process at the time graphics acceleration module 446 is assigned.

[0146] In operation, WD fetch unit 491 in accelerator integration slice 490 fetches the next WD 484, which includes an indication of work to be done by one of the graphics processing engines of graphics acceleration module 446. Data from WD 484 can be stored in registers 445 and used by MMU 439, interrupt management circuit 447, and / or context management circuit 448 as shown. For example, one embodiment of MMU 439 includes segment / page walk circuitry for accessing segment / page tables 486 within OS virtual address space 485. Interrupt management circuit 447 can handle interrupt events 492 received from graphics acceleration module 446. When performing graphics operations, effective addresses 493 generated by graphics processing engines 431-432, N are translated to real addresses by MMU 439.

[0147] In one embodiment, the same set of registers 445 is replicated for each graphics processing engine 431-432, N and / or graphics acceleration module 446, and the set of registers 445 can be initialized by the hypervisor or operating system. Each of these replicated registers can be included in accelerator integration slice 490. Exemplary registers that can be initialized by the hypervisor are shown in Table 1.

[0148] Table 1 - Hypervisor-Initialized Registers

[0149]

[0150]

[0151] Exemplary registers that can be initialized by the operating system are shown in Table 2.

[0152] 1 Effective address (EA) context save / restore pointer 2 Virtual address (VA) accelerator utilization record pointer 3 Virtual address (VA) storage segment table pointer 4 Authority mask 5 Work descriptor 6 Figure 4E

[0153] In one embodiment, each WD 484 is specific to a particular graphics acceleration module 446 and / or graphics processing engines 431-432, N. It contains all information the graphics processing engines 431-432, N require to complete its work, or it can be a pointer to a memory location where a command queue, to which the application has established work to be done, resides.

[0154] Work descriptor (WD) Additional details are shown for one embodiment of a shared model. This embodiment includes a hypervisor real address space 498 in which a list of process elements 499 is stored. The hypervisor real address space 498 is accessible via a hypervisor 496 that virtualizes the graphics acceleration module engines for an operating system 495.

[0155] The shared programming model allows all processes or a subset of processes from all partitions or a subset of partitions in the system to use the graphics acceleration module 446. There are two programming models in which the graphics acceleration module 446 is shared by multiple processes and partitions: time-sliced sharing and graphics directed sharing.

[0156] In this model, the system hypervisor 496 owns the graphics acceleration module 446 and makes its functionality available to all operating systems 495. For the graphics acceleration module 446 to support virtualization by the system hypervisor 496, the graphics acceleration module 446 can adhere to the following requirements: 1) Application job requests must be autonomous (that is, no state needs to be maintained between jobs), or the graphics acceleration module 446 must provide a context save and restore mechanism. 2) The graphics acceleration module 446 guarantees that an application's job request is completed within a specified amount of time (including any translation faults), or the graphics acceleration module 446 provides the ability to preempt processing of a job. 3) The graphics acceleration module 446 must guarantee fairness between processes when operating in the directed shared programming model.

[0157] In one embodiment, for a shared model, application 480 is required to make an operating system 495 system call with a graphics acceleration module 446 type, a work descriptor (WD), an authority mask register (AMR) value, and a context save / restore area pointer (CSRP). The graphics acceleration module 446 type describes the targeted acceleration function for the system call. The graphics acceleration module 446 type can be a system specific value. The WD is formatted specifically for the graphics acceleration module 446 and can take the form of a graphics acceleration module 446 command, an effective address pointer to a user defined structure, an effective address pointer to a command queue, or any other data structure used to describe work to be done by the graphics acceleration module 446. In one embodiment, the AMR value is the AMR state to be used for the current process. The value passed to the operating system is similar to the application setting the AMR. If the accelerator integration circuit 436 and graphics acceleration module 446 implementation does not support a user authority mask override register (UAMOR), the operating system can apply the current UAMOR value to the AMR value before passing the AMR in the hypervisor call. Optionally, the hypervisor 496 can apply the current authority mask override register (AMOR) value before placing the AMR in the process element 483. In one embodiment, the CSRP is one of the registers 445 that contains an effective address of a region in the application's address space 482 to be used by the graphics acceleration module 446 to save and restore context state. This pointer is optional if saving state between jobs or when a job is preempted is not required. The context save / restore region can be pinned system memory.

[0158] Upon receiving the system call, the operating system 495 can verify that the application 480 is registered and has been given authority to use the graphics acceleration module 446. The operating system 495 then calls the hypervisor 496 with the information shown in Table 3.

[0159] 1 Authority mask register (AMR) value (potentially masked) 2 Effective address (EA) context save / restore area pointer (CSRP) 3 Process ID (PID) and optional thread ID (TID) 4 Virtual address (VA) accelerator utilization record pointer (AURP) 5 Virtual address of storage segment table pointer (SSTP) 6 Logical interrupt service number (LISN) 7 Work descriptor (WD)

[0160] Upon receiving the hypervisor call, the hypervisor 496 verifies that the operating system 495 is registered and has been given authority to use the graphics acceleration module 446. The hypervisor 496 then places the process element 483 into a process element linked list for the corresponding graphics acceleration module 446 type. The process element can include the information shown in Table 4.

[0161] 1 Authority mask register (AMR) value (potentially masked) 2 Effective address (EA) context save / restore area pointer (CSRP) 3 Process ID (PID) and optional thread ID (TID) 4 Virtual address (VA) accelerator utilization record pointer (AURP) 5 Virtual address of storage segment table pointer (SSTP) 6 Logical interrupt service number (LISN) 7 Interrupt vector table derived from hypervisor invocation parameters 8 State register (SR) value 9 Logical partition ID (LPID) 10 Real address (RA) hypervisor accelerator utilization record pointer 11 Storage device descriptor register (SDR) 12 Figure 4F

[0162] In one embodiment, the hypervisor initializes a number of accelerator integration slice 490 registers 445.

[0163] As Figure 4FAs shown in FIG. 4, one embodiment of the present application employs unified memory that is addressable via a common virtual memory address space for accessing physical processor memory 401-402 and GPU memory 420-423. In such an implementation, operations performed on GPUs 410-413 utilize the same virtual / effective memory address space for accessing processor memory 401-402 and vice versa, thereby simplifying programmability. In one embodiment, a first portion of the virtual / effective address space is allocated to processor memory 401, a second portion is allocated to second processor memory 402, a third portion is allocated to GPU memory 420, etc. The entire virtual / effective memory space (sometimes referred to as the effective address space) is thereby distributed across each of processor memory 401-402 and GPU memory 420-423, allowing any processor or GPU to access any physical memory (using a virtual address that is mapped to that memory).

[0164] In one embodiment, bias / coherency management circuitry 494A-494E within one or more of MMUs 439A-439E ensures cache coherency between host processors (e.g., 405) and caches of GPUs 410-413, and implements biasing techniques that indicate the physical memory in which certain types of data should be stored. Although Graphics processing pipeline Multiple instances of bias / coherency management circuitry 494A-494E are shown in FIG. 4, but bias / coherency circuitry can be implemented within MMUs of one or more host processors 405 and / or within accelerator integration circuit 436.

[0165] One embodiment allows for the use of shared virtual memory (SVM) technology to access GPU-attached memory 420-423 and map it as part of system memory without incurring the typical performance penalties associated with full system cache coherency. The ability to access GPU-attached memory 420-423 as system memory without the heavy cache coherency overhead provides a beneficial operating environment for GPU offload. This arrangement allows host processor 405 software to set operands and access computation results without the overhead of traditional I / O DMA data copies. Such traditional copies involve driver calls, interrupts, and memory mapped I / O (MMIO) accesses, all of which are inefficient relative to simple memory accesses. At the same time, the ability to access GPU-attached memory 420-423 without cache coherency overhead can be critical to the execution time of the offloaded computation. In cases with substantial streaming write memory traffic, for example, the cache coherency overhead can significantly reduce the effective write bandwidth seen by GPU 410-413. The efficiency of operand setup, the efficiency of result access, and the efficiency of GPU computation all play a role in determining the effectiveness of GPU offload.

[0166] In one implementation, the selection between GPU bias and host processor bias is driven by a bias tracker data structure. For example, a bias table can be used, which can be a page-granularity structure (i.e., controlled at the granularity of a memory page) that includes 1 or 2 bits per GPU-attached memory page. The bias table can be implemented in a stolen memory range of one or more GPU-attached memories 420-423, with or without a bias cache in GPU 410-413 (e.g., to cache frequently / recently used entries of the bias table). Alternatively, the entire bias table can be maintained within the GPU.

[0167] In one implementation, the bias table entry associated with each access to GPU-attached memory 420-423 is accessed prior to the actual access to GPU memory, causing the following operations. First, local requests from GPU 410-413 that look for their page in the GPU bias are forwarded directly to the corresponding GPU memory 420-423. Local requests from the GPU that look for their page in the host bias are forwarded to processor 405 (e.g., over a high-speed link as discussed above). In one embodiment, requests from processor 405 that look for the requested page in the host processor bias complete the request as a normal memory read. Alternatively, requests for pages in the GPU bias can be forwarded to GPU 410-413. The GPU can then transition the page to the host processor bias if the GPU is not currently using the page.

[0168] The bias state of a page can be changed by a software-based mechanism, a hardware- assisted software-based mechanism, or a purely hardware-based mechanism for a limited set of cases.

[0169] One mechanism for changing the bias state employs an API call (e.g., OpenCL), which in turn invokes a device driver of the GPU, which in turn sends a message (or enqueues a command descriptor) to the GPU, instructing it to change the bias state and, for some transitions, to perform a cache flushing operation in the host. The cache flushing operation is needed for biasing transitions from the host processor 405 to the GPU bias, but not for the reverse transition.

[0170] In one embodiment, cache coherency is maintained by temporarily rendering GPU-biased pages that are not cacheable by the host processor 405. To access these pages, the processor 405 can request access from the GPU 410, which can or can not grant access immediately (depending on the implementation). Thus, to reduce communication between the host processor 405 and the GPU 410, it is advantageous to ensure that the GPU-biased pages are those that are required by the GPU but not by the host processor 405 (and vice versa).

[0171] Figure 5

[0172] Figure 2A A graphics processing pipeline 500 is shown in accordance with an embodiment. In one embodiment, a graphics processor can implement the graphics processing pipeline 500 shown. The graphics processor can be included within a parallel processing subsystem (such as parallel processor 200 of FIG. 1) as described herein, which in one embodiment is a variation of parallel processor(s) 112 of FIG. 1. Various parallel processing systems can implement the graphics processing pipeline 500 via one or more instances of a parallel processing unit (e.g., parallel processing unit 202 of FIG. 2) as described herein. For instance, a shader unit (e.g., graphics multiprocessor 234 of FIG. 2) can be configured to perform the functions of one or more of a vertex processing unit 504, a tessellation control processing unit 508, a tessellation evaluation processing unit 512, a geometry processing unit 516, and a fragment / pixel processing unit 524. The functions of a data assembler 502, primitive assemblers (506, 514, 518), a tessellation unit 510, a rasterizer 522, and a raster operations unit 526 can also be performed by other processing engines and corresponding partition units within a processing cluster (e.g., processing cluster 214 of FIG. 2). Figure 1 Figure 2A Figure 2C Figure 2A Figure 2A Figure 2A ​​​​​by the partitioning unit 220A-220N of the parallel processor 200). The graphics processing pipeline 500 can also be implemented using dedicated processing units for one or more functions. In one embodiment, one or more parts of the graphics processing pipeline 500 can be performed by parallel processing logic within a general purpose processor (e.g., a CPU). Figure 2A In one embodiment, the parallel processor 200 includes multiple processing units 220A-220N. In one embodiment, the processing units 220A-220N are connected to a ring interconnect 230. In one embodiment, the ring interconnect 230 is a high-speed interconnect that is used by the processing units 220A-220N to communicate with each other. Figure 2A In one embodiment, the ring interconnect 230 is used by the processing units 220A-220N to communicate with each other, e.g., to pass messages, coordinates of data, and so on. In one embodiment, the ring interconnect 230 is used by the processing units 220A-220N to communicate with the memory interface 218 of the parallel processor 200. In one embodiment, multiple processing units 220A-220N can communicate with the memory interface 218 at the same time. In one embodiment, the ring interconnect 230 is used by all

[0173] In one embodiment, the data assembler 502 is a processing unit that collects vertex data for surfaces and primitives. The data assembler 502 then outputs vertex data including vertex attributes to the vertex processing unit 504. The vertex processing unit 504 is a programmable execution unit that is configured to illuminate and transform vertex data as specified by a vertex shader program. The vertex processing unit 504 reads data stored in caches, local or system memory to perform these operations. The vertex processing unit 504 can be programmed to transform the vertex data from an object-based coordinate representation (e.g., a model space) to the world space coordinate space or the normalized device coordinate space as specified by the vertex shader program.

[0174] A first instance of the primitive assembler 506 receives vertex attributes from the vertex processing unit 504. The primitive assembler 506 reads stored vertex attributes as needed, and constructs graphics primitives for processing by the tessellation control processing unit 508. The graphics primitives include triangles, lines, points, patches, and so on, as supported by various graphics processing application programming interfaces (APIs).

[0175] The tessellation control processing unit 508 treats the input vertices as control points for a geometric patch. The control points are transformed from an input representation (e.g., a basis for the patch) from the patch to a representation suitable for use by the tessellation evaluation processing unit 512 in surface evaluations. The tessellation control processing unit 508 can also compute tessellation factors for edges of the geometric patch. The tessellation factors apply to individual edges and quantify a level of detail for a dependent view associated with that edge. The tessellation unit 510 is configured to receive tessellation factors for edges of the patch, and tessellate the patch into a plurality of geometric primitives such as line, triangle, or quadrilateral primitives, which are passed to the tessellation evaluation processing unit 512. The tessellation evaluation processing unit 512 operates on parametric coordinates of the tessellated patch to generate surface representations and vertex attributes for each vertex associated with the geometric primitives.

[0176] A second instance of primitive assembler 514 receives vertex attributes from tessellation evaluation processing unit 512 (which reads stored vertex attributes as needed) and constructs graphics primitives for processing by geometry processing unit 516. Geometry processing unit 516 is a programmable execution unit that executes a geometry shader program to transform the graphics primitives received from primitive assembler 514 as specified by the geometry shader program. In one embodiment, geometry processing unit 516 is programmed to subdivide the graphics primitives into one or more new graphics primitives and to compute parameters for rasterization of the new graphics primitives.

[0177] In some embodiments, geometry processing unit 516 can add or remove elements in the geometry stream. Geometry processing unit 516 outputs the parameters and vertices that specify the new graphics primitives to primitive assembler 518. Primitive assembler 518 receives the parameters and vertices from geometry processing unit 516 and constructs graphics primitives for processing by viewport scale, cull, and clip unit 520. Geometry processing unit 516 reads data stored in parallel processor memory or system memory for use while processing the geometry data. Viewport scale, cull, and clip unit 520 performs clipping, culling, and viewport scaling and outputs the processed graphics primitives to rasterizer 522.

[0178] Rasterizer 522 can perform depth culling and other depth-based optimizations. Rasterizer 522 also performs scan conversion on the new graphics primitives to generate fragments and outputs those fragments and associated coverage data to fragment / pixel processing units 524. Fragment / pixel processing units 524 are programmable execution units configured to perform a fragment or pixel shader program. Fragment / pixel processing units 524 transform the fragments or pixels received from rasterizer 522 as specified by the fragment or pixel shader program. For example, fragment / pixel processing units 524 can be programmed to perform operations including, but not limited to, texture mapping, shading, blending, texture correction, and perspective correction to produce shaded fragments or pixels that are output to raster operations unit 526. Fragment / pixel processing units 524 can read data stored in parallel processor memory or system memory for use while processing the fragment data. The fragment or pixel shader programs can be configured to color at a sample, pixel, tile, or other granularity depending on the sampling rate configured for the processing units.

[0179] Raster operations unit 526 is a processing unit that performs raster operations including, but not limited to, stencil operations, z-test operations, blending, and the like and outputs pixel data as processed graphics data to be stored in graphics memory (e.g., parallel processor memory 222 as in Figure 1 Machine learning overview ​The system memory 104 in the computer system 100 can include volatile, nonvolatile, removable, and / or non-removable storage components, such as computer readable storage media 106. Computer readable storage media 106 can be used for storage of information and instructions for use by processors 102 and parallel processor(s) 112. Computer readable storage media 106 can be implemented using any appropriate media, including storage devices and supplies, or elements related thereto. The computer readable storage media 106 can include, but is not limited to, number of storage elements, including, for example, physical memory, virtual memory, cache, drive storage, and / or removable media. In some embodiments, the raster operations unit 526 is configured to compress z or color data being written to memory and decompress z or color data being read from memory.

[0180] Figure 6

[0181] The architecture described above can be applied to use machine learning models to perform training and inference operations. Machine learning has successfully solved a wide variety of tasks. The computations that arise in training and using machine learning algorithms, such as neural networks, lend themselves naturally to efficient parallel implementations. As a result, parallel processors such as general purpose graphics processing units (GPGPUs) have played an important role in the practical implementation of deep neural networks. Parallel graphics processors with single instruction multiple thread (SIMT) architectures are designed to maximize the amount of parallel processing in a graphics pipeline. In a SIMT architecture, groups of parallel threads attempt to execute program instructions together as often as possible to improve processing efficiency. The efficiency provided by parallel machine learning algorithm implementations allows the use of high-capacity networks and enables those networks to be trained on larger datasets.

[0182] Machine learning algorithms are algorithms that can learn based on a dataset. Embodiments of machine learning algorithms can be designed to model high-level abstractions within a dataset. For example, an image recognition algorithm can be used to determine which of several categories a given input belongs to; a regression algorithm can output a numerical value given an input; and a pattern recognition algorithm can be used to generate transcribed text or perform text-to-speech and / or speech recognition.

[0183] An exemplary type of machine learning algorithm is a neural network. There are many types of neural networks; a simple type of neural network is a feedforward network. A feedforward network can be implemented as an acyclic graph in which nodes are arranged in layers. Typically, a feedforward network topology includes an input layer and an output layer separated by at least one hidden layer. The hidden layer transforms the input received by the input layer into a representation that is useful in generating output in the output layer. The network nodes are fully connected via edges to nodes in adjacent layers, but there are no edges between nodes within each layer. Data received at the nodes of the input layer of a feedforward network is propagated (i.e., "fed forward") to the nodes of the output layer via an activation function that computes the state of each successive layer of nodes in the network based on coefficients ("weights") respectively associated with each of the edges connecting the layers. The output from a neural network algorithm can take a variety of forms depending on the particular model represented by the algorithm being executed.

[0184] Before a machine learning algorithm can be used to model a particular problem, the algorithm is trained using a training data set. Training a neural network involves selecting a network topology, using a set of training data representing the problem being modeled by the network, and adjusting the weights until the network model performs with minimal error on all instances of the training data set. For example, during a supervised learning training process for a neural network, the output produced by the network in response to an instance in the training data set is compared to the "correct" labeled output for that instance, an error signal representing the difference between the output and the labeled output is computed, and the weights associated with the connections are adjusted to minimize the error as the error signal propagates backwards through the network layers. When the error is minimized for each of the outputs generated from instances of the training data set, the network is considered "trained."

[0185] The accuracy of a machine learning algorithm can be significantly affected by the quality of the data set used to train the algorithm. The training process can be computationally intensive and can require a significant amount of time on a conventional general purpose processor. Therefore, parallel processing hardware is used to train many types of machine learning algorithms. This is particularly useful for optimizing the training of neural networks because the computations performed in adjusting the coefficients in a neural network lend themselves naturally to parallel implementation. In particular, many machine learning algorithms and software applications have been adapted to take advantage of the parallel processing hardware within a general purpose graphics processing device.

[0186] GPGPU machine learning acceleration is a general diagram of the machine learning software stack 600. The machine learning application 602 can be configured to train a neural network using a training data set or to implement machine intelligence using a trained deep neural network. The machine learning application 602 can include training and inference functionality for a neural network and / or can be used to train a neural network prior to deployment. The machine learning application 602 can implement any type of machine intelligence including, but not limited to, image recognition, mapping and localization, autonomous navigation, speech synthesis, medical imaging, or language translation.

[0187] Hardware acceleration for machine learning applications 602 can be enabled via a machine learning framework 604. The machine learning framework 604 can provide a library of machine learning primitives. A machine learning primitive is a basic operation that is commonly performed by machine learning algorithms. Without the machine learning framework 604, a developer of a machine learning algorithm would need to create and optimize the host computing logic associated with the machine learning algorithm, and then re-optimize the computing logic when new parallel processors are developed. Instead, a machine learning application can be configured to perform the necessary computations using the primitives provided by the machine learning framework 604. Exemplary primitives include tensor convolutions, activation functions, and pooling, which are computational operations performed when training a convolutional neural network (CNN). The machine learning framework 604 can also provide primitives to implement basic linear algebra subprograms, such as matrix and vector operations, that are performed by many machine learning algorithms.

[0188] The machine learning framework 604 can process input data received from the machine learning applications 602 and generate appropriate inputs to a compute framework 606. The compute framework 606 can abstract the underlying instructions provided to the GPGPU driver 608 to enable the machine learning framework 604 to utilize hardware acceleration via the GPGPU hardware 610 without the machine learning framework 604 needing to have in-depth knowledge of the architecture of the GPGPU hardware 610. Additionally, the compute framework 606 can enable the machine learning framework 604 to implement hardware acceleration across various types and generations of GPGPU hardware 610.

[0189] Figure 7

[0190] Figure 4A A general purpose graphics processing unit 700 according to an embodiment is shown. In one embodiment, the general purpose processing unit (GPGPU) 700 can be configured to be particularly efficient in processing the type of computational workloads associated with training deep neural networks. Additionally, the GPGPU 700 can be directly linked to other instances of GPGPUs to create a multi-GPU cluster to improve training speed for particular deep neural networks.

[0191] The GPGPU 700 includes a host interface 702 to enable connection to a host processor. In one embodiment, the host interface 702 is a PCI Express interface. However, the host interface can also be a vendor specific communications interface or communications fabric. The GPGPU 700 receives commands from the host processor and uses a global scheduler 704 to distribute execution threads associated with those commands to a set of compute clusters 706A-706H. The compute clusters 706A-706H share a cache memory 708. The cache memory 708 can act as a higher level cache for cache memories within the compute clusters 706A-706H.

[0192] GPGPU 700 includes memory 714A-B that is coupled with the compute clusters 706A-H via a set of memory controllers 712A-B. In various embodiments, the memory 714A-714B can include various types of memory devices including dynamic random access memory (DRAM) or graphics random access memory, such as synchronous graphics random access memory (SGRAM), including graphics double data rate (GDDR) memory. In one embodiment, the memory units 714A-714N can also include 3D stacked memory, including but not limited to high bandwidth memory (HBM).

[0193] In one embodiment, each compute cluster 706A-706H includes a group of graphics processing units, such as the graphics processing units 400 of FIG. 4. Figure 8 The graphics processing units of a compute cluster include multiple types of integer and floating point logic units that can perform computational operations over a range of precisions including those suitable for machine learning computations. For example, and in one embodiment, at least a subset of the floating point units in each of the compute clusters 706A-H can be configured to perform 16-bit or 32-bit floating point operations, while a different subset of the floating point units can be configured to perform 64-bit floating point operations.

[0194] Multiple instances of the GPGPU 700 can be configured to operate as a compute cluster. The communication mechanism that the compute cluster uses for synchronization and data exchange varies by embodiment. In one embodiment, multiple instances of the GPGPU 700 communicate through the host interface 702. In one embodiment, the GPGPU 700 includes an I / O hub 709 that couples the GPGPU 700 with a GPU link 710 that enables a direct connection to other instances of the GPGPU. In one embodiment, the GPU link 710 is coupled to a specialized GPU-to-GPU bridge that enables communication and synchronization between multiple instances of the GPGPU 700. In one embodiment, the GPU link 710 is coupled with a high speed interconnect to transmit and receive data to and from other GPGPUs or parallel processors. In one embodiment, multiple instances of the GPGPU 700 are located in separate data processing systems and communicate via a network device that is accessible via the host interface 702. In one embodiment, the GPU link 710 can be configured to enable connection with a host processor in addition to or as an alternative to the host interface 702.

[0195] While the configuration of GPGPU 700 shown can be configured to train neural networks, one embodiment provides an alternative configuration of GPGPU 700 that can be configured for deployment within a high performance or low power inference platform. In an inference configuration, GPGPU 700 includes fewer compute clusters 706A-706H relative to a training configuration. Additionally, the memory technology associated with memory 714A-714B can differ between inference and training configurations. In one embodiment, the inference configuration of GPGPU 700 can support inference-specific instructions. For example, the inference configuration can provide support for one or more 8-bit integer dot product instructions that are typically used during inference operations for a deployed neural network.

[0196] Figure 7 A multi-GPU computing system 800 according to an embodiment is shown. The multi-GPU computing system 800 can include a processor 802 coupled to a plurality of GPGPUs 806A-806D via a host interface switch 804. In one embodiment, the host interface switch 804 is a PCI Express switch device that couples the processor 802 to a PCI Express bus over which the processor 802 can communicate with a set of GPGPUs 806A-806D. Each of the plurality of GPGPUs 806A-806D can be an instance of the GPGPU 700 of Figure 7 The GPGPUs 806A-806D can be interconnected via a set of high-speed point-to- point GPU-to-GPU links 816. The high-speed GPU-to-GPU links can connect to each of the GPGPUs 806A-806D via a dedicated GPU link such as the GPU link 710 in Machine learning neural network implementation The P2P GPU links 816 enable direct communication between each of the GPGPUs 806A-806D without communicating over the host interface bus to which the processor 802 is connected. With GPU-to-GPU traffic directed to the P2P GPU links, the host interface bus can still be used for system memory access or to communicate with other instances of the multi-GPU computing system 800, e.g., via one or more network devices. While in the illustrated embodiment the GPGPUs 806A-D are connected to the processor 802 via the host interface switch 804, in one embodiment the processor 802 includes direct support for the P2P GPU links 816 and can be directly connected to the GPGPUs 806A-806D.

[0197] Figures 9A-9BThe computing architecture provided by the embodiments described herein can be configured to perform the kind of parallel processing that is particularly well suited for training and deploying neural networks for machine learning. Neural networks can be summarized as networks of functions having graph relationships. As is well known in the art, there are multiple types of neural networks that are used in machine learning. One exemplary type of neural network is a feedforward network, as previously described.

[0198] A second exemplary type of neural network is a convolutional neural network (CNN). CNNs are specialized feedforward neural networks for processing data having a known grid-like topology, such as image data. Thus, CNNs are commonly used for computing vision and image recognition applications, but they can also be used for other types of pattern recognition, such as speech and language processing. Nodes in the input layer of a CNN are organized into sets of "filters" (feature detectors inspired by receptive fields found in the retina), and the output of each set of filters is propagated to nodes in successive layers of the network. The computation for a CNN includes applying a convolution mathematical operation to each filter to produce the output of that filter. Convolution is a specialized mathematical operation performed by two functions to produce a third function that is a modified version of one of the two original functions. In convolution network terminology, the first function to the convolution can be called the input, and the second function can be called the convolution kernel. The output can be called the feature map. For example, the input to a convolution layer can be data that defines a multi-dimensional array of various color components of an input image. The convolution kernel can be a multi-dimensional array of parameters that are adapted for the neural network by a training process.

[0199] A recurrent neural network (RNN) is a series of feedforward neural networks that includes feedback connections between layers. RNNs enable modeling of sequential data by sharing parameter data across different parts of the neural network. The architecture of an RNN includes loops. The loops represent the influence of the current value of a variable on its own value at a future time because at least a portion of the output data from the RNN is used as feedback for processing subsequent input in the sequence. This feature makes RNNs particularly useful for language processing due to the variable nature of the data that makes up language.

[0200] The diagrams described below present exemplary feedforward, CNN, and RNN networks, as well as describe general processes for training and deploying each of those types of networks, respectively. It will be understood that these descriptions are exemplary and non-limiting with respect to any particular embodiment described herein and that the concepts illustrated can be applied generally to deep neural networks and machine learning techniques in general.

[0201] The exemplary neural networks described above can be used to perform deep learning. Deep learning is a type of machine learning that uses deep neural networks. Deep neural networks used in deep learning are artificial neural networks composed of multiple hidden layers, as opposed to shallow neural networks that include only a single hidden layer. Deeper neural networks are typically more computationally intensive to train. However, the additional hidden layers of the network enable multi-step pattern recognition, which results in a reduction in output error relative to shallow machine learning techniques.

[0202] Deep neural networks used in deep learning typically include a front-end network coupled to a back-end network for performing feature recognition, with the back-end network representing a mathematical model that can perform operations (e.g., object classification, speech recognition, etc.) based on feature representations provided to the model. Deep learning enables machine learning to be performed without the need for hand-engineered features to be performed on the model. Instead, the deep neural network can learn features based on statistical structures or correlations within the input data. The learned features can be provided to a mathematical model, which can map the detected features to an output. The mathematical model used by the network is generally specific to the particular task to be performed, and different models will be used to perform different tasks.

[0203] Once a neural network is constructed, a learning model can be applied to the network to train the network to perform a particular task. The learning model describes how to adjust the weights within the model to reduce the output error of the network. Error backpropagation is a common method used to train neural networks. An input vector is presented to the network for processing. The output of the network is compared to a desired output using a loss function and an error value is computed for each of the neurons in the output layer. The error values are then propagated backwards until each neuron has an associated error value that roughly represents its contribution to the original output. The network can then learn from those errors using an algorithm, such as a stochastic gradient descent algorithm, to update the weights of the neural network.

[0204] Figure 9A An exemplary convolutional neural network is shown. Figure 9A Various layers within a CNN are shown. As in Figure 9BAn exemplary CNN used to model image processing, as shown, can receive an input 902 that describes the red, green, and blue (RGB) components of an input image. The input 902 can be processed by a number of convolutional layers, such as convolutional layer 904, convolutional layer 906. The output from the number of convolutional layers can optionally be processed by a set of fully connected layers 908. Neurons in the fully connected layers have full connections to all activations in the previous layers, as described previously for feedforward networks. The output from the fully connected layers 908 can be used to generate an output result from the network. The activations within the fully connected layers 908 can be computed using matrix multiplication rather than convolution. Not all CNN implementations utilize the fully connected layers 908. For example, in some implementations, the convolutional layer 906 is able to generate an output for the CNN.

[0205] Convolutional layers are sparsely connected, unlike the traditional neural network configuration found in the fully connected layers 908. Traditional neural network layers are fully connected, such that each output unit interacts with every input unit. However, as shown, because the output of a convolution of a field is the input to the nodes of a subsequent layer (rather than a respective state value for each of the nodes in the field), convolutional layers are sparsely connected. The kernel associated with a convolutional layer performs a convolution operation, the output of which is sent to the next layer. The dimensionality reduction performed within a convolutional layer is one aspect that enables the CNN to scale to process large images.

[0206] Figure 9A An exemplary computation stage within a convolutional layer of a CNN is shown. The input to a convolutional layer 912 of a CNN can be processed in three stages of a convolutional layer 914. The three stages can include a convolution stage 916, a detector stage 918, and a pooling stage 920. The convolutional layer 914 can then output data to a successive convolutional layer. The last convolutional layer of a network can generate output feature map data or provide input to a fully connected layer, such as to generate a classification value to an input to the CNN.

[0207] In the convolution stage 916, several convolutions are performed in parallel to produce a set of linear activations. The convolution stage 916 can include an affine transformation, which is any transformation that can be specified as a linear transformation plus a translation. Affine transformations include rotations, translations, scaling, and combinations of these transformations. The convolution stage computes the output of a function (e.g., a neuron) connected to a particular region in the input, which can be determined as a local region associated with the neuron. The neuron computes a dot product between the weights of the neuron and the region in the local input to which the neuron is connected. The output from the convolution stage 916 defines a set of linear activations that are processed by successive stages of the convolutional layer 914.

[0208] Linear activations can be processed by the detector stage 918. In the detector stage 918, each linear activation is processed by a non-linear activation function. The non-linear activation function increases the non-linear nature of the overall network without affecting the respective fields of the convolutional layers. Several types of non-linear activation functions can be used. One particular type is a rectified linear unit (ReLU) that uses an activation function defined as f(x) = max(0, x) such that the activation threshold is zero.

[0209] The pooling stage 920 uses a pooling function that replaces the output of the convolutional layer 906 with summary statistics of nearby outputs. Pooling functions can be used to introduce translational invariance into the neural network such that small translations of the input do not change the pooling output. Invariance to local translations can be useful in scenarios where the presence of a feature in the input data is more important than the exact location of the feature. Various types of pooling functions can be used during the pooling stage 920, including max pooling, average pooling, and l2 norm pooling. Additionally, some CNN implementations do not include a pooling stage. Instead, such implementations replace the pooling stage with additional convolutional stages that have an increased stride relative to the previous convolutional stages.

[0210] The output from the convolutional layer 914 can then be processed by the next layer 922. The next layer 922 can be one of an additional convolutional layer or a fully connected layer 908. For example, Figure 10 The first convolutional layer 904 can output to the second convolutional layer 906, and the second convolutional layer can output to the first layer of the fully connected layer 908.

[0211] Figure 11 An exemplary recurrent neural network 1000 is shown. In a recurrent neural network (RNN), the previous state of the network influences the output of the current state of the network. RNNs can be constructed in various ways using various functions. The use of RNNs generally revolves around using mathematical models to make predictions based on prior sequences of inputs. For example, given a sequence of previous words, an RNN can be used to perform statistical language modeling to predict an upcoming word. The RNN 1000 shown can be described as having an input layer 1002 that receives an input vector, a hidden layer 1004 that implements a recurrent function, a feedback mechanism 1005 that enables'memory' of previous states, and an output layer 1006 that outputs a result. The RNN 1000 operates based on time steps. The state of the RNN at a given time step is influenced based on previous time steps via the feedback mechanism 1005. For a given time step, the state of the hidden layer 1004 is defined by the previous state and the input of the current time step. An initial input (x1) at a first time step can be processed by the hidden layer 1004. A second input (x2) can be processed by the hidden layer 1004 using state information determined during processing of the initial input (x1). The given state can be computed as s t = f(Ux t+Ws t-1 ), where U and W are parameter matrices. The function f is generally non-linear, such as a hyperbolic tangent function (Tanh) or a variant of the rectified function f(x) = max(0, x). However, the particular mathematical function used in the hidden layer 1004 can vary depending on the particular implementation details of the RNN 1000.

[0212] In addition to the basic CNN and RNN networks described, variations on these networks can be enabled. One example RNN variation is a long short-term memory (LSTM) RNN. LSTM RNNs are capable of learning long-term dependencies, which are necessary for processing longer language sequences. A variation on CNNs is a convolutional deep belief network, which has a similar structure to a CNN and is trained in a similar manner to a deep belief network. A deep belief network (DBN) is a generative neural network composed of multiple layers of probabilistic (stochastic) variables. A DBN can be trained layer by layer using greedy unsupervised learning. The learned weights of a DBN can then be used to provide a pre-trained neural network by determining an optimal initial set of weights for the neural network.

[0213] Figure 6 Training and deployment of deep neural networks is shown. Once a given network has been constructed for a task, the neural network is trained using a training data set 1102. Various training frameworks 1104 have been developed to enable hardware acceleration of the training process. For example, Figure 12 The machine learning framework 604 can be configured as a training framework 604. The training framework 604 can hook into an untrained neural network 1106 and enable the untrained neural network to be trained using the parallel processing resources described herein to generate a trained neural network 1108.

[0214] To begin the training process, the initial weights can be selected randomly or by using a pre-training of a deep belief network. The training loop is then performed in a supervised or unsupervised manner.

[0215] Supervised learning is a learning method in which training is performed as a mediated operation, such as when the training dataset 1102 includes inputs paired with desired outputs for the inputs, or in cases where the training dataset includes inputs with known outputs and the output of the neural network is graded by a human. The network processes the input and the resulting output is compared to a set of expected or desired outputs. Errors are then propagated backwards through the system. The training framework 1104 can adjust to adjust the weights that control the untrained neural network 1106. The training framework 1104 can provide tools to monitor how well the untrained neural network 1106 is converging to a model suitable for generating correct responses based on known input data. The training process repeats as the weights of the network are adjusted to improve the output generated by the neural network. The training process can continue until the neural network reaches a statistically expected level of accuracy associated with the trained neural network 1108. The trained neural network 1108 can then be deployed to implement any number of machine learning operations to generate inference results 1114 based on input of new data 1112.

[0216] Unsupervised learning is a learning method in which the network attempts to train itself using unlabeled data. Thus, for unsupervised learning, the training dataset 1102 will include input data without any associated output data. The untrained neural network 1106 can learn groupings within the unlabeled input and can determine how individual inputs relate to the overall dataset. Unsupervised training can be used to generate self-organizing maps, which are a type of trained neural network 1108 that can perform operations useful in reducing the dimensionality of data. Unsupervised training can also be used to perform anomaly detection, which allows for the identification of data points in an input dataset that deviate from the normal patterns of the data.

[0217] Variations on supervised and unsupervised training can also be employed. Semi-supervised learning is a technique in which the training dataset 1102 includes a mix of labeled and unlabeled data of the same distribution. Incremental learning is a variation of supervised learning in which input data is continuously used to further train the model. Incremental learning enables the trained neural network 1108 to adapt to new data 1112 without forgetting the knowledge instilled in the network during initial training.

[0218] Whether supervised or unsupervised, the training process for, in particular, deep neural networks can be computationally too intensive for a single computing node. Instead of using a single computing node, a distributed network of computing nodes can be used to speed up the training process.

[0219] Figure 7is a block diagram illustrating distributed learning. Distributed learning is a training model that uses multiple distributed computing nodes to perform supervised or unsupervised training of a neural network. The distributed computing nodes can each include one or more host processors and one or more of general purpose processing nodes, such as, for example, high parallel general purpose graphics processing units 700 in Exemplary machine learning applications As illustrated, distributed learning can be model parallel structure 1202, data parallel structure 1204, or a combination of model and data parallel structure 1204 performed.

[0220] In model parallel structure 1202, different computing nodes in a distributed system can perform training computations on different parts of a single network. For example, each layer of a neural network can be trained by a different processing node of a distributed system. Benefits of model parallel structure include the ability to scale to especially large models. Splitting the computations associated with different layers of a neural network enables training of this very large neural network in which the weights of all layers would not fit in the memory of a single computing node. In some instances, model parallel structure can be particularly useful in performing unsupervised training of large neural networks.

[0221] In data parallel structure 1204, different nodes of a distributed network have a complete instance of a model and each node receives a different portion of data. Results from different nodes are then combined. While different approaches are possible for data parallel structure, data parallel training approaches all require techniques to combine results and synchronize model parameters between each node. Exemplary approaches to combining data include parameter averaging and update-based data parallel structure. Parameter averaging trains each node on a subset of training data and sets global parameters (e.g., weights, biases) to the average of the parameters from each node. Parameter averaging uses a central parameter server, which maintains parameter data. Update-based data parallel structure is similar to parameter averaging, except that updates to the model are transmitted, rather than transmitting parameters from nodes to a parameter server. Additionally, update-based data parallel structure can employ a decentralized approach, in which updates are compressed and transmitted between nodes.

[0222] Combination model and data parallel structure 1206 can be implemented, for example, in a distributed system in which each computing node includes multiple GPUs. Each node can have a complete instance of a model, with independent GPUs within each node used to train different parts of the model.

[0223] Distributed training has increased overhead relative to training on a single machine. However, the parallel processors and GPGPUs described herein can each implement various techniques to reduce the overhead of distributed training, including techniques to implement high-bandwidth GPU-to-GPU data transfers and to accelerate remote data synchronization.

[0224] Figure 7

[0225] Machine learning can be applied to solve a variety of technical problems, including but not limited to computer vision, autonomous driving and navigation, speech recognition and language processing. Traditionally, computer vision has been one of the most active areas of research for machine learning applications. Applications of computer vision range from replicating human vision capabilities (e.g., recognizing faces) to creating new classes of vision capabilities. For example, a computer vision application can be configured to recognize sound waves from vibrations induced in objects visible in a video. Parallel processor-accelerated machine learning enables computer vision applications to train using significantly larger training datasets than previously feasible and enables inference systems to be deployed using low-power parallel processors.

[0226] Parallel processor-accelerated machine learning has applications in autonomous driving, including lane and road sign recognition, obstacle avoidance, navigation, and driving control. Accelerated machine learning techniques can be used to train driving models based on datasets defining appropriate responses to particular training inputs. Parallel processors described herein can enable fast training of increasingly complex neural networks for autonomous driving solutions and enable deployment of low-power inference processors in mobile platforms suitable for integration into autonomous vehicles.

[0227] Parallel processor-accelerated deep neural networks enable machine learning approaches for automatic speech recognition (ASR). ASR includes creating a function that, given an input acoustic sequence, computes the most likely sequence of words. Accelerated machine learning using deep neural networks has enabled replacement of hidden Markov models (HMMs) and Gaussian mixture models (GMMs) previously used for ASR.

[0228] Parallel processor-accelerated machine learning can also be used to accelerate natural language processing. Automated learning procedures can leverage statistical inference algorithms to produce models that are robust to errors or unfamiliar input. An exemplary natural language processor application includes automatic machine translation between human languages.

[0229] Parallel processing platforms for machine learning can be divided into training platforms and deployment platforms. Training platforms are generally highly parallel and include optimizations to accelerate multi-GPU single-node training and multi-node multi-GPU training. Exemplary parallel processors suitable for training include Figure 8 NVIDIA®'s general-purpose graphics processing unit 700 and Figure 13 NVIDIA®'s multi-GPU computing system 800. In contrast, deployed machine learning platforms generally include lower-power parallel processors suitable for use in products such as cameras, autonomous robots, and autonomous vehicles.

[0230] Additional exemplary graphics processing systemsAn exemplary inference system on a chip (SOC) 1300 suitable for performing inference using trained models is shown. The SOC 1300 can integrate processing components including a media processor 1302, a vision processor 1304, a GPGPU 1306, and a multi-core processor 1308. The SOC 1300 can additionally include on-chip memory 1305, which can implement a shared on-chip data pool accessible by each of the processing components. The processing components can be optimized for low-power operation to enable deployment to various machine learning platforms, including autonomous vehicles and autonomous robots. For example, one implementation of the SOC 1300 can be used as part of a main control system of an autonomous vehicle. Where the SOC 1300 is configured for use in an autonomous vehicle, the SOC is designed and configured to comply with relevant functional safety standards of the jurisdiction of deployment.

[0231] During operation, the media processor 1302 and the vision processor 1304 can work in concert to accelerate computer vision operations. The media processor 1302 can implement low-latency decoding of multiple high-resolution (e.g., 4K, 8K) video streams. The decoded video streams can be written to a buffer in the on-chip memory 1305. The vision processor 1304 can then parse the decoded video and perform preliminary processing operations on frames of the decoded video in preparation for processing the frames using trained image recognition models. For example, the vision processor 1304 can accelerate convolution operations for a CNN used to perform image recognition on high-resolution video data, while backend model computations are performed by the GPGPU 1306.

[0232] The multi-core processor 1308 can include control logic to help sequence and synchronize data transfers and shared memory operations performed by the media processor 1302 and the vision processor 1304. The multi-core processor 1308 can also act as an application processor to execute software applications that can leverage the inferencing computational capabilities of the GPGPU 1306. For example, at least a portion of navigation and driving logic can be implemented in software that executes on the multi-core processor 1308. Such software can issue computational workloads directly to the GPGPU 1306 or can issue computational workloads to the multi-core processor 1308, which can offload at least a portion of those operations to the GPGPU 1306.

[0233] The GPGPU 1306 can include compute clusters, such as the low-power configuration of the compute clusters 706A-706H within the general-purpose graphics processing unit 700. The compute clusters within the GPGPU 1306 can support instructions that are specifically optimized to perform inferencing computations on trained neural networks. For example, the GPGPU 1306 can support instructions that perform low-precision computations such as 8-bit and 4-bit integer vector operations.

[0234] Figure 14

[0235] Details of the embodiments described above can be incorporated into the graphics processing systems and devices described below. Figure 14 The graphics processing systems and devices of Figure 26 illustrate alternative systems and graphics processing hardware that can implement any and all of the techniques described above.

[0236] Figure 15 is a block diagram of a processing system 1400 according to an embodiment. The system 1400 can be used in a single-processor desktop system, a multiprocessor workstation system, or a server system having many processors 1402 or processor cores 1407. In one embodiment, the system 1400 is incorporated in a

[0237] In one embodiment, the system 1400 can include, be coupled to, or be integrated within a server-based gaming platform, a game console including a game and media console, a mobile gaming console, a handheld game console, or an online game console. In some embodiments, the system 1400 is part of a mobile telephone, a smart phone, a tablet computing device, or a mobile Internet-connected device such as a laptop that has low internal storage capacity. The processing system 1400 can also include, be coupled to, or be integrated within a wearable device such as a smart watch wearable device, smart eyewear or clothing that is augmented by augmented reality (AR) or virtual reality (VR) features to provide visual, audible, or tactile output to supplement real-world visual, audible, or tactile experiences, or otherwise provide text, audio, graphics, video, holographic images or video, or tactile feedback; other augmented reality (AR) devices; or other virtual reality (VR) devices. In some embodiments, the processing system 1400 includes or is part of a television or set-top box device.

[0238] In some embodiments, the system 1400 can include, be coupled to, or be integrated within an autonomous vehicle such as a bus, tractor trailer, car, motorcycle or electric bicycle, airplane or glider (or any combination thereof). The autonomous vehicle can use the system 1400 to process the environment sensed around the vehicle.

[0239] In some embodiments, one or more processors 1402 each include one or more processor cores 1407 to process instructions which, when executed, implement operations for system and user software. In some embodiments, at least one of the one or more processor cores 1407 is configured to process a specific instruction set 1409. In some embodiments, instruction set 1409 can facilitate complex instruction set computing (CISC), reduced instruction set computing (RISC), or computing via a very long instruction word (VLIW). One or more processor cores 1407 can process a different instruction set 1409, which can include instructions to facilitate the emulation of other instruction sets. Processor core 1407 can also include other processing devices, such as a digital signal processor (DSP).

[0240] In some embodiments, processor 1402 includes cache memory 1404. Depending upon the configuration, processor 1402 can have a single internal cache or multiple levels of internal cache. In some embodiments, cache memory is shared among various components of processor 1402. In some embodiments, processor 1402 also uses an external cache (e.g., a level three (L3) cache or last level cache (LLC)) (not shown), which can be shared among processor cores 1407 using known cache coherency techniques. A register file 1406 is also included in processor 1402 and can include different types of registers to store different kinds of data (e.g., integer registers, floating point registers, status registers, and instruction pointer registers). Some registers can be general registers, while other registers can be specific to processor 1402 design.

[0241] In some embodiments, one or more processors 1402 are coupled with one or more interface buses 1410 to transmit communications signals such as address, data, or control signals between processor 1402 and other components in system 1400. In one embodiment, interface bus 1410 can be a processor bus, such as a version of the direct media interface (DMI) bus. However, processor bus is not limited to DMI bus, but can include one or more peripheral component interconnect buses (e.g., a PCI, PCI Express), memory buses, or other types of interface buses. In one embodiment, (one or more) processor(s) 1402 include an integrated memory controller 1416 and platform controller hub 1430. Memory controller 1416 facilitates communication over a memory bus to memory device and to other components of system 1400, while platform controller hub 1430 provides connections to input / output devices via a local I / O bus.

[0242] The memory device 1420 can be a dynamic random access memory (DRAM) device, a static random access memory (SRAM) device, a flash memory device, a phase- change memory device, or some other memory device having suitable performance to serve as process memory. In one embodiment, the memory device 1420 can operate as system memory for the computing system 1400 to store data 1422 and instructions 1421 for use when the one or more processors 1402 execute an application or process. The memory controller 1416 also couples with an optional external graphics processor 1418, which can communicate with the one or more graphics processors 1408 in the processor(s) 1402 to perform graphics and media operations. In some embodiments, the graphics, media, and computing operations can be assisted or conducted by an accelerator 1412, which is a co-processor that can be configured to perform a specialized set of graphics, media, or compute operations. For example, in one embodiment, the accelerator 1412 is a matrix multiplication accelerator for optimizing machine learning or compute operations. In one embodiment, the accelerator 1412 is a ray-tracing accelerator that can be used in conjunction with the graphics processor 1408 to perform ray-tracing operations. In some embodiments, a display device 1411 can be connected to the processor(s) 1402. The display device 1411 can be an integrated display device, such as in a mobile electronic device or a laptop device, or an external display device attached via a display interface (e.g., DisplayPort, etc.). In one embodiment, the display device 1411 can be a head-mounted display (HMD) such as a stereoscopic display device for virtual reality (VR) applications or augmented reality (AR) applications.

[0243] In some embodiments, platform controller hub 1430 enables peripherals to connect to memory device 1420 and processor 1402 via a high-speed I / O bus. The I / O peripherals include, but are not limited to, audio controller 1446, network controller 1434, firmware interface 1428, wireless transceiver 1426, touch sensors 1425, data storage device 1424 (e.g., non-volatile memory, volatile memory, hard disk drive, flash memory, NAND, 3D NAND, 3D XPoint, etc.). Data storage device 1424 can connect to the storage device interface via a storage interface bus (e.g., SATA), or via a peripheral bus such as a peripheral component interconnect bus (e.g., PCI, PCI Express). Touch sensors 1425 can include touch screen sensors, pressure sensors, or fingerprint sensors. Wireless transceiver 1426 can be a Wi-Fi transceiver, a Bluetooth transceiver, or a mobile network transceiver such as a 3G, 4G, 5G, or Long-Term Evolution (LTE) transceiver. Firmware interface 1428 enables communication with system firmware, and can be, for example, a unified extensible firmware interface (UEFI). Network controller 1434 can enable network connectivity to a wired network. In some embodiments, a high-performance network controller (not shown) is coupled

[0244] It will be appreciated that the system 1400 shown is exemplary and not limiting in that other types of data processing systems that have more or fewer components can be used. For example, external graphics processor 1418 can be integrated into processor 1402. In one embodiment, platform controller hub 1430 and / or memory controller 1416 can be external to one or more processor(s) 1402. For example, system 1400 can include an external memory controller 1416 and platform controller hub 1430 that can be configured as a memory controller hub and platform controller hub in a system-on-a-chip (SoC) that also includes one or more processor(s) 1402.

[0245] For example, a circuit board ("sled") can be used to hold components such as CPUs, memory, and other components, which are designed to improve thermal performance. In some examples, processing components such as processors are located on the top of the sled, while nearby memory such as DIMMs are located on the bottom of the sled. As a result of the enhanced airflow provided by such a design, the components can operate at higher frequency and power levels than in typical systems, improving performance. In addition, the sled is configured to blindly mate with power and data communication cables in a rack, enhancing their ability to be quickly disassembled, upgraded, reinstalled, and / or replaced. Similarly, individual components located on the sled, such as processors, accelerators, memory, and data storage drives, are configured to be easily upgraded due to their increased spacing from one another. In illustrative embodiments, the components additionally include hardware authentication features to prove their authenticity.

[0246] A data center can utilize a single network fabric ("fabric") that supports multiple other network fabrics, including Ethernet and Omni-Path. The sleds can be coupled to switches via fiber, which provides higher bandwidth and lower latency than typical twisted pair cabling (e.g., Category 5, Category 5e, Category 6, etc.). Due to the high bandwidth, low latency interconnect and network fabric, the data center can pool resources, such as memory, accelerators (e.g., GPUs, graphics accelerators, FPGAs, ASICs, neural network and / or artificial intelligence accelerators, etc.) and physically disaggregated data storage drives, in use and provide them to compute resources (e.g., processors) as needed to enable the compute resources to access the pooled resources as if they were local resources.

[0247] A power supply or power source can provide voltage and / or current to the system 1400 or any of the components or systems described herein. In one example, the power supply includes an AC-to-DC (alternating current to direct current) adapter that plugs into a wall outlet. Such AC power can be a renewable energy (e.g., solar) power source. In one example, the power source includes a DC power source, such as an external AC-to-DC converter. In one example, the power source or power supply includes wireless charging hardware that charges via proximity to a charging field. In one example, the power source can include an internal battery, an alternating current power supply, a motion-based power supply, a solar power supply, or a fuel cell power source.

[0248] Figure 15 is a block diagram of an embodiment of a processor 1500 having one or more processor cores 1502A-1502N, an integrated memory controller 1514, and an integrated graphics processor 1508. Figure 16Those elements of the figures having the same reference numerals (or names) as the elements of any other figure herein have the same meaning as described elsewhere herein, and are not limited in their operation to the context in which they are depicted. The processor 1500 can include additional cores up to and including the additional core 1502N represented by the dashed lined boxes. Each of the processor cores 1502A-1502N includes one or more internal cache units 1504A-1504N. In some embodiments, each processor core also has access to one or more shared cache units 1506.

[0249] The internal cache units 1504A-1504N and the shared cache unit 1506 represent a cache memory hierarchy within the processor 1500. The cache memory hierarchy can include at least one level of instruction and data caches within each processor core and one or more levels of shared mid-level cache, such as a level 2 (L2), level 3 (L3), level 4 (L4), or other level cache, with the highest level cache before external memory being classified as an LLC. In some embodiments, cache coherence logic maintains coherence between the various cache units 1506 and 1504A-1504N.

[0250] In some embodiments, the processor 1500 also includes a set of one or more bus controller units 1516 and a system agent core 1510. The one or more bus controller units 1516 manage a set of peripheral buses, such as one or more PCI or rapid PCI buses. The system agent core 1510 provides management functionality for the various processor components. In some embodiments, the system agent core 1510 includes one or more integrated memory controllers 1514 to manage access to various external memory devices (not shown).

[0251] In some embodiments, one or more of the processor cores 1502A-1502N include support for simultaneous multi-threading. In such embodiments, the system agent core 1510 includes components for coordinating and operating the cores 1502A-1502N during multi-threaded processing. The system agent core 1510 can additionally include a power control unit (PCU) including logic and components to regulate the power state of the processor cores 1502A-1502N and the graphics processor 1508.

[0252] In some embodiments, the processor 1500 additionally includes a graphics processor 1508 for performing graphics processing operations. In some embodiments, the graphics processor 1508 is coupled with a set of shared cache units 1506 and system agent cores 1510 (including one or more integrated memory controllers 1514). In some embodiments, the system agent cores 1510 also include a display controller 1511 to drive output to one or more coupled displays. In some embodiments, the display controller 1511 can also be a separate module coupled with the graphics processor via at least one interconnect, or can be integrated within the graphics processor 1508.

[0253] In some embodiments, ring-based interconnect units 1512 are used to couple the internal components of the processor 1500. However, alternative interconnect units can be used, such as point-to-point interconnects, switched interconnects, or other technologies, including those well known in the art. In some embodiments, the graphics processor 1508 is coupled with the ring interconnect 1512 via an I / O link 1513.

[0254] The exemplary I / O link 1513 represents at least one of a variety of I / O interconnects, including an on-package I / O interconnect that facilitates communication between various processor components and a high-performance embedded memory module 1518, such as an eDRAM module. In some embodiments, each of the processor cores 1502A-1502N and the graphics processor 1508 can use the embedded memory module 1518 as a shared last-level cache.

[0255] In some embodiments, the processor cores 1502A-1502N are homogeneous cores executing the same instruction set architecture. In another embodiment, the processor cores 1502A-1502N are heterogeneous in terms of instruction set architecture (ISA), where one or more of the processor cores 1502A-1502N execute a first instruction set and at least one of the other cores executes a subset or a different instruction set. In one embodiment, the processor cores 1502A-1502N are heterogeneous in terms of microarchitecture, where one or more of the cores have a relatively high power consumption and one or more of the cores have a lower power consumption. In one embodiment, the processor cores 1502A-1502N are heterogeneous in terms of computing capability. Additionally, the processor 1500 can be implemented on one or more chips or as a SoC integrated circuit having the illustrated components in addition to other components not shown.

[0256] Graphics processing engineis a block diagram of a graphics processor 1600 that can be a discrete graphics processing unit, or can be graphics processor integrated with a multiple of processing cores or other semiconductor devices such as, but not limited to, a memory device or a network interface. In some embodiments, the graphics processor communicates via a memory mapped I / O interface to registers on the graphics processor and utilizes commands placed into the processor memory to execute graphics processing. In some embodiments, the graphics processor 1600 includes a memory interface 1614 for accessing a memory. The memory interface 1614 can be an interface to local memory, one or more internal caches, one or more shared external caches, and / or to system memory.

[0257] In some embodiments, the graphics processor 1600 also includes a display controller 1602 to drive display output data to a display device 1618. The display controller 1602 includes hardware for composing multiple layers of video or user interface elements into one or more overlay planes of a display. The display device 1618 can be an internal or external display device. In one embodiment, the display device 1618 is a head mounted display device, such as a virtual reality (VR) display device or an augmented reality (AR) display device. In some embodiments, the graphics processor 1600 includes a video codec engine 1606 to

[0258] In some embodiments, the graphics processor 1600 includes a block image transfer (BLIT) engine 1604 to perform two-dimensional (2D) rasterizer operations including, for example, bit-boundary block transfers. However, in one embodiment, 2D graphics operations are performed using one or more components of the graphics processing engine (GPE) 1610. In some embodiments, GPE 1610 is a compute engine for performing graphics operations, including three-dimensional (3D) graphics operations and media operations.

[0259] In some embodiments, GPE 1610 includes a 3D pipeline 1612 for processing 3D operations, such as rendering three-dimensional images and scenes using processing functions that act upon 3D primitive shapes (e.g., rectangle, triangle, etc.). The 3D pipeline 1612 includes programmable and fixed function elements that perform various tasks and / or generate execution threads for the 3D / Media sub-system 1615. While the 3D pipeline 1612 can be used to perform media operations, embodiments of GPE 1610 also include a media pipeline 1616 that is specifically used for

[0260] In some embodiments, media pipeline 1616 includes fixed function or programmable logic units to perform one or more specialized media operations such as video decode acceleration, video de-interlacing, and video codec acceleration. In some embodiments, media pipeline 1616 additionally includes a thread spawning unit to spawn threads for execution on the 3D / Media sub-system 1615. The spawned threads execute on one or more graphics processing clusters included in 3D / Media sub-system 1615 to perform computations for the media operations.

[0261] In some embodiments, 3D / Media sub-system 1615 includes logic to execute threads generated by 3D pipeline 1612 and media pipeline 1616. In one embodiment, the pipelines send thread execution requests to the 3D / Media sub-system 1615, which includes thread dispatch logic to arbitrate the various requests for the available thread execution resources. Execution resources include an array of graphics execution units for processing the 3D and media threads. In some embodiments, 3D / Media sub-system 1615 includes one or more internal caches to cache instructions and data for threads executed on the graphics execution units. In some embodiments, the subsystem also includes shared memory that the threads use to store data and

[0262] Figure 17

[0263] Figure 16 is a block diagram of a graphics processing engine 1710 of a graphics processor of some embodiments. In one embodiment, the graphics processing engine (GPE) 1710 is a version of the GPE 1610 shown in FIG. 16. Figure 17 Figure 16 Elements in FIG. 17 having the same or similar reference numbers (or names) as the elements of any other figure herein can operate or function in any manner similar to the manner described herein elsewhere, but are not limited to such. For example, the elements shown in FIG. 17 Figure 14 ​of the 3D pipeline 1612 and the media pipeline 1616. The media pipeline 1616 is optional in some embodiments of the GPE 1710 and can not be explicitly included in the GPE 1710. For instance, and in at least one embodiment, an independent media and / or image processor is coupled to the GPE 1710.

[0264] In some embodiments, the GPE 1710 is coupled with or includes a command streamer 1703 that provides a command stream to the 3D pipeline 1612 and / or media pipeline 1616. In some embodiments, the command streamer 1703 is coupled with memory, which can be system memory, or one or more of internal cache memory and shared cache memory. In some embodiments, the command streamer 1703 receives commands from the memory and sends commands to the 3D pipeline 1612 and / or media pipeline 1616. A command is an instruction to perform a task such as setting a state value in the 3D pipeline 1612, the media pipeline 1616, or a register in a sub- pipeline. In one embodiment, the ring buffer additionally includes a batch command buffer that stores batches of commands. Commands for the 3D pipeline 1612 can also include references to data stored in memory, for example, but not limited to, vertex and geometry data for the 3D pipeline 1612, and / or image data and memory objects for the media pipeline 1616. The 3D pipeline 1612 and media pipeline 1616 process the commands and data, making use of any operands that are available in the pipeline registers. In various embodiments, the 3D pipeline 1612 and media pipeline 1616 each include one or more fixed function and / or programmable pipelines. The fixed function pipeline can include, in some embodiments, graphics processing units (GPUs) that perform one or more of the following: geometry processing, thread dispatch, and / or other tasks. The programmable pipeline can include one or more programmable processors that implement graphics processing programs.

[0265] In various embodiments, the 3D pipeline 1612 can include fixed function and programmable logic to process one or more shader programs, which can be executed by the graphics cores in the graphics core array 1714. The graphics core array 1714 provides unified execution resources to execute the shader programs. The multiple-purpose execution logic (e.g., execution units) within the graphics core(s) 1715A-1715B of the graphics core array 1714 include support for multiple 3D API shader languages and can execute instructions associated with multiple, simultaneous execution threads.

[0266] In some embodiments, the graphics core array 1714 includes execution logic to perform media functions, such as video and / or image processing. In one embodiment, the execution units include general -purpose logic that can be programmed to perform parallel general -purpose computing operations, in addition to graphics processing operations. The general -purpose logic can be Figure 15 the processor core(s) 1407 or general -purpose logic within the cores 1502A-1502N in Figure 18 may be performed in parallel or in conjunction with the processing operations of the general -purpose logic within the processor core(s) 1407 or cores 1502A-1502N in

[0267] Output data generated by threads executing on the graphics core array 1714 can be output to memory in a unified return buffer (URB) 1718. The URB 1718 can store data for multiple threads. In some embodiments, the URB 1718 can be used to send data between different threads executing on the graphics core array 1714. In some embodiments, the URB 1718 can additionally be used for synchronization and shared function logic 1720 within the fixed function logic between threads on the graphics core array.

[0268] In some embodiments, the graphics core array 1714 is scalable such that the array includes a variable number of graphics cores, each having a variable number of execution units based on target performance and power levels of the GPE 1710. In one embodiment, the execution resources are dynamically scalable, such that execution resources can be enabled or disabled as needed.

[0269] The graphics core array 1714 is coupled with shared function logic 1720 that includes a number of resources shared among the graphics cores in the graphics core array. Shared functions within the shared function logic 1720 are hardware logic units that provide specialized supplemental functionality to the graphics core array 1714. In various embodiments, the shared function logic 1720 includes, without limitation, a sampler 1721, a math 1722, and an inter-thread communication (ITC) 1723 logic. Additionally, some embodiments implement one or more caches 1725 within the shared function logic 1720.

[0270] Shared functions are implemented in cases where the demand for a given specialized function is insufficient to include within the graphics core array 1714. Instead, a single instantiation of the specialized function is implemented as a separate entity within shared function logic 1720 and shared among the execution resources within the graphics core array 1714. The precise set of functions that are shared among the graphics core arrays 1714 and included within the graphics core array 1714 varies across embodiments. In some embodiments, particular shared functions within the shared function logic 1720 that are used extensively by the graphics core array 1714 can be included in the shared function logic 1716 within the graphics core array 1714. In various embodiments, the shared function logic 1716 in the graphics core array 1714 can include some or all of the logic in the shared function logic 1720. In one embodiment, all of the logic elements in the shared function logic 1720 can be duplicated in the shared function logic 1716 of the graphics core array 1714. In one embodiment, the shared function logic 1720 is excluded in favor of the shared function logic 1716 in the graphics core array 1714.

[0271] Figure 18 is a block diagram of hardware logic of a graphics processor core 1800 in accordance with some embodiments described herein. Figure 17 Elements in the figures with the same or similar reference numbers (or names) as the elements of any other figure herein have the same or similar orientation and / or functionality as the elements to which they are assigned, and can operate or function in any manner similar to that described elsewhere herein, but are not limited to such. In some embodiments, the illustrated graphics processor core 1800 includes a graphics core array 1714 in Figure 16 The graphics processor core 1800 (sometimes referred to as a core slice) can be one or more graphics cores in a modular graphics processor. The graphics processor core 1800 is an example of one graphics core slice, and graphics processors described herein can include multiple graphics core slices based on target power and performance envelopes. Each graphics processor core 1800 can contain fixed function blocks 1830 coupled with multiple sub-cores 1801A-1801F (also referred to as sub-slices) that include modular blocks of general-purpose and fixed function logic.

[0272] In some embodiments, the fixed function blocks 1830 include a geometry / fixed function pipeline 1836, which can be shared by all of the sub-cores in the graphics processor core 1800, for example, in lower performance and / or lower power graphics processor implementations. In various embodiments, the geometry / fixed function pipeline 1836 includes a 3D fixed function pipeline (e.g., like 3D pipeline 1612 in FIG. 17), a video front-end unit, a thread generator and thread dispatcher, and a unified return buffer manager that manages a unified return buffer, such as Figure 17 and 3D pipeline 1612 in FIG. 17), a video front-end unit, a thread generator and thread dispatcher, and a unified return buffer manager that manages a unified return buffer, such asFigure 16 unified return buffer 1718.

[0273] In one embodiment, fixed function block 1830 also includes a graphics SoC interface 1837, a graphics microcontroller 1838, and a media pipeline 1839. Graphics SoC interface 1837 provides an interface between graphics processor kernel 1800 and other processor kernels within the system on a chip. Graphics microcontroller 1838 is a programmable sub-processor that is configurable to manage various Figure 17 and Figure 17 Media pipeline 1839 (e.g., media pipeline 1616) includes logic to accelerate encoding, decoding, pre-processing, and / or post-processing of multimedia data (including image and video data). Media operations can be facilitated via requests made by the compute or sampling logic within sub-kernels 1801-1801F to the media pipeline 1839.

[0274] In one embodiment, SoC interface 1837 enables graphics processor kernel 1800 to communicate with general-purpose application processor cores (e.g., CPUs) within the SoC, and / or other components within the SoC, including memory hierarchy elements such as shared L2 cache, system RAM and / or embedded on-chip or package on-lead DRAM. SoC interface 1837 can also enable communication with fixed function devices within the SoC, such as camera imaging pipelines, and enable use of and / or implementation of global memory atoms that are shared between graphics processor kernel 1800 and CPUs within the SoC. SoC interface 1837 can also implement power management controls for graphics processor kernel 1800 and enable an interface between a clock domain of the graphics processor kernel 1800 and other clock domains within the SoC. In one embodiment, SoC interface 1837 enables receipt of command buffers from a command streamer and global thread dispatcher that are configured to provide commands and instructions to each of one or more graphics kernels within the graphics processor. Commands and instructions can be dispatched to media pipeline 1839 when media operations are to be performed, or to the geometry and fixed function pipeline (e.g., geometry and fixed function pipeline 1836, geometry and fixed function pipeline 1814) when graphics processing operations are to be performed.

[0275] The graphics microcontroller 1838 can be configured to perform various scheduling and management tasks for the graphics processor cores 1800. In one embodiment, the graphics microcontroller 1838 can execute graphics and / or compute workload scheduling on various graphics processing engines within the arrays of execution units (EUs) 1802A-1802F, 1804A-1804F within the sub-cores 1801A-1801F. In this scheduling model, host software executing on a CPU core of a SoC that includes the graphics processor cores 1800 can submit workloads to one of the plurality of graphics processor doorbell, which invokes a scheduling operation on the appropriate graphics engine. The scheduling operation includes determining which workload to run next, submitting the workload to a command streamer, pre-empting existing workloads running on the engine, monitoring progress of the workload, and notifying the host software when the workload is complete. In one embodiment, the graphics microcontroller 1838 can also facilitate low power or idle states for the graphics processor cores 1800, thereby providing the graphics processor cores 1800 with the ability to save and restore registers across low power state transitions independent of an operating system and / or graphics driver software on the system.

[0276] The graphics processor cores 1800 can have more or less than the shown sub-cores 1801A-1801F, up to N modular sub-cores. For each set of N sub-cores, the graphics processor cores 1800 can also include shared function logic 1810, shared and / or cache memory 1812, geometry / fixed function pipeline 1814, and additional fixed function logic 1816 to accelerate various graphics and compute processing operations. The shared function logic 1810 can include logic units associated with the shared function logic 1720 (e.g., samplers, mathematical and / or inter-thread communication logic) that can be shared by each N sub-core within the graphics processor cores 1800. The shared and / or cache memory 1812 can be a last level cache for the set of N sub-cores 1801A-1801F in the graphics processor cores 1800 and can also act as shared memory that can be accessed by multiple sub-cores. The geometry / fixed function pipeline 1814 can be included in the fixed function block 1830 instead of the geometry / fixed function pipeline 1836 and can include the same or similar logic units. Execution unit

[0277] ​In one embodiment, graphics processor core 1800 includes additional fixed function logic 1816 which can include various fixed function accelerator logic for use by graphics processor core 1800. In one embodiment, the additional fixed function logic 1816 includes an additional geometry pipeline for use in position only shading. In position only shading, there are two geometry pipelines, the full geometry pipeline in geometry / fixed function pipeline 1816, 1836 and a cull pipeline, which is an additional geometry pipeline, which can be included in additional fixed function logic 1816. In one embodiment, the cull pipeline is a slimmed down version of the full geometry pipeline. The full pipeline and the cull pipeline can execute different instances of the same application, each with a separate context. Position only shading can hide long cull runs of discarded triangles to enable the shading to complete ahead of schedule in some instances. For example, and in one embodiment, the cull pipeline logic in additional fixed function logic 1816 can execute the position shader in parallel with the main application and generally generates critical results faster than the full pipeline because the cull pipeline only takes and shades the position attributes of the vertices without performing rasterization and rendering to the pixel of the frame buffer. The cull pipeline can use the generated critical results to compute visibility information for all triangles regardless of whether the triangles are culled. The full pipeline, which in this case can be referred to as a replay pipeline, can consume the visibility information to skip the culled triangles and only shade the visible triangles that are ultimately passed to the rasterization stage.

[0278] In one embodiment, the additional fixed function logic 1816 can also include machine learning acceleration logic, such as fixed function matrix multiplication logic, for use in the implementation of optimizations including machine learning training or inferencing.

[0279] Within each graphics sub-core 1801A-1801F, a set of execution resources are included which can be used to perform graphics, media, and compute operations in response to requests by graphics pipeline, media pipeline, or shader programs. Graphics sub-cores 1801A-1801F include multiple arrays of execution units 1802A-1802F, 1804A-1804F, thread dispatch and inter-thread communication (TD / IC) logic 1803A-1803F, a 3D (e.g., texture) sampler 1805A-1805F, a media sampler 1806A-1806F, a shader processor 1807A-1807F, and shared local memory (SLM) 1808A-1808F. The arrays of execution units 1802A-1802F, 1804A-1804F each include multiple execution units, which are general-purpose graphics processing units capable of performing floating-point and integer / fixed-point logic operations in service of a graphics, media, or compute application running on the graphics processing unit. The TD / IC logic 1803A-1803F is to perform local thread dispatch and thread control operations for execution units within a sub-core and facilitate communication between threads executing on the execution units of the sub-core. The 3D sampler 1805A-1805F can read texture or other 3D graphics related data into memory. The 3D sampler can read texture data differently based on a configured sampling state and a texture format associated with a given texture. The media sampler 1806A-1806F can perform similar read operations based on a type and format associated with media data. In one embodiment, each graphics sub-core 1801A-1801F can alternatively include a unified 3D and media sampler. Threads executing on the execution units within each of the sub-cores 1801A-1801F can make use of shared local memory 1808A-1808F within each of the sub-cores, to enable threads executing within a thread group to execute using a common pool of on-chip memory.

[0280] Figures 19A-19B

[0281] Figures 19A-19B Thread execution logic 1900 is shown, including an array of processing elements employed in a graphics processor core, in accordance with embodiments described herein. Figure 19A Elements in the figures with the same or similar reference numerals (or names) as elements in any other figure herein have the same or similar functionality (although they can not be identical to each other), and can operate or function in any manner similar to that described elsewhere herein, but are not limited to such. Figure 18 An overview of thread execution logic 1900 is shown, which can include hardware logic employed in each of graphics sub-cores 1801A-1801F. Figure 19B Variations of the hardware logic shown for each sub-core 1801A-1801F of graphics processing unit 1800. Figure 19A Exemplary internal details of an execution unit are shown.

[0282] like Figure 19B As shown, in some embodiments, thread execution logic 1900 includes a shader processor 1902, a thread dispatcher 1904, an instruction cache 1906, a scalable execution unit array (including multiple execution units 1908A-1908N), a sampler 1910, a data cache 1912, and a data port 1914. In one embodiment, the scalable execution unit array can be dynamically scaled by enabling or disabling one or more execution units (e.g., any one of execution units 1908A, 1908B, 1908C, 1908D through 1908N-1 and 1908N) based on workload computational requirements. In one embodiment, the included components are interconnected via an interconnect architecture linking each of the components. In some embodiments, thread execution logic 1900 includes one or more connections to memory (e.g., system memory or cache memory) via one or more of the instruction cache 1906, data port 1914, sampler 1910, and execution units 1908A-1908N. In some embodiments, each execution unit (e.g., 1908A) is an independent programmable general-purpose computing unit capable of executing multiple concurrent hardware threads, processing multiple data elements in parallel for each thread. In various embodiments, the execution unit arrays 1908A-1908N are scalable to include any number of individual execution units.

[0283] In some embodiments, execution units 1908A-1908N are primarily used to execute shader programs. Shader processor 1902 can process various shader programs and dispatch execution threads associated with the shader programs via thread dispatcher 1904. In one embodiment, the thread dispatcher includes logic for arbitrating thread initiation requests from the graphics and media pipeline and instantiating the requested thread on one or more execution units in execution units 1908A-1908N. For example, a geometry pipeline can dispatch vertex, tessellation, or geometry shaders to thread execution logic for processing. In some embodiments, thread dispatcher 1904 can also handle runtime thread generation requests from executing shader programs.

[0284] In some embodiments, execution units 1908A-1908N support an instruction set that includes native support for many standard 3D graphics shader instructions, such that a

[0285] Each of execution units 1908A-1908N is capable of multi-issue single instruction multiple data (SIMD) execution and multi- threaded operation enabling high processing throughput in high, low, and mid pixel rates with thread level parallelism for simultaneous execution. Each hardware thread is able to access an independent thread state from a group of thread state registers. Each thread is able to run a thread- level

[0286] The execution unit instruction set can include SIMD instructions. Various data elements can be stored as a packed data type in a register and the execution unit will process the data elements in the packed data type in parallel. For example, when operating on a 256-bit wide vector, the 256 bits store eight 32-bit data elements, which can be 8 single precision (32-bit) floating point elements, 4 double precision (64-bit) floating point elements, or 16 byte integer data elements. Other vector widths and data element sizes can also be employed. For example, a 128-bit wide execution unit can operate on 128-bit wide data elements, or any other combination of data element size and width.

[0287] In one embodiment, one or more execution units can be combined in a fused execution unit 1909A-1909N, which has thread control logic (1907A-1907N) common to the fused EU. Multiple EU's can be fused into an EU group. Each EU in the fused EU group can be configured to execute a separate SIMD hardware thread. The number of EU's in a fused EU group can vary from one embodiment to another. In addition, a variety of SIMD widths can be employed including, but not limited to, SIMD8, SIMD16, and SIMD32. Each fused graphics execution unit 1909A-1909N includes at least two execution units. For example, fused execution unit 1909A includes first EU 1908A, second EU 1908B, and thread control logic 1907A common to first EU 1908A and second EU 1908B. Thread control logic 1907A controls threads executing on fused graphics execution unit 1909A to allow each EU in fused execution units 1909A-1909N to use a common instruction pointer register for execution.

[0288] One or more internal instruction caches (e.g., 1906) are included in thread execution logic 1900 to cache thread instructions for execution units. In some embodiments, one or more data caches (e.g., 1912) are included to cache thread data during thread execution. In some embodiments, a sampler 1910 is included to provide texture sampling for 3D operations and media sampling for media operations. In some embodiments, sampler 1910 includes specialized texture or media sampling functionality to process texture or media data during a sampling process before providing the sampled data to an execution unit.

[0289] During execution, the graphics and media pipeline sends thread initiation requests to the thread execution logic 1900 via thread generation and dispatch logic. Once a set of geometry objects have been processed and rasterized into pixel data, pixel processor logic (e.g., pixel shader logic, fragment shader logic, etc.) within the pixel shader 1902 is invoked to further calculate output information and cause results to be written to an output surface (e.g., color buffer, depth buffer, stencil buffer, etc.). In some embodiments, the pixel shader or fragment shader calculates values for various vertex attributes to be interpolated across a rasterized object. In some embodiments, the pixel processor logic within the shader processor 1902 then executes an application programming interface (API)-supplied pixel or fragment shader program. To execute the shader program, the shader processor 1902 dispatches threads to execution units (e.g., 1908A) via a thread dispatcher 1904. In some embodiments, the pixel shader 1902 uses texture sampling logic in a sampler 1910 to access texture data in a texture map stored in memory. Arithmetic operations on the texture data and input geometry data calculate pixel color data for each geometry fragment, or discard one or more pixels from further processing.

[0290] In some embodiments, the data port 1914 provides a memory access mechanism for the thread execution logic 1900 to output processed data to memory for further processing on a graphics processor output pipeline. In some embodiments, the data port 1914 includes or is coupled to one or more cache memories (e.g., data cache 1912) to store data for memory accesses via the data port cache.

[0291] As shown in Figure 20 The graphics execution unit 1908 can include an instruction fetch unit 1937, a general register file array (GRF) 1924, an architectural register file array (ARF) 1926, a thread arbiter 1922, a send unit 1930, a branch unit 1932, a set of SIMD floating point units (FPUs) 1934, and in one embodiment, a set of dedicated integer SIMD ALUs 1935. The GRF 1924 and the ARF 1926 include a set of general register files and architectural register files associated with each simultaneous hardware thread that is active in the graphics execution unit 1908. In one embodiment, each thread architectural state is held in the ARF 1926, while data used during thread execution is stored in the GRF 1924. The execution state of each thread, including the program counter for each thread, can be retained in the thread-specific registers in the ARF 1926.

[0292] In one embodiment, graphics execution unit 1908 has an architecture which is a combination of a SIMD and a fine-grained, inter-thread interleaved (IMT) thread execution architecture. The architecture has a modular configuration which can be tuned at design time to facilitate a balance between single thread performance and thread-level parallelism (TLP) according to target number of simultaneous threads (N) and resources (registers, FPU, etc.) per thread.

[0293] In one embodiment, graphics execution unit 1908 can co-issue multiple instructions, each of which can be a different instruction. Thread arbiter 1922 of graphics execution unit thread 1908 can dispatch an instruction to one of send unit 1930, branch unit 1932, or SIMD FPU(s) 1934 for execution. Each execution thread has access to 128 general purpose registers in GRF 1924, where each register can store 32 bytes which can be accessed as an 8 element vector of 32 bit data elements. In one embodiment, each execution unit thread has access to 4Kbytes of the GRF 1924, although embodiments are not so limited and can provide more or less register resources in other embodiments. In one embodiment, up to seven threads can be executed simultaneously, although the number of threads per execution unit can vary according to embodiments. In an embodiment where seven threads have access to 4Kbytes, GRF 1924 can store a total of 28Kbytes. Flexible addressing modes can allow registers to be addressed together to effectively construct wider registers or to represent strided rectangular block data structures.

[0294] In one embodiment, memory operations, sampler operations, and other longer latency system communications are dispatched via a “send” instruction which is executed by message passing send unit 1930. In one embodiment, branch instructions are dispatched to a dedicated branch unit 1932 to facilitate SIMD divergence and eventual convergence.

[0295] In one embodiment, graphics execution unit 1908 includes one or more SIMD floating point units ((one or more) FPUs) 1934 to perform floating point operations. In one embodiment, (one or more) FPUs 1934 also support integer computation. In one embodiment, (one or more) FPUs 1934 can SIMD execute up to M 32-bit floating point (or integer) operations, or up to 2M 16-bit integer or 16-bit floating point operations. In one embodiment, at least one FPU provides extended math capability to support high throughput transcendental math functions and double precision 64-bit floating point. In some embodiments, there is also a set of 8-bit integer SIMD ALUs 1935 and they can be specifically optimized to perform operations associated with machine learning calculations.

[0296] In one embodiment, an array of multiple instances of the graphics execution unit 1908 can be instantiated in a graphics sub-kernel (e.g., a sub slice). For scalability, the exact number of execution units activated for each sub-kernel can be flexible as needed. In one embodiment, the execution units 1908 can execute instructions concurrently. In another embodiment, each thread on the graphics processing unit 1900 can execute threads on different threads.

[0297] Graphics pipeline FIG. 2B is a block diagram illustrating a graphics processor instruction formats according to some embodiments. In one or more embodiment, a graphics processor execution unit supports an instruction set that includes a set of instructions for performing operations on graphics data. More specifically, graphics processor execution units provide high performance data parallel execution for many graphics processing tasks, such as rasterization, shading, and computation. The design of the execution unit allows for instructor- level control over data parallel performance and precision distribution, per task, across the available processing resources. The execution resources are utilized individually when per-task data workloads are small, and jointly when per task data workloads are large.

[0298] In some embodiments, a graphics processor execution unit natively supports instructions in a 128-bit instruction format 2010. A 64-bit compact instruction format 2030 is also supported. An native 128-bit instruction format 2010 provides access to all instruction options available on the graphics processors, while some options and operations are restricted to the 64-bit format 2030. The native 128-bit instruction format 2010 has a prefix field 2011, an operation code field 2012, data fields 2013, a control field 2014, a counter field 2015, an execution size field 2016, and a number of thread context fields 2017. A prefix field 2011 in an instruction encodes information about the execution of the instruction. The prefix field can include information on the size of the operation, the number of threads, and the number of data elements. The prefix field can also include a number of other bits encoding information about the execution of the instruction.

[0299] For each format, the instruction operation code 2012 defines the operation to be performed by the execution unit. The execution unit executes each instruction in parallel across the multiple data elements in each of the operand. For example, in response to an add instruction, the execution unit performs a simultaneous add operation across each color channel for multiple texture elements. Many instructions are performed individually across each data element in the data lane. In some embodiments, the instruction control field 2014 enables control over the application of the operation to resources other than just the all lane data elements. In some embodiments, the instruction control field 2014 enables control over other aspects of the operation, such as the number of threads to execute the operation. The execution size field 2016 specifies the number of data elements in each dimension of an array to which the instruction applies.

[0300] Some execution unit instructions have up to three operands, including two source operands src0 2020, src1 2022 and one destination 2018. In some embodiments, the execution unit supports dual destination instructions in which one of the destinations is implicit. Data manipulation instructions can have a third source operand (e.g., SRC2 2024), where the instruction opcode 2012 determines the number of source operands. The last source operand for an instruction can be an immediate (e.g., hard coded) value passed with the instruction.

[0301] In some embodiments, the 128-bit instruction format 2010 includes an access / address mode field 2026 that specifies whether, for example, a direct register addressing mode or an indirect register addressing mode is used. When using direct register addressing mode, the register address for one or more operands is provided directly by bits in the instruction.

[0302] In some embodiments, the 128-bit instruction format 2010 includes an access / address mode field 2026 that specifies the addressing mode and / or access mode for the instruction. In one embodiment, the access mode is used to define the data access alignment for the instruction. Some embodiments support access modes that include a 16-byte aligned access mode and a 1 -byte aligned access mode, where the byte alignment of the access mode determines the access alignment of the instruction operands. For example, when in a first mode, the instruction can use byte-aligned addressing for source and destination operands, and when in a second mode, the instruction can use 16-byte aligned addressing for all source and destination operands.

[0303] In one embodiment, the addressing mode portion of the access / address mode field 2026 determines whether the instruction will use direct addressing or indirect addressing. When using direct register addressing mode, the register address for one or more operands is provided directly by bits in the instruction. When using indirect register addressing mode, the register address for one or more operands is calculated based on an address immediate field in the instruction and an address register value.

[0304] In some embodiments, instructions are grouped based on the opcode 2012 bit field to simplify opcode decoding 2040. For 8-bit opcodes, bits 4, 5, and 6 allow the execution units to determine the type of opcode. The exact opcode grouping shown is merely an example. In some embodiments, the move and logic opcode group 2042 includes data move and logic instructions (e.g., move (mov), compare (cmp)). In some embodiments, the move and logic group 2042 shares five most significant bits (MSBs), with move (mov) instructions taking the form of 0000xxxxb and logic instructions taking the form of 0001xxxxb. The flow control instruction group 2044 (e.g., call, jump (jmp)) includes instructions that take the form of 0010xxxxb (e.g., Ox20). The miscellaneous instruction group 2046 includes a mix of instructions that includes synchronization instructions (e.g., wait, send) that take the form of 0011xxxxb (e.g., Ox30). The parallel math instruction group 2048 includes per-component arithmetic instructions (e.g., add, multiply (mul)) that take the form of 0100xxxxb (e.g., Ox40). The parallel math group 2048 performs arithmetic operations in parallel across data lanes. The vector math group 2050 includes arithmetic instructions (e.g., dp4) that take the form of 0101xxxxb (e.g., Ox50). The vector math group performs arithmetic such as dot product calculations on vector operands.

[0305] Figure 21

[0306] Figure 21 is a block diagram of another embodiment of a graphics processor 2100. Graphics pipeline programming Elements in the figures having the same or similar reference numerals (or names) as elements in any other figure herein have the same or similar functional operations. The numbering of the elements 1000-1099 is consistent between figures. Numbers are used to indicate various components in the figures. Numbers in different figures often refer to components with similar or identical functionality for ease of reference.

[0307] In some embodiments, the graphics processor 2100 includes a geometry pipeline 2120, a media pipeline 2130, a display engine 2140, thread execution logic 2150, and a render output pipeline 2170. In some embodiments, graphics processor 2100 is a graphics processor included in a multi-core processing system that includes one or more general-purpose processing cores. The graphics processor is controlled by register writes to one or more control registers (not shown) or via commands issued to the graphics processor via the ring interconnect 2102. In some embodiments, the ring interconnect 2102 couples the graphics processor 2100 to other processing components such as other graphics processors or general-purpose processors. Commands from the ring interconnect 2102 are interpreted by the command streamer 2103, which supplies instructions to the individual components of the graphics processor 2100.

[0308] In some embodiments, the command streamer 2103 directs operations of a vertex fetcher 2105, which reads vertex data from memory and performs vertex processing commands provided by the command streamer 2103. In some embodiments, the vertex fetcher 2105 provides vertex data to a vertex shader 2107, which performs coordinate space transformation and lighting operations on each vertex. In some embodiments, the vertex fetcher 2105 and vertex shader 2107 execute vertex processing instructions by dispatching execution threads to execution units 2152A-2152B via thread dispatcher 2131.

[0309] In some embodiments, the execution units 2152A-2152B are an array of similar vector processors including instruction sets that are specifically designed and configured to perform a graphics and media operations. In some embodiments, the execution units 2152A-2152B have an attached Ll cache 2151 that is specific to each array or shared between arrays. The cache can be configured as a data cache, an instruction cache, or a single cache that is partitioned to contain both data and instructions.

[0310] In some embodiments, the geometry pipeline 2120 includes a tessellation component for performing hardware-accelerated tessellation of 3D objects. In some embodiments, a programmable hull shader 2111 configures tessellation operations. A programmable domain shader 2117 provides post-processing evaluation of tessellation output. A tessellator 2113 operates in the direction of the hull shader 2111 and includes specialized logic to generate a set of detailed geometric objects based on a coarse geometric model that is provided as input to the geometry pipeline 2120. In some embodiments, the tessellation component (e.g., the hull shader 2111, the tessellator 2113, and the domain shader 2117) can be bypassed if tessellation is not used.

[0311] In some embodiments, a complete geometric object can be processed by a geometry shader 2119 via one or more threads dispatched to the execution units 2152A-2152B, or can proceed directly to the clipper 2129. In some embodiments, the geometry shader operates on an entire geometric object rather than a patch of vertices or vertices as in previous stages of the graphics pipeline. If tessellation is disabled, the geometry shader 2119 receives input from the vertex shader 2107. In some embodiments, if the tessellation unit is disabled, the geometry shader 2119 can be programmed by a geometry shader program to perform geometric tessellation.

[0312] The clipper 2129 processes vertex data prior to rasterization. The clipper 2129 can be a fixed function clipper or a programmable clipper with clip and geometry shader functionality. In some embodiments, the rasterizer and depth test components 2173 in the render output pipeline 2170 dispatch pixel shaders to convert geometry objects into a per-pixel representation. In some embodiments, the pixel shader logic is included in the thread execution logic 2150. In some embodiments, an application can bypass the rasterizer and depth test components 2173 and access un-rasterized vertex data via a stream out unit 2123.

[0313] The graphics processor 2100 has an interconnect bus, interconnect fabric or some other interconnect mechanism to allow data and messages to be transferred between components of the processor. In some embodiments, the execution units 2152A-2152B and associated logic units (e.g., LI cache 2151, sampler 2154, texture cache 2158, etc.) are interconnected with a data port 2156 to perform memory accesses and communicate with other components of the processor. In some embodiments, the sampler 2154, LI cache 2151, texture cache 2158, and execution units 2152A-2152B each have separate memory access paths. In one embodiment, the texture cache 2158 can also be configured as a sample cache.

[0314] In some embodiments, the render output pipeline 2170 includes a rasterizer and depth test component 2173 that converts based on vertex data into associated pixel-based representations. In some embodiments, the rasterizer logic includes a windower / masker unit for performing fixed function triangle and line rasterization. An associated render cache 2178 and depth cache 2179 are also available in some embodiments. Pixel operation components 2177 perform pixel-based operations on the data, although in some instances pixel operations associated with 2D operations (e.g., bit block image blending for a texture map) are performed by 2D engine 2141, or replaced by display controller 2143 using a composite display plane during display. In some embodiments, a shared L3 cache 2175 is available for all graphics components, allowing sharing of data without the need to use main system memory.

[0315] In some embodiments, graphics processor media pipeline 2130 includes a media engine 2137 and a video front-end 2134. In some embodiments, video front-end 2134 receives pipeline commands from the command streamer 2103. In some embodiments, media pipeline 2130 includes a separate command streamer. In some embodiments, video front-end 2134 processes media commands before sending the media

[0316] In some embodiments, graphics processor 2100 includes a display engine 2140. In some embodiments, display engine 2140 is external to processor 2100 and coupled to the graphics processor via the ring interconnect 2102, or some other interconnect bus or fabric. In some embodiments, display engine 2140 includes a 2D engine 2141 and a display controller 2143. In some embodiments, display engine 2140 contains special purpose logic that is capable of operating independently of the 3D pipeline. In some embodiments, display controller 2143 is coupled to the

[0317] In some embodiments, geometry pipeline 2120 and media pipeline 2130 can be configured to perform operations based on a number of graphics and media programming interfaces and are not specific to any one application programming interface (API). In some embodiments, driver software for a graphics processor translates API calls that are specific to a particular graphics or media library into commands that can be processed by the graphics processor. In some embodiments, support is provided for the Open Graphics Library (OpenGL) and / or the Open Computing Language (OpenCL) from the Khronos Group. In some embodiments, support is also provided for the Direct3D library from the Microsoft Corporation. In some embodiments, a combination of these libraries can be supported. Support can also be provided for the Open Source Computer Vision Library (OpenCV). A future API having a compatible 3D pipeline can also be supported if a mapping of

[0318] Figure 22A

[0319] ​ FIG. 22 is a block diagram illustrating a graphics processor command format 2200 according to some embodiments. Figure 22B FIG. 23 is a block diagram illustrating a graphics processor command sequence 2210 according to some embodiments. Figure 22AThe solid-line boxes in the diagram illustrate components that are generally included in graphics commands, while the dashed-line components include optional or only included in a subset of graphics commands. Figure 22A The exemplary graphics processor command format 2200 includes a data field for identifying a client of the command 2202, a command operation code (opcode) 2204, and data 2206. A sub-opcode 2205 and a command size 2208 are also included in some commands.

[0320] In some embodiments, the client 2202 specifies a client unit of the graphics device that processes the command data. In some embodiments, a graphics processor command parser examines the client field of each command to adjust further processing of the command and route the command data to the appropriate client unit. In some embodiments, the graphics processor client units include a memory interface unit, a rendering unit, a 2D unit, a 3D unit, and a media unit. Each client unit has a corresponding processing pipeline that processes commands. Once a command is received by a client unit, the client unit reads the opcode 2204 and, if present, the sub-opcode 2205 to determine the operation to perform. The client unit uses the information in the data field 2206 to perform the command. For some commands, an explicit command size 2208 is expected to specify the size of the command. In some embodiments, the command parser automatically determines the size of at least some of the commands based on the command opcode. In some embodiments, commands are aligned via multiples of double-words. Other command formats can be used.

[0321] Figure 22B The flow diagram in illustrates an exemplary graphics processor command sequence 2210. In some embodiments, a software or firmware of a data processing system featuring embodiments of a graphics processor uses a version of the command sequence shown to set up, execute, and tear down a set of graphics operations. The sample command sequence is shown and described for exemplary purposes only as embodiments are not limited to these specific commands or to this command sequence. Moreover, the commands can be issued as a batch of commands in a command sequence, such that a graphics processor will process the command sequence in at least partially simultaneous fashion.

[0322] In some embodiments, the graphics processor command sequence 2210 can begin with a pipeline flush command 2212 to cause any active graphics pipeline to complete any ongoing commands in the pipeline. In some embodiments, 3D pipeline 2222 and media pipeline 2224 are not operated simultaneously. The execution pipeline flush can be used to ensure that any pending commands are complete, so that the graphics processor can be safely switched, on a pipeline basis, to a low-power or self-refresh state. In some embodiments, the pipeline flush is performed on the head of a pipeline so that the pipeline flush is completed before the next work issue is taken. In some embodiments, the pipeline flush can be omitted if the pipeline state is not affected by the pipeline flush.

[0323] In some embodiments, a pipeline select command 2213 is used when the command sequence requires an explicit switch between pipelines. In some embodiments, the pipeline select command 2213 is only required once for an execution context before issuing commands to the pipeline, unless the context is issuing commands to two pipelines. In some embodiments, the pipeline flush command 2212 is required immediately prior to a pipeline switch via the pipeline select command 2213.

[0324] In some embodiments, pipeline control commands 2214 configure a graphics pipeline for operation. In some embodiments, pipeline control commands 2214 configure pipeline state for the active pipeline. In one embodiment, pipeline control commands 2214 are used to perform pipeline synchronization and data clean up from one or more cache memories in the active pipeline before taking a batch of commands.

[0325] In some embodiments, a set of return buffers used for a respective pipeline to write data are configured using return buffer state 2216 commands. Some pipeline operations require allocation, selection, or configuration of one or more return buffers that write intermediate data to during processing. In some embodiments, the graphics processor also uses one or more return buffers to store output data and perform cross-thread communication. In some embodiments, return buffer state 2216 includes a set of selections of return buffer size and number to use for pipeline operations.

[0326] The remaining commands in the command sequence vary depending on the active pipeline for operation. Based on the pipeline determination 2220, the command sequence is tailored for the 3D pipeline 2222 beginning at 3D pipeline state 2230 or the media pipeline 2224 beginning at media pipeline state 2240.

[0327] The commands that configure the 3D pipeline state 2230 include 3D state setting commands for vertex buffer state, vertex element state, constant color state, depth buffer state, and other state variables that will be configured prior to processing 3D primitive commands. The values for these commands are determined based at least in part on the particular 3D API in use. In some embodiments, the 3D pipeline state 2230 commands can also selectively disable or bypass certain pipeline elements if those elements will not be used.

[0328] In some embodiments, the 3D primitive 2232 commands are used to submit 3D primitives for processing by the 3D pipeline. The commands and associated parameters passed to the graphics processor via the 3D primitive 2232 commands are forwarded to a fetch vertex function in the graphics pipeline. The fetch vertex function uses the 3D primitive 2232 command data to generate vertex data structures. The vertex data structures are stored in one or more return buffers. In some embodiments, the 3D primitive 2232 commands are used to perform vertex operations on 3D primitives via a vertex shader. To process the vertex shader, the 3D pipeline 2222 dispatches shader execution threads to the graphics processor execution units.

[0329] In some embodiments, the 3D pipeline 2222 is triggered via an execute 2234 command or event. In some embodiments, a register write triggers command execution. In some embodiments, execution is triggered via a 'go' or 'kick' command in the command sequence. In one embodiment, a pipeline synchronization command is used to trigger command execution to clear the command sequence through the graphics pipeline. The 3D pipeline will perform geometry processing for 3D primitives. Once the operations are complete, the resulting geometry is rasterized and pixels are shaded by a pixel engine. Additional commands to control pixel shading and pixel back end operations can also be included for those operations.

[0330] In some embodiments, the graphics processor command sequence 2210 follows the media pipeline 2224 path when performing media operations. Generally, the way in which the media pipeline 2224 is programmed and the particular use depends on the media or compute operations to be performed. During media decode, specific media decode operations can be offloaded to the media pipeline. In some embodiments, the media pipeline can also be bypassed and media decode can be performed entirely or partially using resources provided by one or more general purpose processing cores. In one embodiment, the media pipeline also includes elements for general purpose graphics processor unit (GPGPU) operations where the graphics processor is used to execute SIMD vector operations using a compute shader program that does not explicitly relate to the rendering of graphics primitives.

[0331] In some embodiments, media pipeline 2224 is configured in a similar manner as 3D pipeline 2222. A set of commands configuring media pipeline state 2240 is dispatched or placed into a command queue prior to media object command 2242. In some embodiments, the commands of media pipeline state 2240 include data for configuring media pipeline elements that will be used to process the media object. This includes data for configuring video decode and video encode logic within the media pipeline (such as encoding or decoding formats). In some embodiments, the commands of media pipeline state 2240 also support the use of one or more pointers to "indirect" state elements containing a batch of state settings.

[0332] In some embodiments, media object command 2242 supplies a pointer to a media object for processing by the media pipeline. The media object includes a memory buffer containing video data to be processed. In some embodiments, all media pipeline state must be valid prior to issuing media object command 2242. Once the pipeline state is configured and media object command 2242 is queued, media pipeline 2224 is triggered via execution command 2244 or an equivalent execution event (e.g., a register write). The output from media pipeline 2224 can then be post-processed by operations provided by 3D pipeline 2222 or media pipeline 2224. In some embodiments, GPGPU operations are configured and executed in a similar manner as media operations.

[0333] Graphics software architecture

[0334] Figure 23 An exemplary graphics software architecture for a data processing system 2300 is shown in accordance with some embodiments. In some embodiments, the software architecture includes a 3D graphics application 2310, an operating system 2320, and at least one processor 2330. In some embodiments, processor 2330 includes a graphics processor 2332 and one or more general-purpose processor cores 2334. Graphics application 2310 and operating system 2320 each execute in system memory 2350 of the data processing system.

[0335] In some embodiments, 3D graphics application 2310 includes one or more shader programs including shader instructions 2312. The shader language instructions can be in a high-level shader language, such as the High-Level Shader Language (HLSL) of Direct3D, the OpenGL Shader Language (GLSL), and so forth. The application also includes executable instructions 2314 in a machine language suitable for execution by the general-purpose processor cores 2334. The application also includes graphics objects 2316 defined by vertex data.

[0336] In some embodiments, operating system 2320 is Windows® from Microsoft Corporation Operating system 2320 can support a graphics API 2322, such as a Direct3D API, an OpenGL API, or a Vulkan API. When the Direct3D API is in use, operating system 2320 uses a front-end shader compiler 2324 to compile any shader instructions 2312 in HLSL into a lower-level shader language. The compilation can be a just-in-time (JIT) compilation or the application can perform shader pre- compilation. In some embodiments, high-level shaders are compiled into low-level shaders during compilation of a 3D graphics application 2310. In some embodiments, shader instructions 2312 are provided in an intermediate form, such as versions of the Standard Portable Intermediate Representation (SPIR) used by the Vulkan API.

[0337] In some embodiments, user mode graphics driver 2326 includes a back-end shader compiler 2327 to translate shader instructions 2312 into hardware-specific representations. When the OpenGL API is in use, shader instructions 2312 in GLSL high-level language are passed to user mode graphics driver 2326 for compilation. In some embodiments, user mode graphics driver 2326 communicates with a kernel mode graphics driver 2329 using operating system kernel mode functions 2328. In some embodiments, kernel mode graphics driver 2329 communicates with a graphics processor 2332 to dispatch

[0338] IP core implementation

[0339] One or more aspects of at least one embodiment can be implemented by representative code stored on a machine-readable medium which represents and / or defines logic within an integrated circuit such as a processor. For example, a machine-readable medium can include instructions that represent various logic within the processor. When read by a machine, the instructions can cause the machine to fabricate the logic to perform the techniques described herein. Such representations, known as "IP cores," are reusable units of logic for an integrated circuit that can be stored on a tangible, machine-readable medium as hardware description language, which can then be used in a semiconductor kitting process to manufacture the applicable integrated circuit. The logic of other embodiments is not limited to

[0340] Figure 24Ais a block diagram illustrating an IP core development system 2400 according to an embodiment, which can be used to manufacture an integrated circuit to perform operations. The IP core development system 2400 can be used to generate modular, re-usable designs that can be incorporated into larger designs or used to construct an entire integrated circuit (e.g., an SOC integrated circuit). A design facility 2430 can generate a software simulation 2410 of the IP core design in a high level programming language (e.g., C / C++). The software simulation 2410 can be used to design, test, and verify the behavior of the IP core using a simulation model 2412. The simulation model 2412 can include functional, behavioral, and / or timing simulations. A register transfer level (RTL) design 2415 can then be created or synthesized from the simulation model 2412. The RTL design 2415 is an abstraction of the behavior of the integrated circuit that models the flow of digital signals through the hardware registers, including the associated logic performed using the modeled digital signals. In addition to the RTL design 2415, a lower level of design using logic level or transistor level descriptions can also be created, designed, or synthesized. Thus, the particular details of the initial design and simulation can vary.

[0341] The RTL design 2415 or equivalent can be further synthesized, further designed, or further

[0342] Figure 24BA cross-sectional side view of an integrated circuit package assembly 2470 is shown, in accordance with some embodiments described herein. The integrated circuit package assembly 2470 illustrates an implementation of one or more processor or accelerator devices as described herein. The package assembly 2470 includes multiple units of hardware logic 2472, 2474 connected to a substrate 2480. The logic 2472, 2474 can be implemented at least partially in configurable logic or fixed function logic hardware, and can include one or more portions of any of the processor core(s), graphics processor(s), or other accelerator devices described herein. Each unit of logic 2472, 2474 can be implemented in a semiconductor die and coupled with the substrate 2480 via an interconnect structure 2473. The interconnect structure 2473 can be configured to route electrical signals between the logic 2472, 2474 and the substrate 2480, and can include interconnects such as, but not limited to, bumps or pillars. In some embodiments, the interconnect structure 2473 can be configured to route electrical signals such as, for example, input / output (I / O) signals and / or power or ground signals associated with operation of the logic 2472, 2474. In some embodiments, the substrate 2480 is an epoxy-based laminate substrate. In other embodiments, the substrate 2480 can include other suitable types of substrates. The package assembly 2470 can be connected to other electrical devices via a package interconnect 2483. The package interconnect 2483 can be coupled to a surface of the substrate 2480 to route electrical signals to other electrical devices such as a motherboard, other chipsets, or multi-chip modules.

[0343] In some embodiments, the units of logic 2472, 2474 are electrically coupled with a bridge 2482 that is configured to route electrical signals between the logic 2472, 2474. The bridge 2482 can be a dense interconnect structure that provides routing for electrical signals. The bridge 2482 can include a bridge substrate composed of glass or a suitable semiconductor material. Electrical routing features can be formed on the bridge substrate to provide chip-to-chip connections between the logic 2472, 2474.

[0344] While two units of logic 2472, 2474 and a bridge 2482 are shown, embodiments described herein can include more or fewer units of logic on one or more dies. The one or more dies can be connected by zero or more bridges, as bridges 2482 can be excluded when the logic is contained on a single die. Alternatively, multiple dies or units of logic can be connected through one or more bridges. Additionally, multiple units of logic, dies, and bridges can be connected together in other possible configurations, including three-dimensional configurations.

[0345] Exemplary system-on-a-chip integrated circuit

[0346] Figure 25-26 shows an exemplary integrated circuit and associated graphics processor that can be fabricated using one or more IP cores in accordance with various embodiments described herein. Other logic and circuits can be included, as appropriate, such as additional graphics processors / cores, peripheral interface controllers or general purpose processor cores.

[0347] Figure 25 is a block diagram that shows an exemplary system on a chip integrated circuit 2500 in accordance with an embodiment, which can be fabricated using one or more IP cores. The exemplary integrated circuit 2500 includes one or more application processor(s) 2505, e.g., CPUs, at least one graphics processor 2510, and can additionally include an image processor 2515 and / or a video processor 2520, any of which can be a modular IP core from the same or multiple different design facilities. Integrated circuit 2500 includes peripheral or bus logic including USB controllers 2525, UART controllers 2530, SPI / SDIO controllers 2535, and I 2 S / I 2 C controller 2540. In addition, the integrated circuit can include a display device 2545 coupled to one or more of a high-definition multimedia interface (HDMI) controller 2550 and mobile industry processor interface (MIPI) display interface 2555. Storage can be provided by a flash memory subsystem 2560, including flash memory and a flash memory controller. Memory interfaces can be provided via a memory controller 2565 for access to SDRAM or SRAM memory devices. Some integrated circuits additionally include an embedded security engine 2570.

[0348] Figures 26A-26B is a block diagram that shows an exemplary graphics processor used within a SoC in accordance with an embodiment described herein. Figure 26A shows an exemplary graphics processor of a system on a chip integrated circuit that can be fabricated using one or more IP cores in accordance with an embodiment. Figure 26B shows an additional exemplary graphics processor of a system on a chip integrated circuit that can be fabricated using one or more IP cores in accordance with an embodiment. Figure 26A The graphics processor 2610 of is an example of a low power graphics processor core. Figure 26B The graphics processor 2640 of is an example of a high performance graphics processor core. Each of the graphics processors 2610, 2640 can be a GPGPU core from a different design facility. Figure 25 a variation of the graphics processor 2510.

[0349] As Figure 26AAs shown, the graphics processor 2610 includes a vertex processor 2605 and one or more fragment processor(s) 2615A-2615N (e.g., 2615A, 2615B, 2615C, 2615D, through 2615N-1, and 2615N). The graphics processor 2610 can execute different shader programs via separate logic for vertex processing and / or for fragment / pixel processing. The vertex processor(s) 2605 is optimized to execute operations on one or more

[0350] The graphics processor 2610 additionally includes one or more memory management units (MMUs) 2620A-2620B, cache(s) 2625A-2625B, and circuit interconnects 2630A-2630B. The one or more MMU(s) 2620A-2620B provide for virtual to physical address mapping for memory accesses for the graphics processor 2610, including those for the vertex processor 2605 and / or the one or more fragment processor(s) 2615A-2615N, which can reference vertex and image / texture data stored in memory, in addition to any cache, that can also be stored in memory. Figure 25 The one or more MMU(s) 2620A-2620B can be synchronized with one or more MMUs within the system, including one or more MMUs associated with the one or more application processor(s) 2505, image processor(s) 2515, and / or video processor(s) 2520, such that each processor 2505-2520 can participate in a shared or unified virtual memory system. The one or more circuit interconnects 2630A-2630B enable the graphics processor 2610 to interface with other IP cores within the SoC, either via an internal bus of the SoC or via a direct connection, according to embodiments.

[0351] As shown, Figure 26B The graphics processor 2640 includes a vertex processor 2650 and one or more fragment processor(s) 2660A-2660N (e.g., 2660A, 2660B, 2660C, 2660D, through 2660N-1, and 2660N), as shown. The graphics processor 2640 can execute different shader programs via separate logic for vertex processing and / or for fragment / pixel processing. The vertex processor 2650 is optimized to execute operations on one or more Figure 26Aone or more MMUs 2620A-2620B, caches 2625A-2625B, and circuit interconnects 2630A-2630B of the graphics processor 2610. The graphics processor 2640 includes one or more shader cores 2655A-2655N (e.g., 2655A, 2655B, 2655C, 2655D, 2655E, 2655F, through 2655N-1, and 2655N) that provide a unified shader core architecture in which single programable shader cores or a single type of core can be implemented to perform all types of programmable shader programs including vertex shader programs, fragment shader programs, and / or compute shader programs. The exact number of shader cores to implement can vary in different embodiments and implementations. In addition, the graphics processor 2640 includes an inter-core task manager 2645, which acts as a thread dispatcher to apportion threads to the one or more shader cores 2655A-2655N and tiling unit 2658 to accelerate tiling operations for tile-based rendering in which rendering operations for a scene are subdivided in image space, such as to exploit local spatial coherence within a scene or to optimize use of internal caches.

[0352] Decomposition of SoC architecture Building larger and larger silicon dies is challenging for various reasons. As silicon dies become larger, manufacturing yields become smaller, and process technology requirements for different components can vary. On the other hand, to have a high performance system, key components should be interconnected through high speed, high bandwidth, low latency interfaces. These conflicting requirements pose challenges to the development of high performance chips.

[0353] Embodiments described herein provide techniques for decomposing the architecture of a system-on-a-chip integrated circuit into multiple different chiplets that can be packaged onto a common chassis. In one embodiment, a graphics processing unit or parallel processor is composed of a variety of silicon chiplets that are manufactured separately. A chiplet is an at least partially encapsulated integrated circuit that includes different logic units that can be assembled with other chiplets into a larger package. A diverse set of chiplets with different IP core logic can be assembled into a single device. In addition, chiplets can be integrated into a base die or base chiplet using active interposer technology. The concepts described herein enable interconnection and communication between different forms of IP within a GPU. Development of IP on different processes can be mixed. This avoids the complexity of converging multiple IPs to the same process, especially on large SoCs with several flavors of IP.

[0354] Allowing the use of multiple process technologies improves time to market and provides a cost effective way to create multiple product SKUs. For customers, this means getting products that are better suited to their needs in a cost effective and timely manner. In addition, the disaggregated IP is more easily power gated independently, allowing for components that are not in use on a given workload to be powered down, reducing overall power consumption.

[0355] Figure 27 A parallel computing system 2700 according to an embodiment is shown. In one embodiment, the parallel computing system 2700 includes a parallel processor 2720, which can be a graphics processor or compute accelerator as described herein. The parallel processor 2720 includes a global logic 2701 unit, an interface 2702, a thread dispatcher 2703, a media unit 2704, a set of compute units 2705A-2705H, and a cache / memory unit 2706. In one embodiment, the global logic unit 2701 includes global functionality for the parallel processor 2720, including device configuration registers, a global scheduler, power management logic, etc. The interface 2702 can include a front end interface for the parallel processor 2720. The thread dispatcher 2703 can receive a workload from the interface 2702 and dispatch threads of the workload to the compute units 2705A-2705H. If the workload includes any media operations, at least a portion of those operations can be performed by the media unit 2704. The media unit can also offload some operations to the compute units 2705A-2705H. The cache / memory unit 2706 can include cache memory (e.g., L3 cache) and local memory (e.g., HBM, GDDR) for the parallel processor 2720.

[0356] Figures 28A-28B A hybrid logical / physical view of a disaggregated parallel processor according to embodiments described herein is shown. Figure 28A A disaggregated parallel computing system 2800 is shown. Figure 28B A corelet 2830 of the disaggregated parallel computing system 2800 is shown.

[0357] As Figure 28AAs shown, the disaggregated computing system 2800 can include a parallel processor 2820 in which various components of the parallel processor SOC are distributed across multiple dies. Each die can be a different IP core that is independently designed and configured to communicate with other dies via one or more common interfaces. The dies include, but are not limited to, compute dies 2805, media dies 2804, and memory dies 2806. Each die can be fabricated separately using different process technologies. For example, compute dies 2805 can be fabricated using the smallest or most advanced process technology available at the time of fabrication, while memory dies 2806 or other dies (e.g., I / O, networking, etc.) can be fabricated using larger or less advanced process technologies.

[0358] Various dies can be connected to a base die 2810 and configured to communicate with each other and logic within the base die 2810 via an interconnect layer 2812. In one embodiment, the base die 2810 can include global logic 2801 that can include a dispatcher 2811 and power management 2821 logic units, interfaces 2802, dispatch units 2803, and an interconnect fabric module 2808 coupled or integrated with one or more L3 cache banks 2809A-2809N. The interconnect fabric 2808 can be an inter-die fabric integrated into the base die 2810. Logic dies can use the fabric 2808 to relay information between various dies. Additionally, L3 cache banks 2809A-2809N in the base die and / or L3 cache banks within the memory dies 2806 can cache data read from and transferred to DRAM dies within the memory dies 2806 and system memory of a host.

[0359] In one embodiment, the global logic 2801 is a microcontroller that can execute firmware to perform dispatcher 2811 and power management 2821 functionality of the parallel processor 2820. The microcontroller executing the global logic can be customized for a target use case of the parallel processor 2820. The dispatcher 2811 can perform global scheduling operations for the parallel processor 2820. The power management 2821 functionality can be used to enable or disable individual dies within the parallel processor when those dies are not in use.

[0360] Various dies of the parallel processor 2820 can be designed to perform specific functionality that would be integrated into a single die in existing designs. A set of compute dies 2805 can include clusters of compute units (e.g., execution units, streaming multiprocessors, etc.) that include programmable logic to execute compute or graphics shader instructions. Media dies 2804 can include hardware logic to accelerate media encode and decode operations. Memory dies 2806 can include volatile memory (e.g., DRAM) and one or more SRAM cache memory banks (e.g., L3 banks).

[0361] As shown in Figure 28B Each corelet 2830 can include common components and application specific components. Corelet logic 2836 within a corelet 2830 can include the specific components of the corelet, such as an array of streaming multi-processors, compute units, or execution units as described herein. Corelet logic 2836 can be coupled with or can contain a cache or shared local memory 2838. Corelet 2830 can include a fabric interconnect node 2842 that receives commands via an inter-corelet fabric. Commands and data received via fabric interconnect node 2842 can be temporarily stored in interconnect buffer 2839. Data transferred to and from fabric interconnect node 2842 can be stored in interconnect cache 2840. Power control 2832 and clock control 2834 logic can also be contained within a corelet. Power control 2832 and clock control 2834 logic can receive configuration commands via the fabric and can configure dynamic voltage and frequency scaling for corelet 2830. In one embodiment, each corelet can have independent clock and power domains and can be clock-gated and power-gated independently of other corelets.

[0362] At least a portion of the components shown within corelet 2830 can also be contained in logic embedded within base die 2810 of Figure 28A For example, logic within the base die that communicates with the fabric can include a version of fabric interconnect node 2842. Base die logic that can be independently clock or power gated can include a version of power control 2832 and / or clock control 2834 logic.

[0363] Figures 29A-29B A package view of a disaggregated parallel processor is shown in accordance with an embodiment. Figure 29A A physical layout of package components 2920 is shown. Figure 29B Interconnections between multiple corelets 2904, 2906 and interconnect fabric 2940 are shown.

[0364] As shown in Figure 29AAs shown, the package assembly 2920 can include a plurality of units of hardware logic dice connected to a substrate 2910 (e.g., a base die). The hardware logic dice can include application specific hardware logic dice 2902, logic or I / O dice 2904, and / or memory dice 2905. The hardware logic dice 2902 and logic or I / O dice 2904 can be implemented at least partially in configurable logic or fixed functionality logic hardware, and can include one or more portions of any of the processor core(s), graphics processor(s), parallel processor(s), or other accelerator devices described herein. The memory dice 2905 can be DRAM (e.g., GDDR, HBM) memory or cache (SRAM) memory, as described and illustrated herein.

[0365] Each die can be fabricated as a separate semiconductor die and coupled with the substrate 2910 via an interconnect structure 2903. The interconnect structure 2903 can be configured to route electrical signals between various dice and logic within the substrate 2910. The interconnect structure 2903 can include interconnects such as, but not limited to, bumps or pillars. In some embodiments, the interconnect structure 2903 can be configured to route electrical signals such as, for example, input / output (I / O) signals and / or power or ground signals associated with the operation of the logic, I / O, and memory dice.

[0366] In some embodiments, the substrate 2910 is an epoxy-based laminate substrate. In other embodiments, the substrate 2910 can include other suitable types of substrates. The package assembly 2920 can be connected to other electrical devices via a package interconnect 2914. The package interconnect 2914 can be coupled to a surface of the substrate 2910 to route electrical signals to other electrical devices such as a motherboard, other dice, or a multi-chip module.

[0367] In some embodiments, the logic or I / O dice 2904 and the memory dice 2905 can be electrically coupled via a bridge 2917 configured to route electrical signals between the logic or I / O dice 2904 and the memory dice 2905. The bridge 2917 can be a dense interconnect structure that provides routing for electrical signals. The bridge 2917 can include a bridge substrate composed of glass or a suitable semiconductor material. Electrical routing features can be formed on the bridge substrate to provide chip-to-chip connections between the logic or I / O dice 2904 and the memory dice 2905. The bridge 2917 can also be referred to as a silicon bridge or an interconnect bridge. For example, in some embodiments, the bridge 2917 is an embedded multi-die interconnect bridge (EMIB). In some embodiments, the bridge 2917 can simply be a direct connection from one die to another.

[0368] Substrate 2910 may include hardware components for I / O 2911, cache memory 2912, and other hardware logic 2913. Configuration 2915 may be embedded in substrate 2910 to enable communication between various logic chips and logic 2911, 2913 within substrate 2910.

[0369] In various embodiments, package assembly 2920 may include fewer or more components and dies interconnected via configuration 2915 or one or more bridges 2917. Dies within package assembly 2920 may be arranged in a 3D or 2.5D configuration. Generally, bridge configuration 2917 may be used to facilitate point-to-point interconnections, for example, between logic or I / O dies and memory dies. Configuration 2915 may be used to interconnect various logic and / or I / O dies (e.g., dies 2902, 2904, 2911, 2913) with other logic and / or I / O dies. In one embodiment, cache memory 2912 within the substrate may serve as a global cache of package assembly 2920, part of a distributed global cache, or as a dedicated cache of configuration 2915.

[0370] like Figure 29B As shown, memory chip 2906 can be connected to logic or I / O chip 2904 via chip interconnect 2935 routed through interconnect bridge 2947. Interconnect bridge 2947 can be embedded in, for example... Figure 29A The bridge 2917 in the substrate 2910 of the package assembly 2920 shown is a variation. The logic or I / O chip 2904 can communicate with other chips via interconnect structure 2940, which is... Figure 29A The deformation of the structure 2915.

[0371] In one embodiment, memory die 2906 includes a set of memory banks 2931 corresponding to the memory technology provided by the die. Memory banks 2931 can include any type of memory described herein, including but not limited to DRAM, SRAM, or flash memory, or 3D XPoint memory. Memory control protocol layer 2932 can allow control of memory banks 2931 and can include logic for one or more memory controllers. Interconnect bridge protocol layer 2933 can relay messages between memory control protocol 2932 and interconnect bridge I / O layer 2934. Interconnect bridge I / O layer 2934 can communicate with interconnect bridge I / O layer 2936 over die interconnect 2935. Interconnect bridge I / O layers 2934, 2936 can represent physical layers that transmit or receive signals to or from a corresponding interconnect point over die interconnect 2935. The physical I / O layers can include circuitry that drives signals over die interconnect 2935 and / or receives signals from die interconnect 2935. Interconnect bridge protocol layer 2937 within logic or I / O die 2904 can translate signals from interconnect bridge I / O layer 2936 into messages or signals that can be passed to compute or I / O logic 2939. In one embodiment, digital adapter layer 2938 can be used to facilitate translation of signals into messages or signals for use by compute or I / O logic 2939.

[0372] Compute or I / O logic 2939 can communicate with other logic or I / O dies via interconnect fabric 2940. In one embodiment, compute or I / O logic 2939 includes integrated fabric node logic 2939 that can communicate with interconnect fabric 2940, such as fabric interconnect node 2942 of Figure 28B

[0373] In one embodiment, control layer 2968 in memory die 2906 can communicate with control layer 2970 in logic or I / O die 2904. These control layers 2968, 2970 can be used to propagate or communicate certain control signals in an out-of-band manner, such as sending power and configuration messages between interface bus protocol layer 2937 of logic or I / O die 2904 and interface bus protocol layer 2933, memory control protocol 2932, and / or memory banks 2931 of memory die 2906.

[0374] Figure 30 ​A message transfer system 3000 for an interconnect fabric is shown in accordance with an embodiment. The message transfer system 3000 can be configured to handle traffic at different rates depending on the interface width available. The specific interface width can vary on a per corelet basis, and the speed or configuration of the fabric can be adjusted to enable data to be transferred at a rate appropriate for the various functional units to be interconnected. The transfer layer can also span one or more clock domains via the use of cross-clock domain FIFOs at the boundaries between clock domains, enabling data to be transferred across clock domains. The transfer layer can also be divided into one or more sub-layers, with each sub-layer comprising one or more clock domains. While a transfer layer is described and shown, in some embodiments the operations shown can be performed at the data link layer of the interconnect fabric.

[0375] In one embodiment, a first functional unit 3001A located in an origin layer 3010 can communicate with a second functional unit 3001B located in a destination layer 3013 via one or more transfer layers 3011, 3012. The origin layer 3010 can be logic or I / O within a package assembly (such as the package assembly 2920 in Figure 29A the substrate 2910) or within a corelet. The destination layer 3013 can also be logic or I / O within a corelet or within a substrate. In one embodiment, the origin layer 3010 and / or the destination layer 3013 can be associated with cache memory within a corelet or substrate / base die.

[0376] The one or more transfer layers 3011, 3012 can be within separate clock domains. For example, the transfer layer 3011 can be in a first clock domain, while the transfer layer 3012 can be in a second clock domain. The separate clock domains can operate at different frequencies. Data can be transferred between the clock domains via a clock crossing module 3003 within the transfer layer. In one embodiment, a first buffer or high-speed memory module 3004 within the clock crossing module 3003 can buffer data to be relayed to a second buffer or high-speed memory module 3006 via a cross- crossing FIFO 3005. The first buffer / high-speed memory module 3004 can be in a first clock domain, while the second buffer / high-speed memory module 3006 can be in a second clock domain.

[0377] The functional unit 3001A and the functional unit 3001B can transfer and receive messages to and from the fabric via respective fabric interfaces 3002A-3002B. The fabric interfaces 3002A-3002B can be dynamically configured within the fabric for the width of the connections relaying messages and signals across the transfer layers, as shown below. Figures 31-32

[0378] Figure 31 ​Multiple physical links across an interconnect fabric are shown to carry messages or signals between functional units. In one embodiment, where a communication channel between a set of functional units has bandwidth requirements that exceed what can be provided via a single physical link, multiple links can be used to facilitate communication between the functional units.

[0379] In one embodiment, a first functional unit 3101A can send a message or signal to a first fabric interface 3102A. The first fabric interface 3102A can distribute the message or signal and send the message or signal across multiple physical links as a single virtual channel. For example, multiple physical links can be assigned to the same virtual channel and each physical link can carry the message or signal for the virtual channel.

[0380] Data can be transferred between clock domains via a clock crossing module 3103 within one or more transport layers 3011, 3012 that employ, for example, a clocked FIFO as shown. Figure 30 Messages or signals are shown to be carried for a channel across multiple physical links via multiple buffers or high speed memories 3104A-3104B, 3106A-3106B (and clock domain crossing FIFOs) in a similar manner as shown. The multiple physical links can converge to a second fabric interface 3102B before the message or signal is provided to a second functional unit 3101B.

[0381] Figure 32 The transfer of messages or signals for multiple functional units across a single channel of an interconnect fabric is shown. In one embodiment, where a communication channel between a set of functional units has bandwidth requirements that do not utilize all of the available bandwidth of a physical link, multiple virtual channels can be transferred across the physical link. The virtual channels can be time switched along the physical link and a particular set of data lines in the physical link can be allocated to a particular functional unit.

[0382] In one embodiment, a first set of functional units 3201A, 3211A can communicate with a second set of functional units 3201B, 3211B over a single physical link of an interconnect fabric. The functional units 3201A-3201B can be associated with a first virtual channel and the functional units 3211A-3211B can be associated with a second virtual channel. The first and second virtual channels can converge at a first fabric interface 3202A. Messages or signals can be relayed across one or more transport layers 3011, 3012. For example, a clocked FIFO can be employed as shown. Figure 30Data is transferred in a similar manner shown across clock domains via a plurality of buffers or high speed memories 3204, 3106 (and clock domain spanning FIFOs) through clock spanning modules 3203 in one or more transport layers 3011, 3012. A plurality of virtual lanes can be dispersed at a second set of fabric interfaces 3202B before providing messages or signals to a second set of functional units 3201B, 3211B.

[0383] Figure 33 A method 3300 of configuring fabric connections for functional units within a disaggregated parallel processor is shown. The interconnect fabric of the disaggregated parallel processor described herein can be configured to transfer messages and / or signals for a variety of diverse components, which can be IP cores with different designers and / or manufacturers. The particular type of link used by each functional unit within a corelet or base die logic component can be configurable. In one embodiment, the link is configurable at the fabric interconnect node used by the functional unit.

[0384] In one embodiment, a fabric interconnect node can receive bandwidth configuration data for a functional unit to be configured to communicate via an interconnect fabric within a parallel processor package (block 3302). The configuration data can be provided statically during initial assembly and provisioning of the disaggregated parallel processor, or can be configured dynamically during initialization of the disaggregated parallel processor. For dynamic initialization, the fabric interconnect node can receive a bandwidth configuration request from the functional unit specifying a physical width and frequency of an interconnect between the functional unit and the fabric interconnect node and a bandwidth requirement of the functional unit.

[0385] Next, the fabric interconnect node can analyze the interconnect width and frequency configured for the functional unit (block 3304). The fabric interconnect node can then configure a convergence and / or divergence link for the functional unit (block 3306). Once configured, the fabric interconnect node can relay messages and / or signals for the functional unit across the configured link (block 3308).

[0386] Figure 34 A method 3400 of relaying messages and / or signals across an interconnect fabric within a disaggregated parallel processor is shown. The interconnect fabric of the disaggregated parallel processor described herein can relay messages and / or signals across one or more layers of the disaggregated parallel processor while spanning multiple clock domains.

[0387] In one embodiment, a first functional unit within a corelet or base die of a processor can generate data in the form of a message or signal to be transferred (block 3402). The first functional unit can transfer the message or signal to an interconnect fabric via a first fabric interface node (block 3404). The fabric interface node can converge or diverge virtual lanes into a physical transport link (block 3406), as described above with respect to FIG. 3. The fabric interface node can transmit the message or signal to a second fabric interface node (block 3408). The second fabric interface node can converge or diverge virtual lanes from a physical transport link (block 3410), as described above with respect to FIG. 3. The second fabric interface node can transfer the message or signal to a second functional unit (block 3412).Figure 31 and Figure 32 The messages or signals to be transmitted can be associated with virtual channels and transmitted via the associated virtual channels. How the fabric interconnect performs the forwarding and / or switching operations of the messages or signals can be influenced by the virtual channel assigned to the messages or signals. Additionally, multiple virtual channels can be merged into a single physical link, or a virtual channel can be carried by multiple physical links.

[0388] In one embodiment, the interconnect fabric can transmit messages or signals across multiple clock domains within one or more transport and / or data link layers (block 3408). One or more clock spanning modules (including high speed memory and domain spanning FIFOs) can be used to span the multiple clock domains. In one embodiment, each corelet can have an independent clock domain relative to the interconnect fabric. The interconnect fabric can also have multiple clock domains. Transmitting messages or signals across multiple clock domains can include switching the messages or signals using switching logic within the interconnect fabric.

[0389] The second fabric interface node can receive the messages or signals (block 3410). The second fabric interface node can then de-multiplex or multiplex virtual channels from the physical transport links at the second fabric interface node (block 3412). Multiple virtual channels can be de-multiplexed from a single physical link, or virtual channels can be multiplexed from multiple physical links. A second functional unit within a corelet or base die of the processor can then receive data in the form of messages or signals at the second hardware logic (block 3414). The second functional unit can then perform operations based on the received data.

[0390] Figure 35 A method 3500 of power gating corelets on a per-workload basis is shown. In one embodiment, power control logic within a disaggregated parallel processor can determine which corelets or logic units should be powered based on the requirements of a workload when the workload is executed. In one embodiment, the power control logic can work with other global logic, such as a global scheduler or front end interface, to determine which components will be used to process the workload.

[0391] The method 3500 includes receiving a command buffer for a workload to be executed on a parallel processor (block 3502). For example, the command buffer can be received at a global scheduler or front end interface. The method 3500 additionally includes determining a set of corelets to be used to execute the workload (block 3504). This determination can be performed by determining a global set and / or type of functional units that will be used to execute the commands within the command buffer.

[0392] Method 3500 additionally includes determining whether any functional corelets that are to be used to process the workload are resident on corelets that are power gated, and powering up those corelets if they are not already powered up (block 3506). Additionally, functional units that will not be used to process the workload can be determined. If not all of the functional units within a corelet will be used to process the workload, then the power control logic can power down (e.g., power gate) the corelets that are not used to process the workload (block 3508). To power up and power down corelets, the global power control logic can signal the local power control logic within the corelets. The local power control logic within the corelets can then perform the appropriate power down sequence for the corelets. The disaggregated parallel processor can then execute the workload using the powered up (e.g., active) corelets (block 3510).

[0393] Enabling product SKUs based on die configuration

[0394] Semiconductor dies are tested during the manufacturing process to evaluate the integrated circuits formed on the dies. Standard testing of overall functionality can be performed by probing the dies on a wafer. Burn-in testing can be performed after the dies have been singulated and packaged, or using a test fixture of bare dies. Defective dies can be discarded. However, dies that pass initial testing but fail subsequent testing at the highest frequency can function normally at a lower frequency. This binning process can select dies that are less or more performant and target these dies as higher or lower performance products with different stock keeping units (SKUs). For a single-chip system-on-a-chip, the binning process is a coarse-grained process. Although a processor with a small number of defective compute or graphics cores can be binned down, a minimum number of defect-free components must be present to meet the minimum product requirements.

[0395] Using the disaggregated SoC architecture described herein, individual corelets can be tested and binned at the chip level, and can be determined at the SKU level for a product during assembly based on the requirements of a given product SKU. During assembly, different amounts of memory, different functionality, and different performance can be assembled for different product SKUs with different configurations by specifying particular corelets or different bins of the same corelet design.

[0396] A disaggregated processor package can be configured to accept interchangeable dielets. Interchangeability can be achieved by specifying a standard physical interconnect for the dielets, which can enable the dielets to connect with fabric or bridge interconnects. Dielets from different IP designers can conform to a common interconnect, enabling such dielets to be interchangeable during assembly. Fabric and bridge interconnect logic on the dielets can then be configured to confirm the actual interconnect layout of the dielets' on-board logic. Additionally, encapsulation can be used to pass data from the dielets across the inter-fabric to enable the actual data transmitted to be opaque to the fabric, further enabling interchangeability of individual dielets. Using such interchangeable designs, higher or lower density memory can be inserted into memory dielet slots, while compute or graphics dielets with higher or lower core counts can be inserted into logic dielet slots.

[0397] Functionality can also be determined during assembly. For example, media dielets can be added or excluded based on product specifications and requirements. In some products, networking or other communication dielets can be added to the package. In some products, different dielets can be used to enable different types of host connections. For example, where a host interconnect version change involves changes to interconnect logic without involving changes to the physical form factor, upgrading to a new interconnect version can be performed by changing the host interconnect dielet during assembly, without requiring redesign of the SoC to insert the new interconnect logic into a monolithic die.

[0398] Dielet binning can be further enabled by providing dielet test fixtures that conform to the standardized chassis interface. The dielet test fixtures can enable rapid testing and binning of different SKUs of dielets.

[0399] Figure 36 A parallel processor assembly 3600 including interchangeable dielets 3602 is shown. The interchangeable dielets 3602 can be assembled into standardized slots on one or more base dielets 3604, 3608. The base dielets 3604, 3608 can be coupled via a bridge interconnect 3606, which can be similar to other bridge interconnects described herein. Memory dielets can be connected to logic or I / O dielets via the bridge interconnect. I / O and logic dielets can communicate via an interconnect fabric. The base dielets can each support one or more slots of a standardized format for logic or I / O or memory / cache. Different memory densities can be assembled into the dielet slots based on the target SKU of the product. Additionally, logic dielets with different numbers of types of functional units can be selected at assembly based on the target SKU of the product. Additionally, dielets containing different types of IP logic cores can be inserted into the interchangeable dielet slots.

[0400] Figure 37An interchangeable corelet system 3700 is shown in accordance with an embodiment. In one embodiment, the interchangeable corelet system 3700 includes at least one base corelet 3710 that includes a plurality of memory corelet slots 3701A-3701F and a plurality of logic corelet slots 3702A-3702F. Logic corelet slots (e.g., 3702A) and memory corelet slots (e.g., 3701A) can be connected by an interconnect bridge 3735, which can be similar to other interconnect bridges described herein. The logic corelet slots 3702A-3702F can be interconnected via a fabric interconnect 3708. The fabric interconnect 3708 includes switching logic 3718, which can be configured to relay data packets between logic corelet slots in a data-agnostic manner by encapsulating the data packets into fabric packets. The fabric packets can then be switched to a destination slot within the fabric interconnect 3708.

[0401] The fabric interconnect 3708 can include one or more physical data lanes. One or more programmable virtual lanes can be carried by each physical lane. The virtual lanes can be independently arbitrated, with each virtual lane individually negotiating lane access. Traffic on a virtual lane can be classified into one or more traffic classes. In one embodiment, a priority system allows for relative priorities to be assigned to virtual lanes and traffic classes for arbitration. In one embodiment, a traffic balancing algorithm operates to maintain substantially equal bandwidth and throughput to each node coupled to the fabric. In one embodiment, the fabric interconnect logic operates at a higher clock rate than the nodes coupled to the fabric to allow for reduced interconnect width while maintaining bandwidth requirements between nodes. In cases where higher bandwidth is required at certain nodes, multiple physical links can be combined to carry a single virtual lane, as described above in Figure 31

[0402] Figure 38 is an illustration of a plurality of traffic classes carried on a virtual lane in accordance with an embodiment. A first fabric connector 3802 and a second fabric connector 3804 facilitate communication over a fabric lane 3806 having up to "M" virtual lanes 3806A-3806M. Virtual lanes enable the transmission of variable length information over a fixed set of physical lanes. Virtual lanes can be permanent virtual lanes, or virtual lanes can be dynamically enabled or disabled based on system configuration. The use of permanent virtual lanes allows for a fixed lane ID, which minimizes the overhead of virtual lane management. Dynamically configuring lanes increases design flexibility at the expense of additional lane management overhead.

[0403] ​Multiple traffic classes can be assigned to each virtual lane. Traffic classes are a division of traffic that is relevant to arbitration. Each virtual lane can carry up to "N" traffic classes. Each traffic class is assigned to a particular virtual lane through programming (fuses, configuration registers, etc.). Up to "L" traffic classes of traffic types can be assigned to a given virtual lane.

[0404] # Traffic classes Virtual lanes 1 Class 1 1 2 Class 2 0 3 Class 3 M 4 Class 4 1 5 Class 5 0 … N Class N 2

[0405] Table 5 above shows an exemplary traffic class to virtual lane assignment as shown in Table 5. The fabric interconnect classifies each cell of incoming traffic and can include logic to ensure that the incoming cell travels within its assigned virtual lane. In one embodiment, data transfer on a lane occurs in a first-in-first-out (FIFO) order, and lane arbitration is based on virtual lanes. Traffic within a virtual lane can block the transfer of additional traffic on the same virtual lane. However, a given virtual lane will not block a different virtual lane. Thus, traffic on different virtual lanes is independently arbitrated. Figure 38

[0406] Consistency is maintained during data transfer between fabric interconnect nodes. In one embodiment, data from an initiating thread on a GPGPU or parallel processor is routed in the same traffic class and the traffic class is assigned to a particular virtual lane. Data in a single traffic class on a single virtual lane is transferred in a FIFO order. Thus, data from a single thread is strictly ordered when transferred via the fabric and per-thread consistency is maintained to avoid the risk of read-after-write or write-after-read data. In one embodiment, thread group consistency is maintained via a global synchronization mechanism between resource nodes.

[0407] # Traffic classes Priority 1 Class 1 2 2 Class 2 1 3 Class 3 4 4 Class 4 2 5 Class 5 1 … N Class N 3

[0408] Table 6 above shows exemplary traffic class prioritization. A priority algorithm can be programmed to determine the priority assigned to each traffic class. Programmable traffic class priority allows the use of traffic classes as an arbitrary traffic grouping mechanism, where traffic can simply be grouped in a class to maintain consistency, or specific traffic can be assigned a high priority and dedicated to high priority data. For example, class 1 and class 4 are each assigned to virtual lane 1 3806B, they can be assigned a priority of 2. Class 2 and class 5 are each assigned to virtual lane 0 3806A, they can be assigned a priority of 1. Traffic class "N" can be assigned to virtual lane 2 3806C, which has a priority of 3. Traffic in class 2 can be latency sensitive data that should be transferred as soon as possible or not be blocked by other traffic classes, while traffic in class 1 can be moderately latency sensitive traffic from a single thread that is grouped to maintain consistency. ​

[0409] Traffic classes can be assigned priorities relative to all traffic classes or relative to the priority of traffic classes on the same virtual channel. In one embodiment, the priority scheme is designed by assigning weights to traffic classes, where a higher weight indicates a higher priority. A fair priority algorithm can be employed, where each participant is guaranteed a minimum amount of bandwidth to prevent starvation. In one embodiment, an absolute priority algorithm is used in certain situations, where higher priority traffic always blocks lower priority traffic.

[0410] In the case of using absolute priority, additional algorithms can be employed to appropriately prevent communication deadlock. The combined use of virtual channels and traffic classes reduces the likelihood of deadlock, since a single traffic class with absolute priority on a given virtual channel does not block traffic on a different virtual channel. In one embodiment, if a starvation condition or potential deadlock is detected on one virtual channel, the blocked traffic class can be reassigned to a different virtual channel.

[0411] # Virtual lanes Priority 1 1 2 2 2 1 3 3 3 … M M 4

[0412] Table 7 above shows exemplary virtual channel priorities. As with traffic classes, each virtual channel can also receive a priority, and channel arbitration can take into account the relative priority of virtual channels. For example, data traffic on virtual channel 2 can have a higher relative priority than data on other virtual channels. A weighted priority system can be employed with virtual channel priorities, where a higher weight indicates a higher priority. A fair priority system or an absolute priority system can be employed.

[0413] Figure 39 A method 3900 for agnostic data transfer between slots of interchangeable corelets is shown, in accordance with an embodiment. The method 3900 can be performed by hardware logic in the fabric interconnect and fabric interconnect nodes described herein. In one embodiment, the method 3900 includes causing a first fabric interface node to receive data from a first corelet logical slot (block 3902). The first fabric interface node can encapsulate the data into a fabric packet (block 3904). The first fabric interface node can then transmit the packet to a second fabric interface node via switch logic within the fabric interconnect. The second fabric interface node can receive the packet (block 3908) and decapsulate the data from the packet (block 3910). The second fabric interface node can then transmit the data decapsulated from the packet to a second corelet logical slot (block 3912).

[0414] Figure 40A modular architecture for interchangeable corelets is shown. In one embodiment, corelet design 4030 can be made interchangeable by adapting corelet logic 4002 to interoperate with interface template 4008. Interface template 4008 can include standardized logic such as power control 2832 and clock control 2834 logic, as well as interconnect buffers 2839, interconnect cache 2840, fabric interconnect nodes 2842, as in corelet 2830 of Figure 28B IP designer can then provide corelet logic 4002 that is designed to interface with interface template 4008. The specifics of corelet logic 4002 can vary and can include the execution units, compute units, or streaming multiprocessors described herein. Corelet logic 4002 can also include media encode and / or decode logic, matrix acceleration logic, or ray tracing logic. For a memory corelet, corelet logic 4002 can be replaced with memory units, and fabric interconnect nodes can be replaced with interconnect bridge I / O circuitry, e.g., as shown in memory corelet 2906 of Figure 29B

[0415] Figure 41 A use of a standardized chassis interface for implementing corelet testing, validation, and integration is shown. Corelet 4130 can include logic layer 4110 and interface layer 4112, similar to corelet 4030 of Figure 40 Interface layer 4112 can be a standardized interface that can communicate with temporary interconnect 4114 that enables corelet to be removably coupled to test fixture 4116. Test fixture 4116 can communicate with test host 4118. Under the communication of test host 4118, test fixture 4116 can perform a series of tests on individual corelet 4130 during an initial testing or binning process to check for defects within logic layer 4110 and to determine a performance or functional bin for corelet 4130. For example, logic layer 4110 can be tested to determine the number of functional units that are defective and non-defective and whether a threshold number of special functional units (e.g., matrix accelerators, ray tracing cores, etc.) are functional. Logic layer 4110 can also be tested to determine whether the internal logic can operate at a target frequency.

[0416] Figure 42 A use of individually binned corelets to create various product tiers is shown. A set of untested corelets 4202 can be tested and binned into a set of bins, including performance bin 4204, mainstream bin 4206, and economy bin 4208, depending on whether the individual corelets meet a particular performance or functional tier. Performance bin 4204 can include corelets that exceed the performance (e.g., stable frequency) of mainstream bin 4206, while economy bin 4208 can include corelets that are functional but have a performance that is lower than that of mainstream bin 4206. ​

[0417] Because the corelets can be interchangeably placed during assembly, different product tiers can be assembled based on a selected set of corelets. Tier 1 products 4212 can be assembled from only corelets in the performance bin 4204, while tier 2 products 4214 can include corelets selected from the performance bin 4204 and other corelets selected from the mainstream bin 4206. For example, tier 2 products 4214 designed for workloads that require high bandwidth, low latency memory can use high performance memory corelets from the performance bin 4204, while using compute, graphics, or media corelets from the mainstream bin. Additionally, tier 3 products 4216 can be assembled using mainstream compute corelets from the mainstream bin 4206 and memory from the economy bin 4208 if such products are tailored for workloads that do not have high memory bandwidth requirements. Tier 4 products 4218 can be assembled from the economy bin 4208 of corelets that work but perform poorly.

[0418] Figure 43 A method 4300 of enabling different product tiers based on corelet configuration is shown. The method 4300 includes packaging corelet dies into a test package (block 4302). The corelets can then be tested to bin the corelets based on frequency and / or number of functional units (block 4304). The disaggregated parallel processor can then be assembled using corelets from one or more bins based on product requirements (block 4306). Additional corelets can also be added based on functional requirements (e.g., media, ray tracing, etc.) (block 4308).

[0419] The following clauses and / or examples pertain to specific embodiments or examples. Specific details can be used therin in order to provide a thorough description of the embodiments. The specific details can be used in one or more embodiments without limitation on the scope of the disclosure. Various features that are, or can be, combined together in any of the examples can be used in any of the examples. Examples can include subjects such as methods, components for carrying out the actions of the methods, at least one machine-readable medium comprising instructions that, when executed by a machine, cause the machine to perform actions of the methods according to the embodiments and examples described herein, or devices or systems. Various components can be used as means for performing the operations or functions described.

[0420] Embodiments described herein provide techniques for disaggregating the architecture of a system-on-a-chip integrated circuit into a plurality of different corelets that can be packaged onto a common chassis. In one embodiment, a graphics processing unit or parallel processor is composed of a diverse set of silicon corelets that are manufactured separately. Corelets are at least partially packaged integrated circuits that include different units of logic that can be assembled with other corelets into a larger package. A diverse set of corelets with different IP core logic can be assembled into a single device.

[0421] One embodiment provides a general purpose graphics processor comprising: a base die comprising an interconnect fabric; and one or more cores coupled with the base die and the interconnect fabric via an interconnect structure, the interconnect structure enabling electrical communication between the one or more cores and the interconnect fabric. The one or more cores can comprise a first core and a second core, where the first core is coupled with the base die and connected to the interconnect fabric via a first interconnect structure, and the second core is coupled with the base die and connected to the interconnect fabric via a second interconnect structure. The cores can comprise functional units configured to perform general purpose graphics processing operations, media encode or decode operations, matrix operation acceleration, and / or ray tracing. In one embodiment, the cores comprise network processors and physical network interfaces (e.g., network ports, wireless radios, etc.). The cores can additionally comprise memory, which can be cache memory or DRAM. Each core can be individually and independently power gated. Additionally, logic or memory can be included in the base die. In one embodiment, the base die comprises cache memory. The cache memory in the base die can be a processor-wide cache. The base die cache memory can be configured to work with cache memory in the cores.

[0422] One embodiment provides a data processing system comprising a general purpose graphics processor, the general purpose graphics processor comprising: a base die comprising an interconnect fabric; and a plurality of cores coupled with the base die and the interconnect fabric via a plurality of interconnect structures, the plurality of interconnect structures enabling electrical communication between the plurality of cores and the interconnect fabric, where the interconnect fabric is to receive a message or signal from a first fabric interface node associated with a first core of the plurality of cores and relay the message or signal to a second fabric interface node associated with a second core of the plurality of cores. The interconnect fabric can communicate the message or signal via a plurality of virtual channels over a plurality of physical links of the interconnect fabric. In one embodiment, a plurality of virtual channels can be communicated across a single physical link. In one embodiment, a single virtual channel can be communicated across a plurality of physical links. Idle physical links can be individually power gated.

[0423] One embodiment provides a method comprising: generating data at a first functional unit within a corelet or base die of a processor; transmitting the data to an interconnect fabric via a first fabric interface node; transferring the data across a plurality of clock domains within the processor; receiving the data at a second fabric interface node; transmitting the data to a second functional unit within the corelet or base die of the processor; and performing an operation at the second functional unit based on the received data. The method additionally comprises: associating the data with a virtual lane of the interconnect fabric; and forwarding or switching the data based on the virtual lane. In further embodiments, the method comprises: diverging the virtual lane at the first fabric interface node; transmitting data of the virtual lane across the plurality of clock domains using a plurality of physical links; and converging the virtual lane at the second fabric interface node. In yet another embodiment, the virtual lane is a first virtual lane, and the method additionally comprises: converging the first virtual lane at the first fabric interface node with a second virtual lane; transmitting the first virtual lane and the second virtual lane across the plurality of clock domains using a single physical link; and diverging the first virtual lane at the second virtual lane of the second fabric interface node.

[0424] One embodiment provides a non-transitory machine-readable medium storing firmware of a microcontroller within a processor having a disaggregated architecture, the firmware comprising instructions that cause the microcontroller to perform operations comprising: receiving a command buffer that causes a workload to be executed on the processor; determining a set of corelets on the processor that include functional units to execute the workload; power gating one or more corelets that do not include functional units to execute the workload; and executing the workload using the powered corelets. The operations can additionally comprise: determining whether a corelet that includes a functional unit to execute the workload is powered; and powering the corelet when the corelet is power gated.

[0425] Further embodiments provide a disaggregated processor package configurable to accept interchangeable corelets. Interchangeability can be achieved by specifying a standard physical interconnect for the corelets, which can enable the corelets to connect with fabric or bridge interconnects. Corelets from different IP designers can conform to a common interconnect to enable such corelets to be interchangeable during assembly. Fabric and bridge interconnect logic on the corelets can then be configured to confirm the actual interconnect layout of the on-board logic of the corelets. Additionally, packets can be used to transport data from the corelets across the inter-core fabric, such that the actual data of the transmission is opaque to the fabric, further enabling interchangeability of individual corelets. Using such interchangeable designs, higher or lower density memory can be inserted into memory corelet slots, while compute or graphics corelets with higher or lower core counts can be inserted into logic corelet slots.

[0426] One embodiment provides a general purpose graphics processor, the processor comprising: a base die including an interconnect fabric; and one or more tiles coupled with the base die and the interconnect fabric via an interconnect structure, the interconnect structure enabling electrical communication between the one or more tiles and the interconnect fabric, wherein the one or more tiles are interchangeable during assembly of the general purpose graphics processor. The one or more tiles include a memory tile having memory cells associated with a memory device. The memory tile is coupled to a first memory tile slot. The one or more tiles can additionally include a first logic tile and a second logic tile. The first logic tile can be coupled with the base die and connected to the interconnect fabric via a first interconnect structure. The first interconnect structure is bonded to a first logic tile slot. The second logic tile is coupled with the base die and connected to the interconnect fabric via a second interconnect structure. The second interconnect structure is bonded to a second logic tile slot.

[0427] In one embodiment, the first logic tile slot is configured to accept either the first logic tile or a third logic tile. The first logic tile includes functional units (e.g., execution units, compute units, streaming multi-processors, etc.) configured to perform general purpose graphics processing operations, while the third logic tile includes functional units configured to perform matrix acceleration operations, such as a tensor core. The second logic tile slot can be configured to accept either the second logic tile or a fourth logic tile. The second logic tile includes functional units configured to perform media operations described herein to encode media, decode media, or transcode media between one or more media encoding formats. The fourth logic tile, instead, can include a network processor and a physical network interface. Each tile can be independently power gated.

[0428] In one embodiment, the logic tiles described herein include a first layer including functional units and a second layer including fabric interconnect nodes. The memory tiles described herein can include a first layer including a bank of memory cells and a second layer including I / O circuitry associated with an interconnect bridge between the memory tiles and the logic tiles. In further embodiments, the base die is a first base die, and the first base die is coupled with a second base die via an interconnect bridge.

[0429] One embodiment provides a data processing system comprising a general purpose graphics processor comprising: a base die comprising an interconnect fabric; and a plurality of corelets coupled with the base die and the interconnect fabric via a plurality of interconnect structures. The plurality of interconnect structures enable electrical communication between the plurality of corelets and the interconnect fabric. The interconnect fabric can receive a fabric packet from a first fabric interface node associated with a first logical corelet of the plurality of corelets and relay the fabric packet to a second fabric interface node associated with a second logical corelet of the plurality of corelets. The interconnect fabric can transmit the fabric packet via a plurality of virtual lanes over a plurality of physical links of the interconnect fabric. The interconnect fabric can transmit the fabric packet associated with a single virtual lane across a plurality of physical links of the interconnect fabric. The interconnect fabric can also transmit the fabric packet for a plurality of virtual lanes across a single physical link of the interconnect fabric. A physical link of the plurality of physical links can be power gated when the physical link is idle. In one embodiment, the fabric packet of a virtual lane can be associated with one or more traffic classes. The virtual lanes and the traffic classes can each have an associated priority.

[0430] One embodiment provides a method comprising: receiving data at a first fabric interface node from a first logical corelet slot; packetizing the data into a fabric packet at the first fabric interface node; transmitting the fabric packet to a second fabric interface node via switch logic; receiving the packet at the second fabric interface node; de-packetizing the data from the fabric packet at the second fabric interface node; and transmitting the data from the fabric packet from the second fabric interface node to a second logical corelet slot. The fabric packet can traverse a plurality of clock domains between the first fabric interface node and the second fabric interface node.

[0431] Of course, one or more parts of an embodiment can be implemented using different combinations of software, firmware, and / or hardware. Throughout this detailed description, for the purposes of explanation, numerous specific details were set forth in order to provide a thorough understanding of the present application. It will be apparent, however, to one skilled in the art that the embodiments can be practiced without some or all of these specific details. In certain instances, well known structures and functions were not described in elaborate detail in order to avoid obscuring the subject matter of the present embodiments. Thus, the scope and spirit of the present application should be judged in terms of the appended claims and their equivalents.

Claims

1. A computing device, comprising: The packaged assembly includes multiple chips and multiple interconnect structures, the multiple chips comprising: The first core particle includes a first substrate core particle coupled to a bridge interconnect and an interconnect structure, the first substrate core particle comprising: Interconnect architecture, and The first is a multi-level 3 (L3) cache library used to cache data read from memory and transferred to memory; and The second core particle includes a second substrate core particle, which is interconnected and coupled to the first core particle via the bridge. The third core includes a second plurality of L3 cache libraries, which are stacked on the first substrate core in a 3D arrangement and coupled to the first substrate core through the interconnect structure. The second core includes an input / output I / O core for coupling to one or more additional cores, and the second core is used for interconnecting with the third core via the bridge. Both the first substrate core and the second substrate core are formed in a common substrate, and The substrate includes a plurality of hardware components for the first to the third core.

2. The apparatus of claim 1, wherein, The second multiple level 3 cache library is the SRAM cache library.

3. The apparatus of claim 1, wherein, The first core chip further includes: One or more execution units for executing instructions, the one or more execution units being interconnected and coupled to the second core via the bridge.

4. The apparatus according to any one of claims 1-3, wherein, One or more of the plurality of cores further include: Power control logic is used to configure dynamic voltage and frequency scaling for the core.

5. The apparatus of claim 4, wherein, Each of the plurality of chips has an independent clock domain and an independent power domain.

6. The apparatus according to any one of claims 1-3, wherein, At least one of the plurality of cores includes a plurality of processing cores.

7. The apparatus according to any one of claims 1-3, wherein, The third core particle is bonded to the first substrate core particle.

8. The apparatus of claim 6, wherein, The third chip is a memory chip.

9. A computing system, comprising: Memory chip; as well as A processor includes a packaging assembly comprising a plurality of chips and a plurality of interconnect structures, the plurality of chips comprising: The first core particle includes a first substrate core particle coupled to a bridge interconnect and an interconnect structure, the first substrate core particle comprising: Interconnect architecture, and The first is a multi-level 3 (L3) cache library used to cache data read from memory and transferred to memory; and The second core particle includes a second substrate core particle, which is interconnected and coupled to the first core particle via the bridge. The third core includes a second plurality of L3 cache libraries, which are stacked on the first substrate core in a 3D arrangement and coupled to the first substrate core through the interconnect structure. The second core includes an input / output I / O core for coupling to one or more additional cores, and the second core is used for interconnecting with the third core via the bridge. Both the first substrate core and the second substrate core are formed in a common substrate, and The substrate includes a plurality of hardware components for the first to the third core.

10. The system of claim 9, wherein, The second L3 cache library is an SRAM cache library.

11. The system of claim 9, wherein, The first core chip further includes: One or more execution units for executing instructions, the one or more execution units being interconnected and coupled to the second core via the bridge.

12. The system as described in any one of claims 9-11, wherein, One or more of the plurality of cores further include: Power control logic is used to configure dynamic voltage and frequency scaling for the core.

13. The system of claim 12, wherein, Each of the plurality of chips has an independent clock domain and an independent power domain.

14. The system as claimed in any one of claims 9-11, wherein, At least one of the plurality of cores includes a plurality of processing cores.

15. The system as described in any one of claims 9-11, wherein, The third core particle is bonded to the first substrate core particle.

16. The system of claim 14, wherein, The third chip is a memory chip.

Images