Fastpath CPU mode for mixed XPU workloads

Abstract

One embodiment provides a processor device comprising a general-purpose processor including circuitry for providing a swap mode for executing a swapped instruction, and an accelerator device configured to: execute a first portion of a parallel data workload, wherein at least a second portion of the parallel data workload has a dependency on a scalar operation, and wherein the scalar operation has a dependency on the first portion of the parallel data workload; cause the general-purpose processor to enter the swap mode after completing at least the first portion of the parallel data workload, wherein the general-purpose processor enters the swap mode to perform the scalar operation; and execute the second portion of the parallel data workload based at least in part on an output of the scalar operation.

Description

CROSS-REFERENCE

[0001] This application claims priority from U.S. Provisional Application No. 63 / 565,120, filed March 14, 2024, and from U.S. Patent Application No. 18 / 916,670, filed October 15, 2024, which are hereby incorporated by reference. BACKGROUND OF REVELATION

[0002] As compute workloads become increasingly heterogeneous, they are becoming mixtures of highly parallel computations interspersed with scalar compute components. This leads to challenges with the communication overhead between compute engines, such as a GPU and a CPU, that must cooperate in such workloads. The communication overhead for heterogeneous workloads can be particularly problematic in XPU compute architectures, which are processors that incorporate a variety of hardware architectures that can be used to run heterogeneous workloads. BRIEF DESCRIPTION OF THE DRAWINGS

[0003] The embodiments described herein are illustrated by way of example and not limitation in the figures of the accompanying drawings, in which like reference numerals indicate similar elements, and in which: Fig.1 is a block diagram illustrating a computer system configured to implement one or more aspects of the embodiments described herein; the Fig. 2A to 2E illustrate parallel processor components, including graphics multiprocessors; Fig. Figure 3 shows a graphics processing unit (GPU) containing dedicated sets of graphics processing resources arranged in multi-core groups; the Fig. 4A through 4E illustrate an example architecture in which a plurality of GPUs are communicatively coupled to a plurality of multi-core processors; Fig. 5 illustrates a graphics processing pipeline; Fig. Figure 6 illustrates a machine learning software stack; Fig. 7 illustrates a universal graphics processing unit; Fig. Figure 8 illustrates a multi-GPU computing system; the Fig. 9A and Fig. 9B illustrate layers of example deep neural networks; the Fig. 10A and Fig. 10B illustrate example language models; Fig. 11 illustrates training and deployment of a deep neural network; Fig. 12A is a block diagram illustrating distributed learning; Fig. 12B is a block diagram illustrating a programmable NIC and a programmable DPU; Fig. 13 illustrates an exemplary inference system-on-chip (inference SOC); Fig. 14 is a block diagram of a processing system; the Fig. 15A to 15C illustrate computer systems and graphics processors; Fig. 16 is a block diagram of a discrete or integrated graphics processor; the Fig. 17A and Fig. 17B illustrate block diagrams of additional graphics and compute accelerators; the Fig. 18A to 18C illustrate thread execution logic employed in a graphics processor core; Fig. 19 illustrates a tile of a multi-tile processor according to one embodiment; Fig. 20 is a block diagram illustrating a graphics processor instruction format; Fig. 21 is a block diagram of an additional graphics processor architecture; the Fig. 22A and Fig. 22B illustrates a graphics processor command format and a graphics processor command sequence; Fig. 23 illustrates an exemplary software architecture for a data processing system; Fig. 24 is a block diagram illustrating an IP core development system; Fig.25A illustrates a side cross-sectional view of an integrated circuit package assembly; Fig. 25B illustrates a package assembly including replaceable chiplets; Fig. 26 is a block diagram illustrating an exemplary system-on-chip integrated circuit; Fig. Figure 27 illustrates a CPU mode transition for GPU swapping; Fig. Figure 28 illustrates a fast path CPU mode for mixed XPU workloads. Fig. 29 illustrates a processor 2900 having a CPU and a GPU according to one embodiment; Fig. 30 illustrates a chiplet-based processor having CPU, GPU, and accelerator execution resources according to one embodiment; Fig. Figure 31 illustrates a graph-based path-finding algorithm computed via an outsourced operation; Fig. 32 illustrates a method for performing fast path CPU swapping; and Fig. 33 is a block diagram of a computing device including a graphics processor, according to one embodiment. DETAILED DESCRIPTION

[0004] To increase performance, graphics processors typically implement processing techniques such as pipeline processing, which attempt to process as much graphics data as possible in parallel across different parts of the graphics pipeline. Parallel graphics processors with single-instruction multithreaded (SIMT) architectures are designed to maximize the amount of parallel processing in the graphics pipeline. In a SIMT architecture, groups of parallel threads attempt to execute program instructions synchronously together as often as possible to increase processing efficiency.

[0005] A graphics processing unit (GPU) is communicatively coupled to host processor cores to accelerate, for example, graphics operations, machine learning operations, pattern analysis operations, and / or various general-purpose GPU (GPGPU) functions. The GPU may be communicatively connected to the host processor cores via a bus or other interconnect (e.g., a high-speed interconnect such as PCIe or NVLink). Alternatively, the GPU may be integrated into the same package or die as the cores and communicatively coupled to the cores via an internal processor bus / interconnect (i.e., within the package or die). Regardless of how the GPU is connected, the processor cores may allocate work to the GPU in the form of sequences of commands / instructions contained in a work descriptor.The GPU then uses dedicated circuitry to efficiently process these commands.

[0006] Although some techniques described here are discussed primarily in the context of a GPU, the techniques may also be implemented in other types of processors, including, but not limited to, general-purpose processors and accelerator devices such as artificial intelligence accelerators, image processors, and neural processing units.

[0007] In the following detailed description, numerous specific details are set forth in order to provide a more thorough understanding. However, it will be apparent to one skilled in the art that the embodiments described herein may be practiced without one or more of these specific details. In other examples, well-known features have not been described in order not to obscure the details of the present embodiments. System overview

[0008] Fig.1 is a block diagram illustrating a computing system 100 configured to implement one or more aspects of the embodiments described herein. Computing system 100 includes a processing subsystem 101 having one or more processors 102 and a system memory 104 communicating via an interconnect path that may include a memory hub 105. Memory hub 105 may be a separate component within a chipset component or may be integrated with the one or more processors 102. Memory hub 105 is coupled to an I / O subsystem 111 via a communications link 106. I / O subsystem 111 includes an I / O hub 107 that may enable computing system 100 to receive input from one or more input devices 108.The I / O hub 107 may additionally enable a display controller, which may be included in the one or more processors 102, to provide outputs to one or more display devices 110A. In one embodiment, the one or more display devices 110A connected to the I / O hub 107 may comprise a local, internal, or embedded display device.

[0009] For example, the processing subsystem 101 includes one or more parallel processors 112 coupled to the storage hub 105 via a communication link 113, such as a bus or fabric. The communication link 113 may be one of any number of standards-based communication link technologies or protocols, such as, but not limited to, PCI Express, or it may be a vendor-specific communication interface or fabric. The one or more parallel processors 112 may form a compute-focused parallel or vector processing system that may include a large number of processing cores and / or processing clusters, such as an integrated multi-core processor (MIC processor).For example, the one or more parallel processors 112 form a graphics processing subsystem that can output pixels to one or more display devices 110A coupled via the I / O hub 107. The one or more parallel processors 112 may also include a display controller (not shown) and a display interface (not shown) to enable direct connection to one or more display devices 110B.

[0010] Within the I / O subsystem 111, a system memory unit 114 may be connected to the I / O hub 107 to provide a storage mechanism for the computing system 100. An I / O switch 116 may be used to provide an interface mechanism to enable connections between the I / O hub 107 and other components, such as a network adapter 118 and / or a wireless network adapter 119, that may be integrated into the platform, and various other devices that may be added via one or more add-in devices 120. The add-in device(s) 120 may also include, for example, one or more external graphics processing devices, graphics cards, and / or compute accelerators. The network adapter 118 may be an Ethernet adapter or other wired network adapter.The wireless network adapter 119 may include one or more of a Wi-Fi, Bluetooth, near field communication (NFC), or other network device having one or more wireless radios.

[0011] The computer system 100 may include other components not explicitly shown, including USB or other connectors, optical storage drives, video capture devices, and the like, which may also be connected to the I / O hub 107. The communication paths connecting the various components in Fig.1 can be implemented using any suitable protocols, such as PCI (Peripheral Component Interconnect)-based protocols (e.g., PCI Express) or any other bus or point-to-point communication interfaces and / or protocols, such as the NVLink high-speed interconnect, Compute Express Link™ (CXL™) (e.g., CXL.mem), Infinity Fabric (IF), Ethernet (IEEE 802.3), Remote Direct Memory Access (RDMA), InfiniBand, Internet Wide Area RDMA Protocol (iWARP), Transmission Control Protocol (TCP), User Datagram Protocol (UDP), quick UDP Internet Connections (QUIC), RDMA over Converged Ethernet (RoCE), Intel QuickPath Interconnect (QPI), Intel Ultra Path Interconnect (UPI), Intel On-Chip System Fabric (IOSF), OmniPath, HyperTransport, AMBA (Advanced Microcontroller Bus Architecture) Interconnect, OpenCAPI, Gen-Z, Cache Coherent Interconnect for Accelarators (CCIX), 3GPP Long Term Evolution (LTE) (4G), 3GPP 5G, and variations thereof, or wired or wireless interconnect protocols known in the art. In some examples, data may be transferred using a protocol such as NVMe (Non-Volatile Memory Express).NVMe-oF (Non-Volatile Memory Express over Fabrics) or NVMe, can be copied or stored in virtualized data storage nodes.

[0012] The one or more parallel processors 112 may include circuitry optimized for graphics and video processing, including, for example, video output circuitry, and may form a graphics processing unit (GPU). Alternatively or additionally, the one or more parallel processors 112 may include circuitry optimized for general-purpose processing while maintaining the underlying computing architecture described in more detail herein. Components of the computing system 100 may be integrated with one or more other system elements in a single integrated circuit. For example, the one or more parallel processors 112, the memory hub 105, the one or more processors 102, and the I / O hub 107 may be integrated in a system-on-chip (SoC) integrated circuit.Alternatively, the components of computer system 100 may be integrated into a single package to form a system-in-package (SIP) configuration. In one embodiment, at least some of the components of computer system 100 may be integrated into a multi-chip module (MCM), which may be interconnected with other multi-chip modules to form a modular computer system.

[0013] In some configurations, computing system 100 includes, in addition to the one or more processors 102 and the one or more parallel processors 112, one or more accelerator devices 130 coupled to storage hub 105. The one or more accelerator devices 130 are configured to perform domain-specific acceleration of workloads to handle tasks that are computationally intensive or require high throughput. The one or more accelerator devices 130 may reduce the load imposed on the one or more processors 102 and / or the one or more parallel processors 112 of computing system 100.The one or more accelerator devices 130 may include, but are not limited to, intelligent network interface cards, data processing units, cryptographic accelerators, storage accelerators, artificial intelligence (AI) accelerators, neural processing units (NPUs), storage accelerators, and / or video transcoding accelerators. It is understood that the computing system 100 shown herein is illustrative and that variations and modifications are possible. The interconnect topology, including the number and arrangement of bridges, the number of processor(s) 102, and the number of parallel processor(s) 112, may be modified as desired.For example, system memory 104 may be connected directly to the one or more processors 102 rather than via a bridge, while other devices communicate with system memory 104 via memory hub 105 and the one or more processors 102. In other alternative topologies, the one or more parallel processors 112 are connected to I / O hub 107 or directly to one of the one or more processors 102 rather than to memory hub 105. In other embodiments, I / O hub 107 and memory hub 105 may be integrated into a single chip. It is also possible for two or more sets of processors 102 to be mounted via multiple sockets, which may be coupled to two or more instances of the one or more parallel processors 112.

[0014] Some of the specific components shown here are optional and may not be included in all implementations of computer system 100. For example, any number of add-in cards or peripherals may be supported, or some components may be eliminated. Furthermore, some architectures may use different terminology for components similar to those described in Fig. 1. For example, in some architectures, the memory hub 105 may be referred to as a northbridge, while the I / O hub 107 may be referred to as a southbridge.

[0015] Fig.2A illustrates a parallel processor 200. The parallel processor 200 may be a GPU, GPGPU, or the like, as described herein. The various components of the parallel processor 200 may be implemented using one or more integrated circuit devices, such as programmable processors, application-specific integrated circuits (ASICs), or field-programmable gate arrays (FPGAs). The parallel processor 200 may be one or more of the Fig. 1 shown parallel processors 112.

[0016] The parallel processor 200 includes a parallel processing unit 202. The parallel processing unit includes an I / O unit 204 that enables communication with other devices, including other instances of the parallel processing unit 202. The I / O unit 204 may be directly connected to other devices. For example, the I / O unit 204 is connected to other devices using a hub or switch interface, such as the storage hub 105. The connections between the storage hub 105 and the I / O unit 204 form a communication link 113.Within the parallel processing unit 202, the I / O unit 204 is connected to a host interface 206 and a memory crossbar 216, where the host interface 206 receives commands directed to perform processing operations, and the memory crossbar 216 receives commands directed to perform memory operations. In one embodiment, the I / O unit 204 is configured to enable secure I / O operations via a trusted execution environment (TEE) I / O support. The TEE I / O enables trusted I / O virtualization, where a trust relationship can be established directly between a secure virtual environment, such as a trusted virtual machine, and the parallel processor 200 or secure partitions of the parallel processor.

[0017] When host interface 206 receives a command buffer via I / O unit 204, host interface 206 may instruct work operations to execute those commands in a frontend 208. In one embodiment, frontend 208 is coupled to a scheduler 210 configured to dispatch commands or other work items to a processing cluster array 212. Scheduler 210 ensures that processing cluster array 212 is properly configured and in a valid state before dispatching tasks to the processing clusters of processing cluster array 212. Scheduler 210 may be implemented by firmware logic executing in a microcontroller.The scheduler 210, implemented by a microcontroller, is configurable to perform complex scheduling and work distribution operations at both coarse and fine granularity, enabling fast preemption interrupts and context switching of threads executing in the processing cluster array 212. Preferably, the host software can indicate the workloads to be scheduled in the processing cluster array 212 through one of several graphics processing doorbells. In other examples, polling for new workloads or interrupts can be used to identify or indicate the availability of work to be performed. The workloads can then be automatically distributed across the processing cluster array 212 by the scheduler 210 logic within the scheduler's microcontroller.

[0018] Processing cluster array 212 may include up to "N" processing clusters (e.g., cluster 214A, cluster 214B, through cluster 214N). Each cluster 214A through 214N of processing cluster array 212 may execute a large number of concurrent threads. Scheduler 210 may allocate work to clusters 214A through 214N of processing cluster array 212 using various scheduling and / or work distribution algorithms, which may vary depending on the workload incurred for each type of program or computation. Scheduling may be handled dynamically by scheduler 210 or may be partially assisted by compiler logic during compilation of program logic configured for execution by processing cluster array 212.Optionally, different clusters 214A through 214N of the processing cluster array 212 may be assigned to process different types of programs or to perform different types of calculations.

[0019] Processing cluster array 212 may be configured to perform various types of parallel processing operations. For example, processing cluster array 212 is configured to perform general-purpose parallel computing operations. For example, processing cluster array 212 may include logic for performing processing tasks, including filtering video and / or audio data, performing modeling operations that include physics operations, and performing data transformations.

[0020] The processing cluster array 212 is configured to perform parallel graphics processing operations. In embodiments where the parallel processor 200 is configured to perform graphics processing operations, the processing cluster array 212 may include additional logic to support the execution of these graphics processing operations, including, but not limited to, texture sampling logic for performing texture operations, as well as tessellation and other vertex processing logic. The processing cluster array 212 may also be configured to execute shader programs related to graphics processing, such as vertex shaders, tessellation shaders, geometry shaders, and pixel shaders. The parallel processing unit 202 may transfer data from system memory for processing via the I / O unit 204.The transferred data can be stored in an on-chip memory (e.g., the parallel processor memory 222) during processing and then written back to the system memory.

[0021] In some embodiments where parallel processing unit 202 is used to perform graphics processing, scheduler 210 may be configured to divide the processing workload into approximately equal-sized tasks to better facilitate distribution of graphics processing operations across multiple clusters 214A-214N of processing cluster array 212. In some of these embodiments, portions of processing cluster array 212 may be configured to perform other types of processing. For example, a first portion may be configured to perform vertex shading and topology generation, a second portion may be configured to perform tessellation and geometry shading, and a third portion may be configured to perform pixel shading or other screen-space operations to produce a rendered image for a display.Intermediate data generated by one or more of the clusters 214A through 214N may be stored in buffers to allow the intermediate data to be transferred between the clusters 214A through 214N for further processing.

[0022] During operation, the processing cluster array 212 may receive processing tasks to be executed via the scheduler 210, which receives commands defining processing tasks from the front end 208. For graphics processing operations, processing tasks may include indices of data to be processed, e.g., surface data (surface patch data), primitive data, vertex data, and / or pixel data, as well as state parameters and commands that define how the data should be processed (e.g., which program should be executed). The scheduler 210 may be configured to retrieve the indices corresponding to the tasks or may receive the indices from the front end 208. The front end 208 may be configured to ensure that the processing cluster array 212 has been configured to a valid state before initiating the workload, which may be represented by buffers for incoming commands (e.g., stack buffers, push buffers, etc.).) is specified.

[0023] Each of the one or more instances of parallel processing unit 202 may be coupled to parallel processor memory 222. Parallel processor memory 222 may be accessed via memory crossbar 216, which may receive memory requests from processing cluster array 212 as well as I / O unit 204. Memory crossbar 216 may access parallel processor memory 222 via a memory interface 218. Memory interface 218 may include a plurality of partition units (e.g., partition unit 220A, partition unit 220B, through partition unit 220N), each of which may be coupled to a portion (e.g., a memory unit) of parallel processor memory 222. The number of partition units 220A to 220N may be configured to be equal to the number of storage units such that each partition unit 220A0 to 220N has a corresponding storage unit 224A to 224N.In other embodiments, the number of partition units 220A to 220N need not be equal to the number of storage devices.

[0024] Memory units 224A through 224N may include various types of memory devices, including dynamic random access memories (DRAM) or graphics random access memories, such as synchronous graphics random access memories (SGRAM), including double data rate graphics memories (GDDR). Optionally, memory units 224A through 224N may also include 3D stacked memories, including, but not limited to, high bandwidth memories (HBM). Those skilled in the art will appreciate that the specific implementation of memory units 224A through 224N may vary and may be selected from any of several conventional designs.Render targets, such as frame buffers or texture maps, may be stored across memory units 224A through 224N, allowing partition units 220A through 220N to write portions of each render target in parallel to efficiently utilize the available bandwidth of parallel processor memory 222. In some embodiments, a local instance of parallel processor memory 222 may be eliminated in favor of a unified memory design that utilizes system memory in conjunction with a local cache.

[0025] Optionally, any of the clusters 214A through 214N of the processing cluster array 212 has the capability to process data written to any of the memory units 224A through 224N within the parallel processor memory 222. The memory crossbar 216 may be configured to transfer the output of each cluster 214A through 214N to any partition unit 220A through 220N or to another cluster 214A through 214N, which may perform additional processing operations on the output. Each cluster 214A through 214N may communicate with the memory interface 218 via the memory crossbar 216 to read from or write to various external memory devices.In one embodiment with the memory crossbar 216, the memory crossbar 216 includes a connection to the memory interface 218 to communicate with the I / O unit 204, as well as a connection to a local instance of the parallel processor memory 222 to enable the processing units within the various processing clusters 214A-214N to communicate with system memory or other memory not locally contained within the parallel processing unit 202. In general, the memory crossbar 216 may, for example, be capable of using virtual channels to separate traffic flows between the clusters 214A-214N and the partition units 220A-220N.

[0026] Although a single instance of the parallel processing unit 202 is illustrated within the parallel processor 200, any number of instances of the parallel processing unit 202 may be included. For example, multiple instances of the parallel processing unit 202 may be provided on a single add-in card, or multiple add-in cards may be interconnected. For example, the parallel processor 200 may be an add-in device, such as the add-in device 120 of Fig.1, which may be a graphics card, such as a discrete graphics card, including one or more GPUs, one or more memory devices, and device-to-device or network or fabric interfaces. The various instances of the parallel processing unit 202 may be configured to operate with each other, even if the different instances have different numbers of processing cores, different amounts of local parallel processor memory, and / or other configuration differences. Optionally, some instances of the parallel processing unit 202 may include higher-precision floating-point units relative to other instances.Systems including one or more instances of the parallel processing unit 202 or parallel processor 200 may be implemented in a variety of configurations and form factors, including, but not limited to, desktop, laptop, or handheld personal computers, servers, workstations, gaming consoles, and / or embedded systems. An orchestrator may form federated nodes for workload performance using one or more of: disaggregated processor resources, cache resources, memory resources, data storage resources, and network resources.

[0027] In one embodiment, the parallel processing unit 202 may be partitioned into multiple instances. These multiple instances may be configured to execute workloads associated with different clients in an isolated manner, enabling a predetermined quality of service to be provided to each client. For example, each cluster 214A through 214N may be split and isolated from other clusters, allowing the processing cluster array 212 to be divided into multiple compute partitions or instances. With such a configuration, workloads executing in an isolated partition are protected from disruptions or errors associated with another workload executing in a different partition.Partition units 220A-220N may be configured to enable a dedicated and / or isolated path to storage for clusters 214A-214N associated with the respective compute partitions. This data path isolation allows the compute resources within a partition to communicate with one or more assigned storage units 224A-224N without being subject to interference from the activities of other partitions. In one embodiment, data path isolation may be enhanced by encrypting the data in the memory of the various partitions with encryption keys unique to an associated partition, thus securing data on a per-partition basis while at rest and in transition.

[0028] Fig.2B is a block diagram of a partition unit 220. The partition unit 220 may be an instance of one of the partition units 220A to 220N of Fig. 2A. As illustrated, the partition unit 220 includes an L2 cache 221, a frame buffer interface 225, and a raster operation unit (ROP) 226. The L2 cache 221 is a read / write cache configured to perform load and store operations received from the memory crossbar 216 and the ROP 226. Read misses and urgent write-back requests are issued by the L2 cache 221 to the frame buffer interface 225 for processing. Updates may also be sent to the frame buffer via the frame buffer interface 225 for processing. In one embodiment, the frame buffer interface 225 is connected to a memory unit 224 of the memory units 224A-224N within the parallel processor memory 222. Fig.2A. The partition unit 220 may additionally or alternatively be connected to one of the memory units in the parallel processor memory via a memory control unit (not shown).

[0029] In graphics applications, the ROP 226 is a processing unit that performs raster operations such as stenciling, Z-testing, blending, and the like. The ROP 226 then outputs processed graphics data, which is stored in graphics memory. In some embodiments, the ROP 226 includes or is coupled to a CODEC 227, which includes compression logic to compress depth or color data written to the memory or L2 cache 221 and to decompress depth or color data read from the memory or L2 cache 221. The compression logic may be lossless compression logic using one or more of several compression algorithms. The type of compression performed by the CODEC 227 may vary based on the statistical properties of the data to be compressed.For example, in one embodiment, delta color compression is performed on depth and color data on a per-tile basis. In one embodiment, CODEC 227 includes compression and decompression logic that can compress and decompress computational data associated with machine learning operations. For example, CODEC 227 can compress sparse matrix data for sparse machine learning operations. CODEC 227 can also compress sparse matrix data encoded in a sparse matrix format (e.g., coordinate list encoding (COO), compressed sparse row (CSR), compressed sparse column (CSC), etc.) to produce compressed and encoded sparse matrix data.The compressed and encoded sparse matrix data may be decompressed and / or decoded before being processed by processing elements, or the processing elements may be configured to consume compressed, encoded, or compressed and encoded data for processing. In one embodiment, CODEC 227 may be configured as a general-purpose data compression engine for use in GPU database acceleration and high-volume data analytics.

[0030] The ROP 226 may be included in any processing cluster (e.g., cluster 214A through 214N of Fig.2A). In such an embodiment, read and write requests for pixel data are transmitted via the memory crossbar 216 instead of pixel fragment data. The processed graphics data may be displayed on a display device, such as one of the one or more display devices 110A, 110B of Fig. 1, for further processing by the one or more processors 102 or for further processing by one of the processing units within the parallel processor 200 of the Fig. 2A.

[0031] Fig. 2C is a block diagram of a processing cluster 214 within a parallel processing unit. For example, the processing cluster 214 is an instance of one of the processing clusters 214A through 214N of Fig.2A. The processing cluster 214 may be configured to execute many threads in parallel, where the term "thread" refers to an instance of a specific program executing on a specific set of input data. Optionally, single-instruction multiple-data (SIMD) instruction techniques may be used to support parallel execution of a large number of threads without providing multiple independent instruction units. Alternatively, single-instruction multiple-thread (SIMT) techniques are used to support parallel execution of a large number of generally synchronized threads using a common instruction unit configured to issue instructions to a set of processing engines within each of the processing clusters.Unlike a SIMD execution regime, where all processing engines typically execute identical instructions, SIMT execution allows different threads to more easily follow divergent execution paths through a given threaded program. Those skilled in the art will understand that a SIMD processing regime is a functional subset of a SIMT processing regime.

[0032] The operation of the processing cluster 214 can be controlled by a pipeline manager 232, which distributes processing tasks to SIMT parallel processors. The pipeline manager 232 receives instructions from the scheduler 210. Fig.2A and manages the execution of these instructions via a graphics multiprocessor 234 and / or a texture unit 236. The illustrated graphics multiprocessor 234 is an example instance of a SIMT parallel processor. However, various types of SIMT parallel processors with different architectures may be included within the processing cluster 214. One or more instances of the graphics multiprocessor 234 may be included within a processing cluster 214. The illustrated graphics multiprocessor 234 may also be referred to as a streaming multiprocessor (SM) and is capable of executing a large number of execution threads concurrently.

[0033] The graphics multiprocessor 234 may process data, and a data crossbar 240 may be used to distribute the processed data to one of several possible destinations, including instances of graphics multiprocessors 234 within the processing cluster 214. The pipeline manager 232 may facilitate the distribution of processed data by specifying destinations for processed data to be distributed via the data crossbar 240. Each graphics multiprocessor 234 within the processing cluster 214 may include an identical set of functional execution logic (e.g., arithmetic logic units, load-store units, etc.). The functional execution logic may be configured in a staged manner in which new instructions may be issued before previous instructions have completed.The functional execution logic supports a variety of operations, including integer and floating-point arithmetic, comparison operations, Boolean operations, bit shifting, and calculations using various algebraic functions. The same hardware consisting of functional units can be leveraged to perform different operations, and any combination of functional units can be present.

[0034] The instructions transferred to the processing cluster 214 form a thread. A set of threads executing across the set of parallel processing engines is a thread group. A thread group executes the same program on different input data. Each thread within a thread group can be assigned to a different processing engine within a graphics multiprocessor 234. A thread group can include fewer threads than the number of processing engines within the graphics multiprocessor 234. If a thread group has fewer threads than the number of processing engines, one or more of the processing engines may be idle during the cycles in which that thread group is processing. A thread group can also include more threads than the number of processing engines within the graphics multiprocessor 234.If the thread group includes more threads than the number of processing engines within the graphics multiprocessor 234, the processing may be performed over consecutive clock cycles. Optionally, multiple thread groups may execute concurrently in the graphics multiprocessor 234.

[0035] The graphics multiprocessor 234 may include an internal cache for performing load and store operations. Optionally, the graphics multiprocessor 234 may forgo an internal cache and use a cache (e.g., a Level 1 (L1) cache 248) within the processing cluster 214. Each graphics multiprocessor 234 also has access to Level 2 (L2) caches within the partition units (e.g., partition units 220A through 220N of the Fig.2A) that are shared by all processing clusters 214 and that can be used to transfer data between threads. The graphics multiprocessor 234 can also access off-chip global memory, which can include one or more of the local parallel processor memory and / or the system memory. Any memory external to the parallel processing unit 202 can be used as global memory. Embodiments where the processing cluster 214 includes multiple instances of the graphics multiprocessor 234 can share common instructions and data, which can be stored in the L1 cache 248.

[0036] Each processing cluster 214 may include a memory management unit (MMU) 245 configured to map virtual addresses to physical addresses. In other embodiments, one or more instances of the MMU 245 may be located within the memory interface 218 of the Fig. 2A. The MMU 245 includes a set of page table entries (PTEs) used to map a virtual address to a physical address of a tile and, optionally, a cache line index. The MMU 245 may include translation lookaside buffers (TLBs) or caches that may be located within the graphics multiprocessor 234 or the L1 cache 248 of the processing cluster 214. The physical address is processed to distribute access locality of surface data to allow efficient interleaving of requests between partition units. The cache line index may be used to determine whether a request for a cache line is a hit or a miss.

[0037] In graphics and computing applications, a processing cluster 214 may be configured such that each graphics multiprocessor 234 is coupled to a texture unit 236 for performing texture mapping operations, such as determining texture sample positions, reading texture data, and filtering the texture data. The texture data is read from an internal texture L1 cache (not shown) or, in some embodiments, an L1 cache within the graphics multiprocessor 234 and retrieved from an L2 cache, local parallel processor memory, or system memory as needed. Each graphics multiprocessor 234 issues processed tasks to the data crossbar 240 to provide the processed tasks to another processing cluster 214 for further processing or to store the processed task in an L2 cache, local parallel processor memory, or system memory via the memory crossbar 216.A pre-ROP 242 (pre-raster operation unit) is configured to receive data from the graphics multiprocessor 234 and to forward data to ROP units located in the partition units described herein (e.g., partition units 220A through 220N in . Fig. 2A). The preROP unit 242 can perform optimizations for color mixing, organize pixel color data, and perform address translations.

[0038] It should be understood that the core architecture described herein is illustrative and that variations and modifications are possible. Any number of processing units, e.g., the graphics multiprocessor 234, the texture units 236, the preROPs 242, etc., may be included within a processing cluster 214. A parallel processing cluster as described herein may include any number of instances of the processing cluster 214. Optionally, each processing cluster 214 may be configured to operate independently of other instances of the processing cluster 214 using, for example, separate and individual processing units, L1 caches, L2 caches, etc.

[0039] Fig.2D shows an example of graphics multiprocessor 234, where graphics multiprocessor 234 is coupled to pipeline manager 232 of processing cluster 214. Graphics multiprocessor 234 has an execution pipeline including, but not limited to, an instruction cache 252, an instruction unit 254, an address mapping unit 256, a register file 258, one or more general-purpose graphics processing unit (GPGPU) cores 262, and one or more load / store units 266. GPGPU cores 262 and load / store units 266 are coupled to cache memory 272 and shared memory 270 via a memory-and-cache interconnect 268. The graphics multiprocessor 234 may additionally include ray tracing cores 263, which include hardware logic for accelerating ray tracing operations, and tensor cores 264, which include hardware logic for accelerating tensor operations (e.g., matrix operations).Instruction cache 252 may receive a stream of instructions to be executed from pipeline manager 232. The instructions are cached in instruction cache 252 and dispatched for execution by instruction unit 254. Instruction unit 254 may dispatch instructions as thread groups (e.g., chains), with each thread of the thread group assigned to a different execution unit within GPGPU core 262. An instruction may access any local, shared, or global address space by specifying an address within a unified address space. Address mapping unit 256 may be used to translate addresses in the unified address space into a unique memory address accessible by load / store units 266.

[0040] Register file 258 provides a set of registers for the functional units of graphics multiprocessor 234. Register file 258 provides short-term storage for operands associated with the data paths of the functional units (e.g., GPGPU cores 262, load / store units 266) of graphics multiprocessor 234. Register file 258 may be partitioned between each of the functional units, so that each functional unit is assigned a dedicated portion of register file 258. For example, register file 258 may be partitioned between the various concatenations executed by graphics multiprocessor 234.

[0041] The GPGPU cores 262 may each include floating-point units (FPUs) and / or integer arithmetic logic units (ALUs) used to execute instructions of the graphics multiprocessor 234. In some implementations, the GPGPU cores 262 may include hardware logic that may otherwise be located in the tensor cores 264 and / or ray tracing cores 263. The GPGPU cores 262 may have a similar architecture or differ in their architecture. For example, and in one embodiment, a first portion of the GPGPU cores 262 includes a single-precision FPU and an integer ALU, while a second portion of the GPGPU cores includes a double-precision FPU. The FPUs may optionally implement the IEEE Standard 754-2008 for floating-point arithmetic or enable variable-precision floating-point arithmetic.In one embodiment, the single-precision FPUs or a separate set of FPUs are configurable to perform operations on 16-bit floating-point operands, such as operands in a half-precision or bfloat16 format (e.g., Brain floating-point), which is a 16-bit floating-point format with one sign bit, eight exponent bits, and eight significant bits, seven of which are explicitly stored. The FPUs within one or more of the GPGPU cores 262 may also support one or more 8-bit floating-point formats. Supported 8-bit floating-point formats include the E4M3 format, which has a 4-bit exponent and a 3-bit mantissa, and the E5M2 format, which has a 5-bit exponent and a 2-bit mantissa.Graphics multiprocessor 234 may also include one or more fixed-function or special-function units to perform specific functions, such as copy rectangle or pixel blending operations. One or more of the GPGPU cores may also include fixed-function or special-function logic.

[0042] The GPGPU cores 262 may include SIMD logic capable of performing a single instruction on multiple data sets. Optionally, the GPGPU cores 262 may physically execute SIMD4, SIMD8, and SIMD16 instructions and logically execute SIMD1, SIMD2, and SIMD32 instructions. The SIMD instructions for the GPGPU cores may be generated at compile time by a shader compiler or automatically generated when executing programs written and compiled for single-program multiple data (SPMD) or SIMT architectures. Multiple threads of a program configured for the SIMT execution model may execute via a single SIMD instruction. For example, and in one embodiment, eight SIMT threads performing the same or similar operations may execute in parallel as a single SIMD8 instruction.In one embodiment, a chain of 32 SIMT threads can be executed as a single SIMD32 instruction. Chain divergence can be handled by multiple SIMD instructions.

[0043] The memory and cache interconnect 268 is an interconnect network that connects each of the functional units of the graphics multiprocessor 234 to the register file 258 and to the shared memory 270. For example, the memory and cache interconnect 268 is a crossbar interconnect that allows the load / store unit 266 to implement load and store operations between the shared memory 270 and the register file 258. The register file 258 may operate at the same frequency as the GPGPU cores 262, so that data transfer between the GPGPU cores 262 and the register file 258 has very low latency. The shared memory 270 may be used to enable communication between threads executing in the functional units within the graphics multiprocessor 234.Shared memory 270 can also be used as a programmatically managed cache. For example, cache 272 can be used as an automatically managed data cache to cache texture data communicated between the functional units and texture unit 236. Shared memory 270 and cache 272 can be coupled to data crossbar 240 to enable communication with other components of the processing cluster, thereby enabling cooperative execution of cluster workgroups across multiple graphics multiprocessors within a processing cluster. Threads executing in GPGPU cores 262 can programmatically store data in shared memory in addition to the automatically cached data stored in cache 272.In one embodiment, the shared memory 270 and the cache memory 272 may be combined into a single configurable memory unit that may be selectively configured as the cache memory 272 or the shared memory 270.

[0044] Fig. Figure 2E illustrates a graphics multiprocessor 235 having an alternative configuration with respect to the graphics multiprocessor 234 of Fig. 2D. The disclosure of any features in combination with the graphics multiprocessor 235 described herein also discloses a corresponding combination with the graphics multiprocessor 234 of the Fig. 2D, but is not limited to it. The graphics multiprocessor 235 of the Fig. 2E includes several additional instances of execution resource units 286A to 286D with respect to the graphics multiprocessor 234 of the Fig.2D. For example, the graphics multiprocessor 235 may include multiple instruction units 254A through 254D, the register files 258A through 258D, and the texture units 280A through 280D. The graphics multiprocessor 235 also includes multiple sets of graphics or compute execution units (e.g., the GPGPU cores 262A through 262D, the ray tracing cores 263A through 263D, and the tensor cores 264A through 264D) and multiple sets of load / store units 266A through 266D. The execution resources 286A-286D may cooperate with one or more texture units 280A-280D for texture operations while sharing an instruction cache 252 and shared memory 270 and cache memories 272A, 272B. In one embodiment, the execution resources 286A-286D additionally include multi-function units (MUFU) and / or special-purpose units (SFU) (e.g.,the MFU 267A to 267D), which are used to perform specialized mathematical operations, such as transcendental operations including exponential, logarithmic and trigonometric functions.

[0045] The various components may communicate with each other via an interconnect fabric 290. The interconnect fabric 290 may include one or more crossbar switches to enable communication between the various components of the graphics multiprocessor 235. The GPGPU cores 262A to 262D, the ray tracing cores 263A, 263B, and the tensor cores 264A to 264D may each communicate with each other via the shared memory 270 via the interconnect fabric 290. The interconnect fabric 290 may mediate communication within the graphics multiprocessor 235 to ensure fair bandwidth allocation between the components. In one embodiment, the interconnect fabric 290 is a separate high-speed network fabric layer upon which each component of the graphics multiprocessor 235 is stacked.The components of the graphics multiprocessor 235 can also communicate with remote components via the interconnect structure 290.

[0046] In one embodiment, graphics multiprocessor 235 includes a tensor transfer engine 292, which is a copy engine configurable to accelerate the movement of tensor data into and out of graphics multiprocessor 235. Tensor transfer engine 292 can accelerate tensor memory operations by asynchronously performing address generation and data movement operations for N-dimensional blocks of tensor data, offloading operations that would otherwise be performed manually by program code executed by graphics multiprocessor 235. Tensor transfer engine 292 is configurable to copy data, for example, between shared memory 270 and / or cache memory 272A, 272B of graphics multiprocessor 235 and memory external to graphics multiprocessor 235, such as global graphics processor memory (e.g., parallel processor memory 222).In one embodiment, data transfers performed by the tensor transfer engine 292 may be configured to selectively bypass various levels of intermediate data storage between the source and destination memory. For example, a transfer between global memory and shared memory 270 may skip register files 258A through 258D. In one embodiment, threads during asynchronous tensor transfers may be synchronized via a non-blocking barrier synchronization mechanism.

[0047] In various embodiments, the graphics multiprocessor 235 may be customized for specific use cases through the inclusion or exclusion of certain components, allowing for various implementations of the graphics multiprocessor 235 customized for target performance, capability, and range characteristics. For example, compute-oriented variants of the graphics multiprocessor 235 that will not perform graphics operations may exclude the ray tracing cores 263A through 263D. Fully graphics-oriented variants may exclude the tensor transfer engine 292, while graphics-oriented variants additionally configured to accelerate neural network inference may include at least one version of the tensor transfer engine 292.

[0048] For the expert it is self-evident that the Fig. 1 and the Fig.2A to 2E are descriptive, but not limiting, of the scope of the present embodiments. Thus, the techniques described herein may be implemented in any properly configured processing device, including, but not limited to, one or more mobile application processors, one or more desktop or server central processing units (CPUs) including multi-core CPUs, one or more parallel processing units, such as the parallel processing unit 202 of Fig. 2A, as well as one or more graphics processors or dedicated processing units, without departing from the scope of the embodiments described herein.

[0049] The parallel processor or GPGPU, as described herein, may be communicatively coupled to host processor / cores to accelerate graphics operations, machine learning operations, pattern analysis operations, and various general-purpose GPU (GPGPU) functions. The GPU may be communicatively coupled to the host processor / cores via a bus or other interconnect (e.g., a high-speed interconnect such as PCIe or NVLink, or other known, standardized, or proprietary protocols). In other embodiments, the GPU may be integrated in the same package or die as the cores and communicatively coupled to the cores via an internal processor bus / interconnect (i.e., within the package or die).Regardless of how the GPU is connected, the processor cores can assign work to the GPU in the form of sequences of commands / instructions contained in a work descriptor. The GPU then uses dedicated circuitry / logic to efficiently process these commands / instructions.

[0050] Fig. 3C illustrates a graphics processing unit (GPU 380) that includes dedicated sets of graphics processing resources arranged in multi-core groups 365A through 365N. Multi-core groups 365A through 365N correspond to graphics multiprocessor 234 of Fig. 2D or the graphics multiprocessor 235 Fig.2E. Although the details of a single example of multi-core groups 365A through 365N (e.g., multi-core group 365A) are provided, it should be understood that the other multi-core groups 365B through 365N may be equipped with the same or similar sets of graphics processing resources. The details described with respect to multi-core groups 365A through 365N may also be applied to graphics multiprocessor 234 or graphics multiprocessor 235, as described herein.

[0051] As illustrated, a multi-core group 365A may include a set of graphics cores 370, a set of tensor cores 371, and a set of ray tracing cores 372. The graphics cores 370 are analogous to the GPGPU cores 262A through 262D and are configurable to execute instructions to perform graphics and / or general-purpose computing operations. A scheduler / dispatcher 368 schedules and dispatches the graphics threads for execution on the various cores in the multi-core group 365A. Register files 369 are included that store operand values ​​used by the cores when performing graphics or general-purpose computing operations for executing threads.These register files 369 may, for example, contain registers configurable to store integer or floating-point values, and they may contain vector registers for storing packed integer and / or floating-point data elements and tile registers for storing tensor / matrix values. The tile registers may be implemented as multidimensional registers containing combined sets of vector registers.

[0052] One or more combined Level 1 (L1) caches and shared memory units 373 store graphics data such as texture data, vertex data, pixel data, ray data, hull data, etc. locally within each multi-core group 365A. One or more texture units 374 may also be used to perform texturing operations such as texture mapping and sampling. A Level 2 (L2) cache 375, shared by all or a subset of the multi-core groups 365A through 365N, stores graphics data and / or instructions for multiple concurrent graphics threads. As illustrated, the L2 cache 375 may be shared across a plurality of multi-core groups 365A through 365N. One or more memory controllers 367 couple the GPU 380 to a memory 366, which may be system memory (e.g., DRAM) and / or dedicated graphics memory (e.g., GDDR6 memory).

[0053] Input / output (I / O) circuitry 363 couples the GPU 380 to one or more I / O devices 362, such as digital signal processors (DSPs), network controllers, or user input devices. An on-chip interconnect may be used to couple the I / O devices 362 to the GPU 380 and the memory 366. One or more I / O memory management units (IOMMUs 364) of the I / O circuitry 363 couple the I / O devices 362 directly to the memory 366. The IOMMU 364 optionally manages multiple sets of page tables to map virtual addresses to physical addresses in the system memory 366. The I / O devices 362, the one or more CPUs 361, and the GPU 380 may then share the same virtual address space.

[0054] In one implementation of the IOMMU 364, the IOMMU 364 supports virtualization. In this case, it can manage a first set of page tables to map virtual guest / graphics addresses to physical guest / graphics addresses and a second set of page tables to map the physical guest / graphics addresses to physical system / host addresses (e.g., within memory 366). The base addresses of both the first and second sets of page tables can be stored in control registers and swapped out upon a context switch (e.g., so that the new context is provided with access to the relevant set of page tables). Although this is Fig.3, each of the cores within multi-core groups 365A through 365N may include address translation buffers (TLBs) to cache guest virtual to physical address translations, guest physical to physical address translations, and guest virtual to physical address translations.

[0055] The CPU(s) 361, the GPU 380, and the I / O devices 362 may be integrated into a single semiconductor chip and / or a chip package. The memory 366 may be integrated into the same chip or coupled to the one or more memory controllers 367 via an off-chip interface. In one implementation, the memory 366 comprises GDDR6 memory that shares the same virtual address space with other physical system-level memories, although the underlying principles described herein are not limited to this particular implementation.

[0056] The tensor cores 371 may include a plurality of execution units specifically configured to perform matrix operations, which are the fundamental computational operations used to perform deep learning operations. For example, concurrent matrix multiplication operations may be used to train and infer neural networks. The tensor cores 371 may perform matrix processing using a variety of operand precisions, including single-precision floating-point (e.g., 32 bits), half-precision floating-point (e.g., 16 bits), integer words (16 bits), bytes (8 bits), and nibbles (4 bits). For example, a neural network implementation extracts features from each rendered scene, potentially combining details from multiple frames to construct a high-quality final image.

[0057] In deep learning implementations, parallel matrix multiplication functions can be scheduled for execution in the tensor cores 371. Training neural networks, in particular, requires a significant number of matrix dot product operations. To process an inner product formulation of multiplying an N x N x N matrix, the tensor cores 371 can include at least N dot product processing elements. Before matrix multiplication begins, an entire matrix is ​​loaded into the tile registers, and at least one column of a second matrix is ​​loaded every cycle for N cycles. In each cycle, N dot products are processed.

[0058] The matrix elements can be stored with different precisions depending on the specific implementation, including 16-bit words, 8-bit bytes (e.g., INT8), and 4-bit nibbles (e.g., INT4). Different precision modes can be specified for the tensor cores 371 to ensure that the most efficient precision is used for different workloads (such as inferencing workloads, which can tolerate quantization into bytes and nibbles). Supported formats additionally include 64-bit floating-point (FP64) and non-IEEE floating-point formats, such as the bfloat16 format. One embodiment includes support for a reduced-precision tensor floating-point mode (TF32), which performs calculations using the range of FP32 (8 bits) and the precision of FP16 (10 bits).Reduced-precision TF32 operations can be performed on FP32 inputs and produce FP32 outputs with higher performance relative to FP32 and greater precision relative to FP16. In one embodiment, one or more of 8-bit floating-point (FP8), 6-bit floating-point (FP6), and 4-bit floating-point (FP4) formats are supported, including floating-point formats represented as microscale (MX) formats.

[0059] In one embodiment, tensor cores 371 support a mode of operation for sparse matrices in which the vast majority of values ​​are zero. Tensor cores 371 include support for sparse input matrices encoded in a sparse matrix representation (e.g., coordinate list encoding (COO), compressed sparse row (CSR), compressed sparse column (CSC), etc.). Tensor cores 371 also include support for compressed sparse matrix representations in case the sparse matrix representation needs to be further compressed. Compressed, encoded, and / or compressed and encoded matrix data, along with associated compression and / or encoding metadata, may be read by tensor cores 371, and the non-zero values ​​may be extracted.For example, for a given input matrix A, a non-zero value may be loaded from the compressed and / or encoded representation of at least a portion of matrix A. Based on the position in matrix A for the non-zero value, which may be determined from index or coordinate metadata associated with the non-zero value, a corresponding value may be loaded into input matrix B. Depending on the operation to be performed (e.g., multiply), loading the value from input matrix B may be skipped if the corresponding value is a zero value. In one embodiment, the pairings of values ​​for certain operations, such as multiply operations, may be pre-scanned by the scheduling logic, and only operations between non-zero inputs are scheduled.Depending on the dimensions of matrix A and matrix B and the operation to be performed, the output matrix C may be dense or sparse. If the output matrix C is sparse, and depending on the configuration of the tensor kernels 371, the output matrix C may be output in a compressed format, a sparse encoding, or a compressed sparse encoding.

[0060] Ray tracing kernels 372 may accelerate ray tracing operations for both real-time and non-real-time ray tracing implementations. In particular, ray tracing kernels 372 may include ray traversal or ray intersection circuitry for performing ray traversal using bounding body hierarchies (BVHs) and for identifying intersections between rays and primitives enclosed within the BVH volumes. Ray tracing kernels 372 may also include circuitry for performing depth checks and depth suppression (e.g., using a Z-buffer or similar arrangement).In one implementation, ray tracing cores 372 perform traversal and intersection operations consistent with the image denoising techniques described herein, at least a portion of which may be performed in tensor cores 371. For example, tensor cores 371 may implement a deep learning neural network to denoise frames generated by ray tracing cores 372. However, CPU(s) 361, graphics cores 370, and / or ray tracing cores 372 may also implement all or part of the denoising and / or deep learning algorithms.

[0061] As described above, a distributed denoising approach may also be employed, where the GPU 380 is located in a computing device coupled to other computing devices via a network or high-speed interconnect. In this distributed approach, the interconnected computing devices can share neural network learning / training data to improve the speed at which the overall system learns to perform denoising for different frame types and / or different graphics applications.

[0062] Ray tracing cores 372 can process all intersections of a BVH traversal and ray primitives, protecting graphics cores 370 from being overloaded with thousands of instructions per ray. For example, each ray tracing core 372 includes a first set of specialized circuitry for performing bounding box checks (e.g., for traversal operations) and / or a second set of specialized circuitry for performing ray triangle intersection checks (e.g., intersecting rays that have been traversed). Thus, for example, multi-core group 365A can simply initiate a ray probe, and ray tracing cores 372 can independently perform ray traversal and intersection determination and return hit data (e.g., a hit, no hit, multiple hits, etc.) to the thread context.The graphics cores 370 and the tensor cores 371 are then free for other graphics or computational tasks, while the ray tracing cores 372 perform the traversal and intersection operations. Optionally, each ray tracing core 372 may include a traversal unit for performing BVH checking operations and / or an intersection unit for performing intersection tests of the ray primitives. The intersection unit generates a "hit," "no hit," or "multiple hits" response, which it provides to the associated thread. During the traversal and intersection operations, the execution resources of the other cores (e.g., the graphics cores 370 and the tensor cores 371) are free to perform other forms of graphics work.In an optional embodiment described below, a hybrid rasterization / ray tracing approach is used in which the rendering operations are divided between the graphics cores 370 and the ray tracing cores 372.

[0063] Ray tracing cores 372 may include hardware support for a ray tracing instruction set, such as Microsoft's DirectX Ray Tracing (DXR), which includes a DispatchRays instruction, as well as ray generation, a nearest hit, any hit, and a miss shader, which enable the assignment of unique sets of shaders and textures for each object. Another ray tracing platform that may be supported by ray tracing cores 372, graphics cores 370, and tensor cores 371 is the Vulkan API (e.g., Vulkan version 1.1.85 and later). However, it should be noted that the underlying principles described here are not limited to any specific ray tracing ISA.In general, ray tracing cores 372, tensor cores 371, and graphics cores 370 may support a ray tracing instruction set that includes instructions / functions for one or more of ray generation, nearest hit, any hit, ray primitive intersection, primitive-wise and hierarchical bounding box construction, no hit, visit, and exceptions. In particular, a preferred embodiment includes ray tracing instructions for performing one or more of the following functions: Ray Generation - Ray generation instructions can be executed on any pixel, sample, or other user-defined work assignment. Nearest Match - A nearest match instruction can be executed to find the nearest intersection point of a ray with primitives within a scene. Any Hits - An any hit instruction identifies multiple intersection points between a ray and primitives within a scene to potentially identify a new nearest intersection point. Intersection - An intersection statement performs a ray primitive intersection check and returns a result. Primitive-wise bounding box construction - This instruction creates a bounding box around a specific primitive or group of primitives (e.g., when constructing a new BVH or other acceleration data structure). Miss - Indicates that a ray misses all geometry within a scene or a specific region of a scene. Visit - Shows the subvolumes that a ray will traverse. Exceptions - Contains various types of exception handlers (e.g., those called for numerous error conditions).

[0064] In one embodiment, ray tracing kernels 372 may be adapted to accelerate general-purpose computational operations using computational techniques analogous to ray intersection checks. A computational framework may be provided that enables shader programs to be compiled into low-level instructions and / or primitives that perform general-purpose computational operations by the ray tracing kernels. Example computational problems that may benefit from the computational operations performed in ray tracing kernels 372 include computations involving bundle, wave, ray, or particle propagation within a coordinate space. Interactions associated with this propagation may be computed with respect to a geometry or network within the coordinate space.For example, computations associated with the propagation of electromagnetic signals through an environment can be accelerated by using instructions or primitives executed by ray tracing kernels. Diffraction and reflection of signals from objects in the environment can be computed as direct ray tracing analogies.

[0065] Ray tracing kernels 372 can also be used to perform computations that are not directly analogous to ray tracing. For example, mesh projection, mesh refinement, and volume sampling computations can be accelerated using ray tracing kernels 372. Generic coordinate space computations, such as nearest neighbor computations, can also be performed. For example, the set of points near a particular point can be determined by defining a bounding box in the coordinate space around the point. BVH and ray sampling logic within ray tracing kernels 372 can then be used to determine the set of point intersection points within the bounding box. The intersection points form the origin and the nearest neighbors of that origin.Computations performed using the ray tracing cores 372 may be performed in parallel with computations performed in the graphics cores 370 and the tensor cores 371. A shader compiler may be configured to compile a computational shader or other general-purpose graphics processing program into low-level primitives that can be parallelized across the graphics cores 370, the tensor cores 371, and the ray tracing cores 372. Techniques for connecting a GPU to a host processor

[0066] Fig. 4A illustrates an example architecture in which a plurality of GPUs 410 to 413, such as those shown in Fig.2A, are communicatively coupled to a plurality of multi-core processors 405, 406 via high-speed interconnects 440A through 440D (e.g., buses, point-to-point interconnects, etc.). The high-speed interconnects 440A through 440D may support communication throughput of 4 GB / s, 30 GB / s, 80 GB / s, or higher, depending on the implementation. Various interconnect protocols may be used, including, but not limited to, PCIe 4.0, PCIe 5.0, PCIe 6.0, and various NVLink and NVlink C2C (chip-to-chip) interconnect protocols (e.g., NVLink v5). However, the underlying principles described herein are not limited to any particular communication protocol or data throughput.

[0067] Two or more of the GPUs 410 to 413 may be interconnected via high-speed interconnects 442A, 442B, which may be implemented using the same or different protocols / connections than those used for the high-speed interconnects 440A to 440D. Similarly, two or more of the multi-core processors 405, 406 may be interconnected via high-speed interconnect 443, which may be symmetric multiprocessor (SMP) buses operating at 20 GB / s, 30 GB / s, 120 GB / s, or lower or higher speeds. Alternatively, all communication between the various Fig.4A using the same protocols / connections (e.g., via a common interconnect fabric). However, as mentioned, the underlying principles described here are not limited to any specific type of interconnect technology.

[0068] Each of the multi-core processor 405 and the multi-core processor 406 may be communicatively coupled to a processor memory 401, 402 via memory interconnects 430A, 430B, respectively, and each GPU 410 to 413 is communicatively coupled to a GPU memory 420 to 423 via GPU memory interconnects 450A to 450D, respectively. The memory interconnects 430A, 430B, and 450A to 450D may utilize the same or different memory access technologies. By way of example and not limitation, the processor memories 401, 402 and the GPU memories 420 to 423 may be volatile memories, such as dynamic random access memories (DRAMs) (including stacked DRAMs), graphics DDR SDRAM (GDDR) (e.g., GDDR5, GDDR6), or high bandwidth memory (HBM), and / or they may be non-volatile memories, such as 3D XPoint / Optane or Nano-RAM.For example, some of the memory may be volatile memory and some may be non-volatile memory (e.g., using a two-level memory hierarchy (2LM hierarchy)). A memory subsystem as described here may be compatible with a variety of memory technologies, such as double-data-rate versions published by JEDEC (Joint Electronic Device Engineering Council).

[0069] As described below, although the various processors 405, 406 and GPUs 410-413 may each be physically coupled to a specific memory 401, 402, 420-423, a unified memory architecture may be implemented in which the same virtual system address space (also referred to as the "effective address space") is distributed across all the various physical memories. For example, the processor memories 401, 402 may each comprise 64 GB of the system memory address space, and the GPU memories 420-423 may each comprise 32 GB of the system memory address space (resulting in a total of 256 GB of addressable memory in this example).

[0070] Fig.4B illustrates additional optional details for an interconnect between a processor 407 and a graphics accelerator 446. The graphics accelerator 446 may include one or more GPU chips integrated on a line card coupled to the processor 407 via the high-speed interconnect 440. Alternatively, the graphics processor 446 may be integrated in the same package or chip as the processor 407. The processor 407 includes a plurality of cores 460A-460D, each having a translation buffer (TLB) 461A-461D and one or more caches 462A-462D. The cores may include numerous other components for executing instructions and processing data, which are not illustrated to avoid obscuring the underlying principles of the components described herein (e.g.,Instruction fetch units, branch prediction units, decoders, execution units, reorder buffers, etc.) may be obfuscated. Caches 462A through 462D may include Level 1 (L1) caches and Level 2 (L2) caches. Additionally, one or more shared caches 456 may be included in the caching hierarchy and shared by sets of cores 460A through 460D. For example, one embodiment of processor 407 includes 24 cores, each with its own L1 cache, twelve shared L2 caches, and twelve shared L3 caches. In this embodiment, one of the L2 and L3 caches is shared by any two adjacent cores. Processor 407 is coupled to system memory 441, which may include processor memories 401, 402.

[0071] Coherency is maintained for data and instructions stored in the various caches 462A through 462D, 456, and system memory 441 via inter-core communication over a coherency bus 464. For example, each cache may have dedicated cache coherency logic / circuitry associated with it to communicate with it over the coherency bus 464 in response to detected reads or writes to particular cache lines. In one implementation, a cache sniffing protocol is implemented over the coherency bus 464 to sniff cache accesses. Cache sniffing / coherency techniques are well known to those skilled in the art and are not described in detail here to avoid obscuring the underlying principles described herein.A proxy circuit 425 may be provided that communicatively couples the graphics accelerator 446 to the coherence bus 464, allowing the graphics accelerator 446 to participate in the cache coherence protocol as a peer of the cores. Specifically, an interface 435 provides connectivity to the proxy circuit 425 via a high-speed interconnect 440 (e.g., a PCIe bus, NVLink, etc.), and an interface 437 connects the graphics accelerator 446 to the high-speed interconnect 440.

[0072] In one implementation, the interface 436 is coupled to an accelerator integration circuit 436, which provides cache management, memory access, context management, and interrupt management services on behalf of multiple graphics processing engines 431, 432, ..., N of the graphics accelerator 446. The graphics processing engines 431, 432, ..., N may each comprise a separate graphics processing unit (GPU). Alternatively, the graphics processing engines 431, 432, ..., N may comprise different types of graphics processing engines within a GPU, such as graphics execution units, media processing engines (e.g., video encoders / decoders), sampling units, and block image transfer (BLIT) engines. In other words, the graphics accelerator may include graphics processing engines 431, 432, ..., N of a single GPU, or the graphics processing engines 431, 432, ..., N may be associated with multiple GPUs integrated in a common package, on a common line card, or on a common chip. The graphics processing engines 431-432, N may be configured with any graphics processor or compute accelerator architecture described herein. The functions to be performed by the graphics processing engines 431, 432 may be specified via work descriptors, which provide an indication of the functions to be performed by the graphics accelerator 446.

[0073] The accelerator integration circuit 436 may include a memory management unit (MMU 439) for performing various memory management functions, such as virtual-to-physical memory translations (also referred to as effective-to-real memory translations) and memory access protocols for accessing the system memory 441. The MMU 439 may also include an address translation buffer (TLB) (not shown) for caching the virtual / effective to physical / real address translations. In one implementation, a cache 438 stores instructions and data for efficient access by the graphics processing engines 431, 432, ..., N. The data stored in the cache 438 and the graphics memories 433, 434, ..., M may be kept coherent with the core caches 462A through 462D, 456, and the system memory 441.As mentioned, this may be achieved via a proxy circuit 425 that participates in the cache coherence mechanism on behalf of the cache 438 and the graphics memories 433, 434, ..., M (e.g., by sending updates to the cache 438 related to modifications / accesses of cache lines in the processor caches 462A to 462D and the shared caches 456, and receiving updates from the cache 438).

[0074] Registers 445 store context data for threads executed by graphics processing engines 431, 432, ..., N, and a context management circuit 448 manages the thread contexts. For example, context management circuit 448 may perform save and restore operations to save and restore contexts of the various threads during context switches (e.g., when a first thread is saved and a second thread is restored so that the second thread can be executed by a graphics processing engine). For example, during a context switch, context management circuit 448 may store current register values ​​in a designated area in memory (identified, for example, by a context pointer). It may then restore the register values ​​when returning to the context.For example, an interrupt management circuit 447 may receive and process interrupts received from system devices.

[0075] In one implementation, virtual / effective addresses from a graphics processing engine are translated by the MMU 439 into real / physical addresses in system memory 441. Optionally, the accelerator integration circuit 436 supports multiple (e.g., 4, 8, 16) address translation accelerators 446 and / or other accelerator devices. The graphics accelerator module 446 may be dedicated to a single application executing in the processor 407, or it may be shared among multiple applications. Optionally, a virtualized graphics execution environment is provided in which the resources of the graphics processing engines 431, 432, ..., N are shared among multiple applications, virtual machines (VMs), or containers.The resources can be divided into "partitions," which are allocated to different VMs and / or applications based on processing requirements and priorities associated with the VMs and / or applications, or based on a predetermined partitioning profile for a graphics accelerator 446. VMs and containers can be used interchangeably here.

[0076] A virtual machine (VM) can be a piece of software that runs an operating system and one or more applications. A VM can be defined by a specification, configuration files, a virtual disk file, a non-volatile random access memory (NVRAM) settings file, and the log file, and is secured by the physical resources of a host computing platform. A VM can contain an operating system (OS) or an application environment installed as software that mimics dedicated hardware. The end user has the same experience on a virtual machine as they would with dedicated hardware. Specialized software called a hypervisor fully emulates the CPU, memory, disk, network, and other hardware resources of the PC client or server, allowing virtual machines to share resources.The hypervisor can emulate multiple virtual hardware platforms that are isolated from each other, allowing virtual machines to run Linux®, Windows® Server, VMware ESXi, and other operating systems on the same underlying physical host.

[0077] A container can be a software package of applications, configurations, and dependencies, allowing the applications to run reliably in one computing environment as well as another. Containers can share an operating system installed on the server platform and run as isolated processes. A container can be a software package that contains everything the software needs to run, such as system tools, libraries, and settings. Containers are not installed like traditional software programs, allowing them to be isolated from other software and the actual operating system. The isolated nature of containers provides several advantages. First, the software in a container will run the container in different environments.For example, a container containing PHP and MySQL can run identically on both a Linux® computer and a Windows® machine. Second, containers provide additional security because the software will not interfere with the host operating system. While an installed application can change system settings and modify resources, such as the Windows Registry, a container can only modify settings within the container.

[0078] Thus, accelerator integration circuitry 436 functions as a bridge to the system for graphics accelerator 446, providing services for address translation and system memory cache. In one embodiment, to facilitate bridging functionality, accelerator integration circuitry 436 may also include shared I / O 497 (e.g., PCIe, USB, or other) and hardware to enable system control of voltage, clock, performance, thermal, and security. Shared I / O 497 may use separate physical connections or may traverse high-speed interconnect 440. Accelerator integration circuitry 436 may also provide virtualization facilities for the host processor to manage virtualization of the graphics processing engines, interrupts, and memory management.

[0079] Because hardware resources of graphics processing engines 431, 432, ..., N are explicitly mapped to the real address space seen by processor 407, each host processor can address these resources directly using an effective address value. An optional feature of accelerator integration circuitry 436 is the physical separation of graphics processing engines 431, 432, ..., N so that they appear to the system as independent entities. In one embodiment, accelerator integration circuitry 436 includes security circuitry 444 that enables configurable cryptographic isolation of data associated with each partition of resources. Different partitions may be associated with different security domains, so that data associated with the different security domains is encrypted using different cryptographic keys.In one embodiment, the security domains of graphics accelerator 446 may be integrated into trusted execution environments supported by processor 407. In one embodiment, secure I / O capabilities may be enabled that allow each security domain to be presented as a separate trusted I / O device, enabling support for trusted DMA and MMIO operations.

[0080] One or more graphics memories 433, 434, ..., M may be coupled to each of the graphics processing engines 431, 432, ..., N, respectively. The graphics memories 433, 434, ..., M store instructions and data processed by each of the graphics processing engines 431, 432, ..., N. The graphics memories 433, 434, ..., M may be volatile memories, such as DRAMs (including stacked DRAMs), GDDR memory (e.g., GDDR5, GDDR6), or HBM, and / or may be non-volatile memories, such as 3D XPoint / Optane, Samsung Z-NAND, or Nano-RAM.

[0081] To reduce data traffic over the high-speed link 440, biasing techniques can be used to ensure that the data stored in the graphics memories 433, 434, ..., M is the data most frequently used by the graphics processing engines 431, 432, ..., N, and preferably not used by the cores 460A to 460D (at least not frequently). Similarly, the biasing mechanism attempts to keep data needed by the cores (and preferably not by the graphics processing engines 431, 432, ..., N) in the caches 462A to 462D, 456 of the cores and system memory 441.

[0082] In an alternative variant, the accelerator integration circuit 436 is integrated within the processor 407, and the graphics processing engines 431, 432, ..., N communicate with the accelerator integration circuit 436 via the high-speed interconnect 440 through the interface 437 and the interface 435 (which may again utilize any form of bus, fabric, or interface protocol). In this variant, the accelerator integration circuit 436 performs the same operations as those described above.

[0083] The described embodiments may support various programming models, including a dedicated process programming model (without graphics accelerator virtualization) and shared programming models (with virtualization). The latter may include programming models controlled by the accelerator integration circuit 436 and programming models controlled by the graphics accelerator 446. In the dedicated process model embodiments, the graphics processing engines 431, 432, ... N may be dedicated to a single application or process under a single operating system. The single application may route other application requests to the graphics engines 431, 432, ... N and provide virtualization within a VM / partition. In the dedicated process programming models, the graphics processing engines 431, 432, ..., N are shared by multiple VM / application partitions. The shared models require a system hypervisor to virtualize the graphics processing engines 431, 432, ..., N to allow access by any operating system. For single-partition systems without a hypervisor, the graphics processing engines 431, 432, ..., N are owned by the operating system. In both cases, the operating system can virtualize the graphics processing engines 431, 432, ..., N to provide access to any process or application. For the shared programming model, the graphics accelerator 446 or a single graphics processing engine 431, 432, ..., N selects a process element using a process routine. The process elements may be stored in system memory 441 and may be addressable using the effective address to real address translation techniques described herein.The process handle may be an implementation-specific value provided to the host process when its context is registered in the graphics processing engines 431, 432, ..., N, which is performed by the host process by invoking system software to add the process element to the list of linked process elements. The lower 16 bits of the process routine may be the offset of the process element within the list of linked process elements.

[0084] Fig.4C illustrates an accelerator integration partition 490. As used herein, "partition" comprises a specific portion of the processing resources of the accelerator integration circuit 436. Process elements 483 are stored in the application address space 482 within the system memory 441. The process elements 483 may be stored in response to GPU calls 481 from an application 480 executing in the processor 407. A process element 483 contains the process state for the application 480. A work descriptor (WD) 484 contained in the process element 483 may be a single work job requested by an application or it may contain a pointer to a queue of work jobs. In the latter case, the WD 484 is a pointer to the work job request queue in the application address space 482.

[0085] The graphics accelerator 446 and / or the individual graphics processing engines 431, 432, ..., N may be shared by all or a subset of the processes in the system. For example, the technologies described herein may include an infrastructure for establishing process state and sending a WD 484 to a graphics accelerator 446 to start a job in a virtualized environment.

[0086] In one implementation, the dedicated process programming model is implementation-specific. In this model, the graphics accelerator 446 or an individual graphics processing engine 431 is owned by a single process. Because a single process owns the graphics accelerator 446, the hypervisor initializes the accelerator integration circuit 436 for the owning partition, and the operating system initializes the accelerator integration circuit 436 for the owning process at the time the graphics accelerator 446 is allocated.

[0087] In operation, the fetch unit 491 in the accelerator integration partition 490 fetches a WD 484 to be processed. The WD 484 includes the indication of functions to be performed by one or more graphics processing engines of the graphics accelerator 446. Data from the WD 484 may be stored in registers 445 and used by the MMU 439, the interrupt management circuitry 447, and / or the context management circuitry 448 as illustrated. For example, the MMU 439 may include segment / page browsing circuitry for accessing segment / page tables 486 within the OS's virtual address space 485. The interrupt management circuitry 447 may process interrupt events 492 received from the graphics accelerator 446. When performing graphics operations, an effective address 493 generated by a graphics processing engine 431-432, ..., N is translated into a real address by the MMU 439.

[0088] Registers 445 may be duplicated for each graphics processing engine 431, 432, ..., N and / or the graphics accelerator 446 and may be initialized by the hypervisor or the operating system. Each of these duplicated registers may be included in an accelerator integration partition 490. In one embodiment, each graphics processing engine 431, 432, ..., N may be presented to the hypervisor 496 as a separate graphics processor device. Quality of service (QoS) settings may be configured for clients of a specific graphics processing engine 431, 432, ..., N. Cryptographic and physical data isolation between the clients of each engine may be enabled via isolated memory access paths and automatic data encryption for each client. Table 1 shows example registers that may be initialized by the hypervisor. Table 1 - Registers initialized by the hypervisor 1 Partition control register 2 Area pointers from scheduled processes to real addresses (RA) 3 Overwrite register for authorization masks 4 Offset of table entries of interrupt vectors 5 Limit for table entries of interrupt vectors 6 Condition register 7 Logical partition ID 8 Record pointer of hypervisor accelerator usages to real addresses (RA) 9 Memory descriptor register

[0089] Example registers that can be initialized by the operating system are shown in Table 2. Table 2 - Registers initialized by an operating system 1 Process and thread identification 2 Context save / restore pointer to effective addresses (EA) 3 Record pointer of accelerator usages to virtual addresses (VA) 4 Pointers to memory segment tables of virtual addresses (VA) 5 Authorization mask 6 Work descriptor

[0090] Each WD 484 may be specific to a particular graphics accelerator 446 and / or a graphics processing engine 431, 432, ..., N. It contains all the information a graphics processing engine 431, 432, ...N needs to do its work, or it may be a pointer to a memory location where the application has set up a command queue with work to be performed.

[0091] Fig.4D illustrates additional optional details of a shared model. This includes a real hypervisor address space 498 in which a process element list 499 is stored. The real hypervisor address space 498 can be accessed via a hypervisor 496 that virtualizes the graphics accelerator engines for the operating system 495.

[0092] The shared programming models allow all or a subset of processes from all or a subset of partitions in the system to use a graphics accelerator 446. There are two programming models in which the graphics accelerator 446 is shared among multiple processes and partitions: time-split sharing and graphics-focused sharing.

[0093] In this model, the hypervisor 496 owns the graphics accelerator 446 and makes its functionality available to all operating systems 495. For a graphics accelerator 446 to support virtualization through the hypervisor 496, the graphics accelerator 446 can meet the following requirements: 1) An application's job request must be autonomous (i.e., state must not be maintained between jobs), or the graphics accelerator 446 must provide a mechanism for context storage and recovery. 2) An application's job request is guaranteed by the graphics accelerator 446 to complete in a specified amount of time, including any compilation errors, or the graphics accelerator 446 provides the ability to interrupt task processing.3) The graphics accelerator 446 must be guaranteed fairness between processes when operating in the directed shared programming model.

[0094] For the directed shared model, application 480 may be required to execute an operating system 495 system call with a graphics accelerator type 446, a work descriptor (WD), an authority mask register (AMR) value, and a context save / restore area pointer (CSRP). The graphics accelerator type 446 describes the targeted acceleration function for the system call. The graphics accelerator type 446 may be a system-specific value.The WD is formatted specifically for the graphics accelerator 446 and may be in the form of a graphics accelerator 446 instruction, an effective address pointer in a user-defined structure, an effective address pointer in a queue of instructions, or any other data structure for describing the functions to be performed by the graphics accelerator 446. In one embodiment, the AMR value is the AMR state to be used for the current process. The value passed to the operating system is similar to an application setting the AMR. If the implementations of the accelerator integration circuit 436 and the graphics accelerator 446 do not support a User Authority Mask Override Register (UAMOR), the operating system may apply the current UAMOR value to the AMR value before the AMR is passed in the hypervisor call.The hypervisor 496 can optionally apply the current value in the Authority Mask Override Register (AMOR) before placing the AMR into the process element 483. The CSRP can be one of the registers 445 containing the effective address of an area in the application's address space 482 for the graphics accelerator 446 to save and restore context state. This pointer is optional if no state needs to be saved between jobs or if a job terminates prematurely. The context save / restore area can be a designated system memory.

[0095] After receiving the system call, the operating system 495 can verify that the application 480 has registered and that it has received the calculation to use the graphics accelerator 446. The operating system 495 then calls the hypervisor 496 with the information shown in Table 3. Table 3 - Call parameters from the OS to the hypervisor 1 A work descriptor (WD) 2 An authorization mask register (AMR) value (potentially masked). 3 A context save / restore (CSRP) range pointer to effective addresses (EA) 4 A process ID (PID) and optional thread ID (TID) 5 A record pointer from accelerator usages (AURP) to virtual addresses (VA) 6 The virtual address of the memory segment table pointer (SSTP) 7 A logical interrupt service number (LISN)

[0096] After receiving the hypervisor call, the hypervisor 496 verifies that the operating system 495 has registered and has been granted permission to use the graphics accelerator 446. The hypervisor 496 then inserts the process element 483 into the list of associated process elements for the corresponding type of graphics accelerator 446. The process element may include the information shown in Table 4. Table 4 - Process element information 1 A work descriptor (WD) 2 An authorization mask register (AMR) value (potentially masked). 3 A context save / restore (CSRP) range pointer to effective addresses (EA) 4 A process ID (PID) and optional thread ID (TID) 5 A record pointer from accelerator usages (AURP) to virtual addresses (VA) 6 The virtual address of the memory segment table pointer (SSTP) 7 A logical interrupt service number (LISN) 8 Interrupt vector table derived from the hypervisor call parameters. 9 A status register value (SR value) 10 Eine logische Partitions-ID (LPID) 11 Ein Datensatzzeiger für Beschleunigernutzungen auf reale Adressen (RA) 12 Das Speicherdeskriptorregister (SDR)

[0097] The hypervisor can initialize multiple registers 445 of the accelerator integration partition 490.

[0098] As in Fig. 4E, an optional implementation employs a unified memory addressable via a common virtual memory address space used to access the physical processor memories 401, 402 and GPU memories 420-423. In this implementation, operations performed in the GPUs 410-413 use the same virtual / effective memory address space to access the processor memories 401, 402 and vice versa, thereby simplifying programmability. A first portion of the virtual / effective memory address space may be assigned to the processor memory 401, a second portion may be assigned to the second processor memory 402, a third portion may be assigned to the GPU memory 420, etc.The total virtual / effective memory space (sometimes referred to as effective address space) can thereby be divided between each of the processor memories 401, 402 and the GPU memories 420 to 423, allowing any processor or GPU to access any physical memory with a virtual address assigned to that memory.

[0099] Bias / coherence management circuitry 494A to 494E may be provided within one or more of the MMUs 439A to 439E, thereby ensuring cache coherence between the caches of the host processors (e.g., the multi-core processor 405) and the GPUs 410 to 413 and implementing biasing techniques that indicate the physical memories in which certain types of data should be stored. Although multiple instances of bias / coherence management circuitry 494A to 494E may be provided in Fig. 4E, the bias / coherence circuitry may be implemented within the MMU of one or more host processors 405 and / or within the accelerator integration circuit 436.

[0100] The GPU-attached memory 420 to 423 can be mapped as part of the system memory and accessed using shared virtual memory (SVM) technology, but without incurring the typical performance penalties associated with full system cache coherence. The ability to access the GPU-attached memory 420 to 423 as system memory without burdensome cache coherence overhead provides a favorable operating environment for GPU swapping. This arrangement allows the host processor 405 software to set up operands and access computation results without the overhead of conventional I / O DMA data copies. These conventional copies include driver calls, interrupts, and memory-mapped I / O (MMIO) accesses, all of which are inefficient compared to simple memory accesses.At the same time, the ability to access the GPU-attached memories 420 through 423 without cache coherence overheads can be critical to the execution time of a swapped computation. For example, in cases with significant streaming traffic when writing to the memories, the cache coherence overhead can significantly reduce the effective write bandwidth seen by a GPU 410 through 413. Operand setup efficiency, result access efficiency, and GPU computation efficiency all play a role in determining the effectiveness of GPU swapping.

[0101] A choice between GPU bias and host processor bias can be driven by a bias tracker data structure. For example, a bias table can be used, which can have a page-granular structure (i.e., controlled at the granularity of a memory page) containing 1 or 2 bits per memory page attached to the GPU. The bias table can be implemented in a stolen memory area of ​​one or more GPU-attached memories 420 to 423, with or without a bias cache in the GPU 410 to 413 (e.g., to cache frequently / recently used bias table entries). Alternatively, the entire bias table can be maintained within the GPU.

[0102] In one implementation, the bias table entry associated with each access to the GPU-attached memory 420-423 causes the following operations prior to the actual GPU memory access. First, local requests from the GPU 420-423 whose page is found in the GPU bias are forwarded directly to a corresponding GPU memory 410-413. Local requests from the GPU whose page is found in the host bias are forwarded to the processor (e.g., via a high-speed connection, as discussed above). Requests from the host processor 405 that find the requested page in the host processor bias optionally complete the request like a normal memory read. Alternatively, requests directed to a GPU-biased page may be forwarded to the GPU 410-413.The GPU can then pass the page to a host processor bias if it is not currently using the page. The bias state of a page can be changed either through a software-based mechanism, a hardware-assisted software-based mechanism, or, for a limited set of cases, a purely hardware-based mechanism. A mechanism for changing the bias state employs an API call (e.g., OpenCL), which in turn invokes the GPU's device driver, which in turn sends a message to the GPU (or queues a command descriptor) instructing it to change the bias state and, for some transitions, perform a cache flush operation on the host. The cache flush operation is required for a transition from host processor bias to GPU bias, but is not required for the reverse transition.

[0103] Cache coherence can be maintained by temporarily uncaching GPU-biased pages by the host processor. To access these pages, the host processor can request access from GPU 410, which may or may not grant access immediately depending on the implementation. Thus, to reduce communication between the host processor and GPU 410, it is beneficial to ensure that GPU-biased pages are those required by the GPU but not by host processor 405, and vice versa. Graphics processing pipeline

[0104] Fig. 5 illustrates a graphics processing pipeline 500. A graphics multiprocessor, such as the graphics multiprocessor 234 as shown in Fig. 2D, or the graphics multiprocessor 235 of the Fig. 2E may implement the graphics processing pipeline 500. The graphics multiprocessor may be included in the parallel processing subsystems described herein, such as parallel processor 200 of the Fig. 2A, which is connected to the parallel processors 112 of the Fig. 1 and may be used instead thereof. The various parallel processing systems, as described herein, may implement the graphics processing pipeline 500 via one or more instances of the parallel processing unit (e.g., the parallel processing unit 202 of Fig. 2A). For example, a shader unit (e.g., the graphics multiprocessor 234 of the Fig. 2C) may be configured to perform the functions of one or more of a vertex processing unit 504, a tessellation control processing unit 508, a tessellation evaluation processing unit 512, a geometry processing unit 516, and a fragment / pixel processing unit 524. The functions of the data assembler 502, the primitive assembler 506, 514, 518, the tessellation unit 510, the rasterization unit 522, and the raster operation unit 526 may also be performed by other processing engines within a processing cluster (e.g., the processing cluster 214 of the Fig. 2A) and a corresponding partition unit (e.g. the partition unit 220A to 220N of the Fig. 2A). The graphics processing pipeline 500 may also be implemented using dedicated processing units for one or more functions. It is also possible for one or more sections of the graphics processing pipeline 500 to be performed by parallel processing logic within a general-purpose processor (e.g., a CPU). Optionally, one or more sections of the graphics processing pipeline 500 may be accessed via a memory interface 528, which may be an instance of the memory interface 218 of Fig. 2A, to an on-chip memory (e.g., the parallel processor memory 222 as in Fig. 2A). The graphics processor pipeline 500 may also have a multi-core group 365A as in Fig. 3 be implemented.

[0105] Data assembler 502 is a processing unit that can collect vertex data for surfaces and primitives. Data assembler 502 then outputs the vertex data, including the vertex attributes, to vertex processing unit 504. Vertex processing unit 504 is a programmable execution unit that executes vertex shader programs and illuminates and transforms vertex data as specified by the vertex shader programs. Vertex processing unit 504 reads data stored in cache, local memory, or system memory for use in processing the vertex data and can be programmed to transform the vertex data from an object-based coordinate representation into a world-space coordinate space or a normalized device coordinate space.

[0106] A first instance of a primitive assembler 506 receives vertex attributes from the vertex processing unit 504. The primitive assembler 506 reads stored vertex attributes as needed and constructs graphics primitives for processing by the tessellation controller processing unit 508. The graphics primitives include triangles, line segments, points, patches, and so on, as supported by numerous graphics processing application programming interfaces (APIs).

[0107] The tessellation control processing unit 508 processes the input vertices (control points) for a geometric patch. The control points are transformed from an input representation of the patch (e.g., the bases of the patch) into a representation suitable for use in surface evaluation by the tessellation evaluation processing unit 512. The tessellation control processing unit 508 may also calculate tessellation factors for edges of geometric patches. A tessellation factor applies to a single edge and quantifies a view-dependent level of detail associated with the edge.A tessellation unit 510 is configured to receive tessellation factors for edges of a patch and to tessellate the patch into multiple geometric primitives, such as a line, triangle, or quadrilateral primitives, which are transmitted to a tessellation evaluation processing unit 512. The tessellation evaluation processing unit 512 operates on parameterized coordinates of the subdivided patch to generate a surface representation and vertex attributes for each vertex associated with the geometric primitives.

[0108] A second instance of a primitive assembler 514 receives the vertex attributes from the tessellation evaluation processing unit 512, reads the stored vertex attributes as needed, and constructs graphics primitives for processing by the geometry processing unit 516. The geometry processing unit 516 is a programmable execution unit that executes geometry shader programs to transform graphics primitives received from the primitive assembler 514 as specified by the geometry shader programs. The geometry processing unit 516 may be programmed to subdivide the graphics primitives into one or more new graphics primitives and calculate parameters used to rasterize the new graphics primitives.

[0109] The geometry processing unit 516 may be capable of adding or deleting elements from the geometry stream. The geometry processing unit 516 outputs the parameters and vertices that specify new graphics primitives to the primitive assembler 518. The primitive assembler 518 receives the parameters and vertices from the geometry processing unit 516 and constructs graphics primitives for processing by a viewport scaling, suppression, and clipping unit 520. The geometry processing unit 516 reads data stored in the parallel processor memory or the system memory for use in processing the geometry data. The viewport scaling, suppression, and clipping unit 520 performs clipping, suppression, and scaling of the viewport and outputs processed graphics primitives to a rasterization unit 522.

[0110] Rasterization unit 522 may perform depth suppression and other depth-based optimizations. Rasterization unit 522 also performs scan conversion on the new graphics primitives to generate fragments and outputs these fragments and the associated coverage data to fragment / pixel processing unit 524. Fragment / pixel processing unit 524 is a programmable execution unit configured to execute fragment shader programs or pixel shader programs. Fragment / pixel processing unit 524 transforms fragments or pixels received from rasterization unit 522 as specified by the fragment or pixel shader programs.For example, fragment / pixel processing unit 524 may be programmed to perform operations including, but not limited to, texture mapping, shading, blending, texture correction, and perspective correction to generate shaded fragments or pixels that are output to a raster operations unit 526. Fragment / pixel processing unit 524 may read data stored in either parallel processor memory or system memory for use when processing the fragment data. Fragment or pixel shader programs may be configured to shade samples, pixels, tiles, or other granularities depending on the sampling rate configured for the processing units.

[0111] The raster operation unit 526 is a processing unit that performs raster operations, including, but not limited to, stenciling, Z-testing, blending, and the like, and processes pixel data as processed graphics data stored in the graphics memory (e.g., the parallel processor memory 222 as shown in Fig. 2A and / or the system memory 104 as in Fig. 1) to be stored for display on the one or more display devices 110A, 110B, or for further processing by one of the one or more processors 102 or parallel processors 112. The raster operation unit 526 may be configured to compress z or color data written to the memory and decompress z or color data read from the memory. Overview of Machine Learning

[0112] The architecture described above can be applied to perform training and inference operations using machine learning models. Machine learning has proven successful in solving many types of tasks. The computations involved in training and using machine learning algorithms (e.g., neural networks) are inherently amenable to efficient parallel implementations. Accordingly, parallel processors such as general-purpose graphics processors (GPGPUs) have played a significant role in the practical implementation of deep neural networks. Parallel graphics processors with single-instruction multithreaded (SIMT) architectures are designed to maximize the amount of parallel processing in the graphics pipeline.In a SIMT architecture, groups of parallel threads attempt to execute program instructions synchronously together as often as possible to increase processing efficiency. The efficiency provided by implementations of parallel machine learning algorithms allows for the use of high-capacity networks and enables these networks to be trained on larger datasets.

[0113] A machine learning algorithm is an algorithm that can learn based on a set of data. For example, a machine learning algorithm can be designed to model high-level abstractions within a dataset. For example, image recognition algorithms can be used to determine which of several categories belongs to which given input; regression algorithms can output a numerical value given an input; and pattern recognition algorithms can be used to generate translated text or to perform text-to-speech and / or speech recognition.

[0114] An example type of machine learning algorithm is a neural network. There are many types of neural networks; a simple type of neural network is a feedforward network. A feedforward network can be implemented as an acyclic graph in which the nodes are arranged in layers. Typically, the topology of a feedforward network includes an input layer and an output layer separated by at least one hidden layer. The hidden layer transforms an input received at the input layer into a representation useful for generating an output at the output layer. The network nodes are fully connected by edges to the nodes in adjacent layers, but there are no edges between the nodes within each layer.Data received at the nodes of an input layer of a feedforward network is passed (i.e., "fed forward") to the nodes of the output layer via an activation function, which calculates the states of the nodes of each subsequent layer in the network based on coefficients ("weights") associated with each of the edges connecting the layers. Depending on the specific model represented by the algorithm being executed, the output from the neural network algorithm can take various forms.

[0115] Before a machine learning algorithm can be used to model a specific problem, the algorithm is trained using a training dataset. Training a neural network involves selecting a network topology, using a training dataset representing a problem modeled by the network, and adjusting the weights until the network model works with minimal error for all instances of the training dataset.For example, during a supervised learning training process for a neural network, the output produced by the network in response to the input representing an instance in a training dataset is compared to the designated "correct" output for that instance, an error signal representing the difference between the output and the designated output is calculated, and the weights associated with the connections are adjusted to minimize this error as the error signal propagates backward through the layers of the network. The network is considered "trained" when the errors for each of the outputs produced from the instances in the training dataset are minimized.

[0116] The accuracy of a machine learning algorithm can be significantly affected by the quality of the dataset used to train the algorithm. The training process can be computationally intensive and may require a significant amount of time on a conventional general-purpose processor. Accordingly, parallel processing hardware is used to train many types of machine learning algorithms. This is particularly useful for optimizing the training of neural networks, as the computations performed when adjusting the coefficients in neural networks naturally lend themselves to parallel implementations. In particular, many machine learning algorithms and software applications have been adapted to take advantage of the parallel processing hardware in general-purpose graphics processing devices.

[0117] Fig. 6 is a generalized diagram of a machine learning software stack 600. A machine learning application 602 is any logic that can be configured to train a neural network using a training dataset or to use a trained deep neural network to implement machine intelligence. The machine learning application 602 can include neural network training and inference functionality and / or specialized software that can be used to train a neural network prior to deployment. The machine learning application 602 can implement any type of machine intelligence, including, but not limited to, image recognition, matching, and localization, autonomous navigation, speech synthesis, medical imaging, or language translation.Example machine learning applications 602 include, but are not limited to, voice-based virtual assistants, image or facial recognition algorithms, autonomous navigation, and the software tools used to train the machine learning models used by the machine learning applications 602.

[0118] Hardware acceleration for the machine learning application 602 may be enabled via a machine learning framework 604. The machine learning framework 604 may provide a library of machine learning primitives. Machine learning primitives are basic operations typically performed by machine learning algorithms. Without the machine learning framework 604, developers of machine learning algorithms would need to create and optimize the core computational logic associated with the machine learning algorithm, and then re-optimize the computational logic whenever new parallel processors are developed. Instead, the machine learning application may be configured to perform the necessary computations using primitives provided by the machine learning framework 604.Example primitives include tensor convolutions, activation functions, and pooling, which are computational operations performed during training of a convolutional neural network (CNN). Machine learning framework 604 may also provide primitives for implementing basic linear algebra subroutines performed by machine learning algorithms, such as matrix and vector operations. Examples of a machine learning framework 604 include, but are not limited to, TensorFlow, TensorRT, PyTorch, MXNet, Caffe, and other high-level machine learning frameworks.

[0119] The machine learning framework 604 may process input data received from the machine learning application 602 and generate the appropriate input for a compute framework 606. The compute framework 606 may abstract the underlying instructions provided to the GPGPU driver 608 to enable the machine learning framework 604 to take advantage of hardware acceleration via the GPGPU hardware 610 without requiring the machine learning framework 604 to have detailed knowledge of the architecture of the GPGPU hardware 610. Furthermore, the compute framework 606 may enable hardware acceleration for the machine learning framework 604 across a variety of types and generations of the GPGPU hardware 610. In one example, the compute frameworks 606 may include the CUDA compute framework and associated machine learning libraries, such as the CUDA Deep Neural Network (cuDNN) library.The machine learning software stack 600 may also include communication libraries or frameworks to enable multi-GPU and multi-node computing. Accelerating GPGPU machine learning

[0120] Fig. Figure 7 illustrates a general-purpose graphics processing unit (GPGPU 700) that the parallel processor 200 of Fig. 2A or the one or more parallel processors 112 of Fig. 1. The general-purpose processing unit may be configured to provide support for hardware acceleration of primitives provided by a machine learning framework to facilitate processing of the type of computational workloads associated with training deep neural networks. The GPGPU 700 may also be directly linked with other instances of the GPGPU to create a multi-GPU cluster to improve training speed for particularly deep neural networks. The primitives are also supported to accelerate inference operations for deployed neural networks.

[0121] The GPGPU 700 includes a host interface 702 for enabling a connection to a host processor. The host interface 702 may be a PCI Express interface. However, the host interface may also be a vendor-specific communication interface or a communication fabric. The GPGPU 700 receives commands from the host processor and uses a global scheduler 704 to distribute the execution threads associated with those commands among a set of processing clusters 706A-706H. The processing clusters 706A-706H share a cache 708. The cache 708 may serve as a higher-level cache for caches within the processing clusters 706A-706H. The illustrated processing clusters 706A-706H may be interconnected with the processing clusters 214A-214N as shown in Fig. 2A.

[0122] The GPGPU 700 includes memories 714A, 714B, which are coupled to the processing clusters 706A through 706H via a set of memory controllers 712A, 712B. The memories 714A, 714B may include various types of memory devices, including dynamic random access memory (DRAM) or graphics random access memory, such as synchronous graphics random access memory (SGRAM), including double data rate graphics memory (GDDR). The memories 714A, 714B may also include 3D stacked memories, including, but not limited to, high bandwidth memory (HBM).

[0123] Each of the processing clusters 706A to 706H may include a set of graphics multiprocessors, such as the graphics multiprocessor 234 of Fig. 2D, the graphics multiprocessor 235 from Fig. 2E, or may include a multi-core group 365A to 365N as shown in Fig. 3. The graphics multiprocessors of the compute cluster include multiple types of integer and floating-point logic units capable of performing computational operations in a range of accuracies, including those suitable for machine learning computations. For example, at least a subset of the floating-point units in each of the processing clusters 706A through 706H may be configured to perform 16-bit or 32-bit floating-point operations, while another subset of the floating-point units may be configured to perform 64-bit floating-point operations.

[0124] Multiple instances of the GPGPU 700 may be configured to function as a compute cluster. The communication mechanism used by the compute cluster for synchronization and data exchange varies in different embodiments. For example, the multiple instances of the GPGPU 700 communicate with each other via the host interface 702. In one embodiment, the GPGPU 700 includes an I / O hub 709 that couples the GPGPU 700 to a GPU link 710 that enables direct connection to other instances of the GPGPU. The GPU link 710 may be coupled to a dedicated GPU-to-GPU bridge that enables communication and synchronization between multiple instances of the GPGPU 700. Optionally, the GPU link 710 is coupled to a high-speed interconnect to send and receive data to other GPGPUs or parallel processors.The multiple instances of GPGPU 700 may be located in separate computing systems and communicate via a network device accessible via host interface 702. GPU link 710 may be configured to enable connection to a host processor in addition to, or as an alternative to, host interface 702.

[0125] While the illustrated configuration of GPGPU 700 may be configured to train neural networks, an alternative configuration of GPGPU 700 may be configured for use in a high-performance or low-power inference platform. In an inference configuration, GPGPU 700 includes a smaller number of processing clusters 706A through 706H relative to the training configuration. Additionally, the memory technology associated with memories 714A, 714B may be different for inference and training configurations. In one embodiment, the inference configuration of GPGPU 700 may support inference-specific instructions. For example, an inference configuration may provide support for one or more 8-bit integer or floating-point dot product instructions commonly used during inference operations for deployed neural networks.

[0126] Fig. 8 illustrates a multi-GPU computing system 800. The multi-GPU computing system 800 may include a processor 802 coupled to a plurality of GPGPUs 806A through 806D via a host interface switch 804. The host interface switch 804 may be a PCI Express switch device that couples the processor 802 to a PCI Express bus over which the processor 802 can communicate with the set of GPGPUs 806A through 806D. Each of the plurality of GPGPUs 806A through 806D may be an instance of the GPGPU 700 of Fig. 7. The GPGPUs 806A to 806D may be interconnected via a set of high-speed point-to-point GPU-to-GPU links (P2P GPU links 816). The high-speed GPU-to-GPU links may be connected via a dedicated GPU link, such as GPU link 710, as shown in Fig. 7, to each of the GPGPUs 806A to 806D. The P2P GPU links 816 enable direct communication between each of the GPGPUs 806A to 806D without requiring communication over the host interface bus to which the processor 802 is connected. When the GPU-to-GPU traffic is directed to the P2P GPU links, the host interface bus remains available for system memory access or for communicating with other instances of the multi-GPU computing system 800, for example, over one or more network devices. Although in Fig. 8, GPGPUs 806A through 806D are connected to processor 802 via host interface switch 804. Alternatively, processor 802 may include direct support for P2P GPU links 816 and be directly connected to GPGPUs 806A through 806D. In one embodiment, P2P GPU link 816 enables multi-GPU computing system 800 to operate as a single logical GPU. Implementations of neural networks for machine learning

[0127] The computer architecture described herein can be configured to perform the types of parallel processing particularly suitable for training and deploying neural networks for machine learning. A neural network can be generalized as a network of functions that have a graph relationship. As is known in the art, there are a variety of types of neural network implementations used in machine learning. One exemplary type of neural network is, as described above, the feedforward network. A second exemplary type of neural network is the convolutional neural network (CNN), while a third exemplary type of neural network is recurrent neural networks (RNNs).

[0128] A CNN is a specialized feedforward neural network for processing data that has a known grid-like topology, such as image data. Accordingly, CNNs are typically used for computer vision and image recognition applications, but they can also be used for other types of pattern recognition, such as speech and language processing. The nodes of the CNN's input layer are organized into a set of "filters" (feature detectors inspired by the receptive fields found in the retina), and the output of each set of filters is propagated to nodes in successive layers of the network. The computations for a CNN involve applying the mathematical operation of convolution to each filter to produce that filter's output.A convolution is a specialized type of mathematical operation performed on two functions to produce a third function that is a modified version of one of the two original functions. In convolutional neural network terminology, the first function of the convolution may be referred to as the input, while the second function may be referred to as the convolution kernel. The output may be referred to as the feature map. For example, the input to a convolutional layer may be a multidimensional array of data defining the various color components of an input image. The convolution kernel may be a multidimensional array of parameters, with the parameters being adjusted through the training process for the neural network.

[0129] Recurrent neural networks (RNNs) are a family of feedforward neural networks that incorporate feedback connections between layers. RNNs enable the modeling of sequential data by sharing parameter data across different parts of the neural network. The architecture for an RNN includes cycles that represent the influence of a variable's current value on its own value at a future time, as at least a portion of the RNN's output data is used as feedback for processing subsequent input in a sequence. This feature makes RNNs particularly suitable for language processing due to the variable nature of speech data.

[0130] The figures described below illustrate exemplary feedforward, CNN, and RNN networks and describe a general process for training and deploying each of these types of networks, respectively. It should be understood that these descriptions are exemplary and not limiting of any specific embodiment described herein, and that the concepts illustrated can be applied generally to deep neural networks and machine learning techniques in general.

[0131] Deep neural networks used in deep learning typically involve a front-end network for performing feature detection coupled with a back-end network representing a mathematical model that can perform operations (e.g., object classification, speech recognition, etc.) based on the feature representation provided to the model. Deep learning allows machine learning to be performed without requiring manual feature construction for the model. Instead, deep neural networks can learn features based on a statistical structure or correlation within the input data. The learned features can be provided to a mathematical model that can map detected features to an output.The mathematical model used by the network is generally specialized for the specific task to be performed, and other models are used to perform other tasks.

[0132] Once the neural network is structured, a learning model can be applied to the network to train the network to perform specific tasks. The learning model describes how to adjust the weights within the model to reduce the network's output error. Backpropagation of errors is a common technique used to train neural networks. An input vector is presented to the network for processing. The network's output is compared to the desired output using a loss function, and an error value is calculated for each of the neurons in the output layer. The error values ​​are then backpropagated until each neuron has an associated error value that roughly represents its contribution to the original output.The network can then learn from these errors using an algorithm, such as the stochastic gradient descent algorithm, to update the weights of the neural network.

[0133] The Fig. Figures 9A to 9B show an example convolutional neural network. Fig. Figure 9A illustrates different layers within a CNN. As shown in Fig. As shown in Figure 9A, an exemplary CNN used to model image processing may receive an input 902 describing the red, green, and blue (RGB) components of an input image. The input 902 may be processed by multiple convolutional layers (e.g., a first convolutional layer 904, a second convolutional layer 906). The output from the multiple convolutional layers may optionally be processed by a set of fully connected layers 908. Neurons in a fully connected layer have full connections with all activations in the previous layer, as previously described for a feedforward network. The output from the fully connected layers 908 may be used to generate an output result from the network.The activations within the fully connected layers 908 may be computed using matrix multiplications instead of convolutions. Not all CNN implementations use fully connected layers 908. For example, in some implementations, the second convolutional layer 906 may generate an output for the CNN.

[0134] The convolutional layers are sparsely connected, which differs from the conventional neural network configuration found in the fully connected layers 908. The layers of conventional neural networks are fully connected, so each output unit interacts with each input unit. However, the convolutional layers are sparsely connected because the output of the convolution of an array (rather than the respective state value of each of the nodes in the array) is input to the nodes of the subsequent layer, as illustrated. The kernels associated with the convolutional layers perform convolution operations, the output of which is sent to the next layer. The dimensionality reduction performed within the convolutional layers is one aspect that allows the CNN to scale to process large images.

[0135] Fig. Figure 9B illustrates exemplary computation stages within a convolutional layer of a CNN. An input to a convolutional layer 912 of a CNN may be processed in three stages of a convolutional layer 914. The three stages may include a convolution stage 916, a detector stage 918, and a pooling stage 920. The convolutional layer 914 may then output data to a subsequent convolutional layer. The final convolutional layer of the network may generate output feature mapping data or provide input to a fully connected layer, for example, to generate a classification score for the input to the CNN.

[0136] In the convolution stage 916, multiple convolutions are performed in parallel to generate a set of linear activations. The convolution stage 916 may include an affine transformation, which is any transformation that can be specified as a linear transformation plus a translation. Affine transformations include rotations, translations, scaling, and combinations of these transformations. The convolution stage computes the output of functions (e.g., neurons) connected to specific regions in the input, which can be determined as the local region associated with the neuron. The neurons compute a dot product between the weights of the neurons and the region in the local input to which the neurons are connected. The output from the convolution stage 916 defines a set of linear activations that are processed by successive stages of the convolution layer 914.

[0137] The linear activations can be processed by a detector stage 918. In the detector stage 918, each linear activation is processed by a nonlinear activation function. The nonlinear activation function increases the nonlinear properties of the overall network without affecting the receptive fields of the convolutional layer. Several types of nonlinear activation functions can be used. One specific type is the rectified linear unit (ReLU), which uses an activation function defined as f(x) = max(0, x), so that the activation has a threshold of zero.

[0138] The pooling stage 920 uses a pooling function that replaces the output of the second convolutional layer 906 with a summary statistic of the nearby outputs. The pooling function can be used to introduce translation invariance into the neural network, so that small translations of the input do not change the pooled outputs. Invariance to local translation can be useful in scenarios where the presence of a feature in the input data is more important than the exact location of the feature. Various types of pooling functions can be used during the pooling stage 920, including max pooling, mean pooling, and 12-norm pooling. Some CNN implementations also do not include a pooling stage. Instead, these implementations employ an additional convolution stage that has an increased stride size relative to previous convolution stages.

[0139] The output from the convolutional layer 914 may then be processed by the next layer 922. The next layer 922 may be an additional convolutional layer or one of the fully connected layers 908. For example, the first convolutional layer 904 of Fig. 9A may perform an output to the second convolutional layer 906, while the second convolutional layer may perform an output to a first layer of the fully connected layers 908.

[0140] A variant of the CNN is a deep belief convolutional network, which has a similar structure to a CNN and is trained in a similar way to a deep belief network. A deep belief network (DBN) is a generative neural network composed of multiple layers of stochastic (random) variables. DBNs can be trained layer by layer using eager unsupervised learning. The learned weights of the DBN can then be used to provide pre-training neural networks by determining an optimal initial set of weights for the neural network.

[0141] The Fig. Figures 10A to 10B illustrate example language models. Fig. Figure 10A illustrates a recurrent neural network (RNN). In a recurrent neural network (RNN), the previous state of the network influences the output of the current state of the network. RNNs can be constructed in many different ways using a variety of features. The general idea behind using RNNs is to use mathematical models to predict the future based on a previous sequence of inputs. For example, an RNN can be used to perform statistical language modeling to predict an upcoming word based on a previous sequence of words.The illustrated RNN 1000 can be described as having an input layer 1002 that receives an input vector, hidden layers 1004 for implementing a recurrent function, a feedback mechanism 1005 for enabling a "memory" of previous states, and an output layer 1006 for outputting a result. The RNN 1000 operates based on time steps. The state of the RNN at a given time step is influenced based on the previous time step via the feedback mechanism 1005. For a given time step, the state of the hidden layers 1004 is defined by the previous state and the input at the current time step. An initial input (x1) at a first time step can be processed by the hidden layer 1004.A second input (x2) may be processed by hidden layer 1004 using state information determined during processing of the initial input (x1). A given state may be denoted as s. t = f(Ux t + Ws t-1), where U and W are parameter matrices. The function f is generally a nonlinearity, such as the hyperbolic tangent function (tanh) or a variant of the rectification function f (x) = max(0, x). However, the specific mathematical function used in the hidden layers 1004 may vary depending on the specific implementation details of the RNN 1000. Acceleration for variations in RNN networks may also be enabled. An exemplary RNN variant is the long short-term memory (LTSM) RNN. LTSM RNNs are suitable for learning long-term dependencies, which may be necessary for processing longer speech sequences.

[0142] Fig. Figure 10B illustrates baseline components of a transformer model 1010. The transformer model 1010 addresses problems found in RNN models with long input sequences and enables greater parallelization. The transformer model 1010 and variants thereof are used to build language models (LLMS) that perform various tasks, such as machine translation, automatic summarization, and dialogue management. A transformer model 1010 can also be configured to perform image generation tasks, such as text-to-image generation.

[0143] The transformer model 1010 includes multiple instances of an encoder 1016 and a decoder 1026. The encoders are stacked end-to-end, with the output of the last encoder routed as input to a layer of multi-head attention by each decoder 1026. An input entering the bottommost instance of encoder 1016 is processed by input embeddings 1012, which convert input tokens into vectors that can be processed by encoder 1016. The output dictionary is vectorized by output embeddings 1022 before entering the bottommost instance of decoder 1026. The position encodings 1014, 1024 are added to input vectors and output vectors at the bottom of the encoder 1016 and decoder stacks 1026.The position encodings 1014, 1024 inject information about the relative or absolute position of tokens into a sequence of tokens to be processed when the transformer model 1010 encodes the order of the tokens in a non-natural manner.

[0144] The encoder 1016 of the transformer model 1010 analyzes the input text and generates a number of hidden states that protect the context and meaning of the text data. The layers of the encoder 1016 form part of the core of the transformer architecture, although variants of the transformer model 1010 are also possible through the decoder 1026. The encoder 1016 includes two sublayers: a multi-head attention (MHA) sublayer and a feedforward network (FFN) sublayer. The MHA sublayer performs multiple simultaneous self-attention operations to compute attention scores that enable the transformer model 1010 to context-awarely weigh the importance and relative relationships of different tokens in the input sequence. The FFN sublayer is a positionally fully connected feedforward neural network.The output of each sublayer is processed by an addition and normalization (A&N) operation defined as LayerNorm(X + Sublayer(x)), where the sublayer's output is added to the sublayer's input and normalized. In some implementations, the normalization operation may be performed before the sublayer instead of after.

[0145] Decoder 1026 includes three sublayers: a masked MHA sublayer, an MHA sublayer, and an FFN sublayer. The masked MHA sublayer is similar to the MHA layer, except that masking is applied to prevent the query position from serving the keys of future positions. The MHA sublayer of decoder 1026 is similar to the MHA sublayer of the encoder with an additional input from encoder 1016. The FFN of decoder 1026 is the same as the FFN of encoder 1016. The Linear and SoftMax block takes the output of the last instance of decoder 1026 of the decoder stack and generates a probability distribution representing the output probabilities.

[0146] GPU acceleration is also available for variants of the Transformer Model 1010 that replace some or all FFN sublayers with a sparse mixture of experts (MoE). The MoE layers each contain a number of experts, with each expert being a neural network. The MoE layers can be FFNs themselves or they can also be MoEs, which allows for hierarchical MoE layers.

[0147] The training of a transformer model 1010 can be optimized through the use of adaptive accuracy logic, which adjusts the computational accuracy applied during training. The adaptive accuracy logic can attempt to use the smallest possible data type during training without significantly reducing training accuracy. For example, to minimize data loss due to the use of 16-bit, 8-bit, and 4-bit floating-point formats, dynamic scaling and shaping can be applied during training based on the statistical analysis of the tensor data generated during training. Tensor data generated during training can be statistically analyzed using a variety of analysis techniques to determine a set of scaling factors to be applied to the data blocks, such as the input to each layer of the transformer model 1010.For example, absolute minimum and / or maximum values ​​can be used to determine scaling factors that can prevent underflow or overflow of low-precision floating-point data types.

[0148] Fig. Figure 11 illustrates training and deployment of a deep neural network. Once a given network has been structured for a task, the neural network is trained using a training dataset 1102. Various training frameworks 1104 have been developed to enable hardware acceleration of the training process. For example, the machine learning framework 604 of Fig. 6 may be configured as a training framework 1104. The training framework 1104 may interface with an untrained neural network 1106 and allow the untrained neural network to be trained using the parallel processing resources described herein to generate a trained neural network 1108. To start the training process, the initial weights may be chosen randomly or through pre-training using a deep belief network. The training cycle may then be performed in either a supervised or unsupervised manner.

[0149] Supervised learning is a learning method in which training is performed as a mediated operation, e.g., where the training dataset 1102 contains inputs paired with the desired output for the input, or where the training dataset includes an input that has a known output, and where the output of the neural network is manually ranked. The network processes the inputs and compares the resulting outputs to a set of expected or desired outputs. Errors are then propagated back through the system. The training framework 1104 may adjust the weights that control the untrained neural network 1106. The training framework 1104 may provide tools to monitor how well the untrained neural network 1106 converges toward a model capable of generating correct answers based on known input data.The training process occurs repeatedly as the network weights are adjusted to refine the output generated by the neural network. The training process may continue until the neural network achieves a statistically desirable accuracy associated with a trained neural network 1108. The trained neural network 1108 may then be used to implement any number of machine learning operations to generate an inference result 1114 based on an input of new data 1112.

[0150] Unsupervised learning is a learning method in which the network attempts to train itself using unlabeled data. Thus, the training dataset 1102 for unsupervised learning will include input data without associated output data. The untrained neural network 1106 can learn groupings within the unlabeled input and can determine how individual inputs relate to the overall dataset. Unsupervised training can be used to generate a self-organizing association, which is a type of trained neural network 1108 capable of performing operations useful for reducing the dimensionality of data. Unsupervised training can also be used to perform anomaly detection, which enables the identification of data points in an input dataset that deviate from the normal patterns of the data.

[0151] Variations of supervised and unsupervised training can also be used. Semi-supervised learning is a technique in which the training dataset 1102 contains a mixture of labeled and unlabeled data with the same distribution. Incremental learning is a variation of supervised learning in which input data is continuously used to further train the model. Incremental learning allows the trained neural network 1108 to adapt to new data 1112 without forgetting the knowledge imparted to the network during initial training. Regardless of whether the training process is supervised or unsupervised, for particularly deep neural networks, it may be too computationally intensive for a single compute node. Instead of using a single compute node, a distributed network of compute nodes can be used to accelerate the training process.

[0152] Fig. 12A is a block diagram illustrating distributed learning. Distributed learning is a training model that uses multiple distributed computing nodes to perform supervised or unsupervised training of a neural network. The distributed computing nodes may each include one or more host processors and one or more of the general-purpose processing nodes, such as the highly parallel general-purpose graphics processing unit 700 of Fig. 7. As illustrated, distributed learning can be performed with model parallelism 1202, data parallelism 1204, or a combination of model and data parallelism 1206.

[0153] With model parallelism 1202, different computing nodes in a distributed system can perform training computations for different parts of a single network. For example, each layer of a neural network can be trained by a different processing node of the distributed system. The advantages of model parallelism include the ability to scale up to particularly large models. Dividing the computations associated with different layers of the neural network enables the training of very large neural networks in which the weights of all layers would not fit into the memory of a single computing node. In some cases, model parallelism can be particularly useful for performing unsupervised training of large neural networks. Some implementations of model parallelism 1202 may also be referred to as tensor parallelism.

[0154] In data parallelism 1204, the different nodes of the distributed network have a complete instance of the model, and each node receives a different portion of the data. The results from the different nodes are then combined. Although different approaches to data parallelism are possible, all data-parallel training approaches require a technique for combining results and synchronizing model parameters between the individual nodes. Example approaches to combining data include parameter averaging and update-based data parallelism. Parameter averaging trains each node on a subset of the training data and sets the global parameters (e.g., weights, biases) to the mean of the parameters of each node. Parameter averaging uses a central parameter server that maintains the parameter data.Update-based data parallelism is similar to parameter averaging, except that instead of transmitting parameters from nodes to the parameter server, model updates are transmitted. Additionally, update-based data parallelism can be implemented in a decentralized manner, with updates compressed and transmitted between nodes.

[0155] The combined model and data parallelism 1206 may, for example, be implemented in a distributed system in which each compute node includes multiple GPUs. The combined model and data parallelism 1206 may also be referred to as hybrid parallelism. Each node may have a complete instance of the model, with separate GPUs within each node being used to train different parts of the model. Distributed training has increased overhead relative to training on a single machine. However, the parallel processors and GPGPUs described herein may each implement various techniques for reducing the overhead of distributed learning, including techniques for enabling high-bandwidth data transfer between GPUs and accelerated remote data synchronization.Pipeline parallelism is a variant of combined model and data parallelism, where different nodes contain less than the entire model but more than a single layer of the model. Pipeline parallelism distributes different groups of layers or submodels across different processing nodes. Another variant is expert parallelism, which routes requests for specific experts within the model to different GPUs. Expert parallelism can be used, for example, with MoE transformer models.

[0156] Fig. 12B is a block diagram illustrating a programmable network interface 1210 and a data processing unit. The programmable network interface 1210 is a programmable network engine that can be used to accelerate network-based computing tasks within a distributed environment. The programmable network interface 1210 can be coupled to a host system via a host interface 1270. The programmable network interface 1210 can be used to accelerate network or memory operations for CPUs or GPUs of the host system. The host system can, for example, be a node of a distributed learning system used to perform distributed training, such as in Fig. 12A. The host system may also be a data center node within a data center.

[0157] In one embodiment, access to a remote data store containing model data may be accelerated by the programmable network interface 1210. For example, the programmable network interface 1210 may be configured to present remote data storage devices to the host system as local data storage devices. The programmable network interface 1210 may also accelerate remote direct memory access (RDMA) operations performed between the host system's GPUs and GPUs of remote systems. In one embodiment, the programmable network interface 1210 may enable data storage functionality, such as, but not limited to, NVME-oF.The programmable network interface 1210 can also accelerate encryption, data integrity, compression, and other operations for remote data storage on behalf of the host system, allowing the remote data storage to approach the latencies of data storage devices attached directly to the host system.

[0158] The programmable network interface 1210 can also perform resource allocation and management on behalf of the host system. Data storage security operations can be offloaded to the programmable network interface 1210 and performed in conjunction with the allocation and management of remote data storage resources. Network-based operations for managing access to the remote data storage, which would otherwise be performed by a host system processor, can instead be performed by the programmable network interface 1210.

[0159] In one embodiment, network and / or data security operations may be offloaded from the host system to the programmable network interface 1210. Data center security policies for a data center node may be handled by the programmable network interface 1210 instead of the host system's processors. For example, the programmable network interface 1210 may detect and mitigate an attempted network-based attack (e.g., DDoS) on the host system, preventing the attack from impacting the host system's availability.

[0160] The programmable network interface 1210 may include a system-on-chip (SoC 1220) executing an operating system across multiple processor cores 1222. The processor cores 1222 may include general-purpose processor cores (e.g., CPU cores). In one embodiment, the processor cores 1222 may also include one or more GPU cores. The SoC 1220 may execute instructions stored in a memory device 1240. A memory device 1250 may store local operating system data. The memory device 1250 and the memory device 1240 may also be used to cache remote data for the host system. The network ports 1260A, 1260B enable connection to a network or fabric and provide network access to the SoC 1220 and, via the host interface 1270, the host system. The programmable network interface 1210 may also include an I / O interface 1275, such as a USB interface.The I / O interface 1275 can be used to couple external devices to the programmable network interface 1210 or as a debug interface. The programmable network interface 1210 also includes a management interface 1230 that enables software in the host device to manage and configure the programmable network interface 1210 and / or the SoC 1220. In one embodiment, the programmable network interface 1210 can also include one or more accelerators or GPUs 1245 for accepting offloading of parallel computing tasks from the SoC 1220, the host system, or remote systems coupled via the network ports 1260A, 1260B. Example applications for machine learning

[0161] Machine learning can be applied to solve a variety of technological problems, including, but not limited to, computer vision, autonomous driving, speech recognition, and natural language processing. Computer vision has traditionally been one of the most active research areas for machine learning applications. Applications of computer vision range from replicating human visual abilities, such as face recognition, to creating new categories of visual abilities. For example, computer vision applications can be configured to detect sound waves from the vibrations induced in objects visible in a video.Parallel processor-accelerated machine learning makes it possible to train computer vision applications using a significantly larger training dataset than was previously achievable and enables inference systems to be deployed using low-performance parallel processors.

[0162] Parallel processor-accelerated machine learning has applications in autonomous driving, including lane and traffic sign recognition, obstacle avoidance, navigation, and driving control. Accelerated machine learning techniques can be used to train driving models based on datasets that define the appropriate responses to specific training inputs. The parallel processors described here can enable rapid training of the increasingly complex neural networks used for autonomous driving solutions and enable the use of low-power inference processors in a mobile platform suitable for integration into autonomous vehicles.

[0163] Parallel processor-accelerated deep neural networks have enabled machine learning approaches for automatic speech recognition (ASR). ASR involves generating a function that computes the most probable linguistic sequence given an acoustic input sequence. Accelerated machine learning using deep neural networks has enabled the replacement of the hidden Markov models (HMMs) and Gaussian mixture models (GMMs) previously used for ASR.

[0164] Parallel processor-accelerated machine learning can also be used to accelerate natural language processing. Automatic learning workflows can use statistical inference algorithms to generate models that are resilient to erroneous or unfamiliar inputs. Example natural language processing applications include automatic machine translation between human languages.

[0165] The parallel processor platforms used for machine learning can be divided into training platforms and deployment platforms. The training platforms are generally highly parallel and include optimizations to accelerate multi-GPU single-node training and multi-node multi-GPU training. Example parallel processors suitable for training include the general-purpose graphics processing unit 700 of Fig. 7 and the multi-GPU computing system 800 from Fig. 8. The machine learning platforms used, however, generally include low-performance parallel processors suitable for use in products such as cameras, autonomous robots, and autonomous vehicles.

[0166] Machine learning techniques can also be applied to accelerate or enhance graphics processing activities. For example, a machine learning model can be trained to recognize output generated by a GPU-accelerated application and generate an upscaled version of that output. These techniques can be applied to accelerate the generation of high-resolution images for a gaming application. Numerous other graphics pipeline activities can benefit from the use of machine learning. For example, machine learning models can be trained to perform tessellation operations on geometry data to increase the complexity of geometric models, allowing geometry with a relatively low level of detail to be automatically generated into high-detail geometry.

[0167] Fig. 13 illustrates an exemplary inferencing system-on-chip (SOC 1300) suitable for performing inference using a trained model. The SOC 1300 may integrate processing components including a media processor 1302, a vision processor 1304, a GPGPU 1306, and a multi-core processor 1308. The GPGPU 1306 may be a GPGPU as described herein, such as the GPGPU 700, and the multi-core processor 1308 may be a multi-core processor described herein, such as the multi-core processors 405, 406. The SOC 1300 may additionally include an on-chip memory 1305 that may enable a shared on-chip data pool accessible to each of the processing components. The processing components can be optimized for low-power operation to enable use on a variety of machine learning platforms, including autonomous vehicles and autonomous robots.For example, an implementation of the SoC 1300 may be used as part of the main control system for an autonomous vehicle. When configured for use in autonomous vehicles, the SOC is designed and configured to comply with the relevant functional safety standards of the applicable legislation.

[0168] During operation, media processor 1302 and vision processor 1304 may cooperate to accelerate computer vision operations. Media processor 1302 may enable low-latency decoding of multiple high-resolution video streams (e.g., 4K, 8K). The decoded video streams may be written to a buffer in on-chip memory 1305. Image processing processor 1304 may then analyze the decoded video and perform preliminary processing operations on the frames of the decoded video in preparation for processing the frames using a trained image recognition model. For example, vision processor 1304 may accelerate convolution operations for a CNN used to perform image recognition on the high-resolution video data, while the backend model computations are performed by GPGPU 1306.

[0169] The multi-core processor 1308 may include control logic to support the sequencing and synchronization of data transfers and shared memory operations performed by the media processor 1302 and the vision processor 1304. The multi-core processor 1308 may also serve as an application processor for executing software applications that may utilize the inference computing power of the GPGPU 1306. For example, at least a portion of the navigation and driving logic may be implemented in software executing on the multi-core processor 1308. Such software may issue computational workloads directly to the GPGPU 1306, or the computational workloads may be issued to the multi-core processor 1308, which may offload at least a portion of these operations to the GPGPU 1306.

[0170] The GPGPU 1306 may include compute clusters, such as a low-performance configuration of processing clusters 706A through 706H within the general-purpose graphics processing unit 700. The compute clusters within the GPGPU 1306 may support instructions specifically optimized to perform inference calculations in a trained neural network. For example, the GPGPU 1306 may support instructions to perform low-precision calculations, such as 8-bit and 4-bit integer vector operations. Overview of additional systems

[0171] Fig. 14 is a block diagram of a processing system 1400. The elements of Fig. 14 have the same or similar names as the elements of any other figure described herein, describe the same elements as in the other figures, may operate or function in a similar manner, may include the same components, and may be associated with entities other than, but are not limited to, those described elsewhere herein. Processing system 1400 may be deployed in a single-processor desktop system, a multi-processor workstation system, or a server system having a large number of processors 1402 or processor cores 1407.The system 1400 may be a processing platform incorporated into a system-on-chip (SoC) integrated circuit for use in mobile, portable, or embedded devices, such as within Internet of Things (IoT) devices with wired or wireless connectivity to a local or remote network.

[0172] The system 1400 may be a processing system having components similar to those of Fig. 1. In other configurations, for example, the one or more processors 1402 or processor cores 1407 may correspond to the one or more processors 102 of the Fig. 1. The one or more graphics processors 1408 may be associated with the one or more parallel processors 112 of the Fig. 1. The external graphics processor 1418 may be one of the add-in device(s) 120 of the Fig. 1.

[0173] The processing system 1400 may include, be coupled to, or be integrated with: a server-based gaming platform, a gaming console, including a gaming and media console; a mobile gaming console, a portable gaming console, or an online gaming console. The system 1400 may be part of a mobile phone, a smartphone, a tablet computing device, or a mobile, internet-connected device such as a laptop with low internal storage capacity. The processing system 1400 may also include, be coupled to, or be integrated with: a wearable device, such as a wearable smartwatch device; smart glasses; or smart clothing configured with augmented reality (AR) features.Augmented Reality (AR) or Virtual Reality (VR) experiences that provide visual, auditory, or tactile outputs to supplement real-world visual, auditory, or tactile experiences, or that otherwise provide text, audio, graphics, video, holographic images or videos, or tactile feedback. Processing system 1400 may include or be a part of a television or set-top box device. Processing system 1400 may include, be coupled to, or be integrated with: a self-driving vehicle, such as a bus, tractor unit, passenger vehicle, motorcycle or electric bicycle, aircraft, or glider (or any combination thereof). The self-driving vehicle may use processing system 1400 to process the environment perceived around the vehicle.

[0174] The one or more processors 1402 may include one or more processor cores 1407 to process instructions that, when executed, perform operations for system and application software. At least one of the processor cores 1407 may be configured to process a specific instruction set 1409. The instruction set 1409 may support complex instruction set computing (CISC), reduced instruction set computing (RISC), or very long instruction word (VLIW) computing. In one embodiment, one of the processor cores 1407 may process a different instruction set 1409, which may include instructions to facilitate emulation of other instruction sets.The processor core 1407 may also include other processing devices, such as a digital signal processor (DSP).

[0175] The one or more processors 1402 may include a cache memory 1404. Depending on the architecture, the one or more processors 1402 may have a single internal cache or multiple levels of internal cache. In some embodiments, the cache memory is shared by different components of the one or more processors 1402. In some embodiments, the one or more processors 1402 also utilizes an external cache (e.g., a Level 3 (L3) cache or a Last Level Cache (LLC) cache (not shown)), which may be shared by the processor cores 1407 using known cache coherence techniques. A register file 1406 may additionally be included in the processor 1402 and may include different types of registers for storing different types of data (e.g.,Integer registers, floating-point registers, status registers, and an instruction pointer register. Some registers may be general-purpose registers, while other registers may be specific to the design of the processor 1402.

[0176] The one or more processors 1402 may be coupled to one or more interface buses 1410 to communicate communication signals, such as address, data, or control signals, between the one or more processors 1402 and other components in the processing system 1400. The one or more interface buses 1410, in one of these embodiments, may be a processor bus, such as a version of the Direct Media Interface (DMI) bus. However, the processor buses are not limited to the DMI bus and may include one or more Peripheral Component Interconnect buses (e.g., PCI, PCI Express), memory buses, or other types of interface buses. The one or more processors 1402 may include, for example, a memory controller 1416 and a platform controller hub 1430.The storage controller 1416 facilitates communication between a storage device and other components of the processing system 1400, while the platform controller hub (PCH) 1430 provides connections to I / O devices via a local I / O bus.

[0177] The memory device 1420 may be a dynamic random-access memory (DRAM) device, a static random-access memory (SRAM) device, a flash memory device, a phase-change memory device, or other memory device having suitable performance to serve as process memory. For example, the memory device 1420 may operate as system memory for the processing system 1400 to store data 1422 and instructions 1421 for use when the one or more processors 1402 are executing an application or process. The memory controller 1416 may optionally be coupled to an external graphics processor 1418 that can communicate with the one or more graphics processors 1408 in the one or more processors 1402 to perform graphics and media operations.In some embodiments, the graphics, media, and / or compute operations may be supported by an accelerator 1412, which is a coprocessor that may be configured to perform a specialized set of graphics, media, or compute operations. For example, the accelerator 1412 may be a matrix multiplication accelerator used to optimize machine learning or compute operations. The accelerator 1412 may be a ray tracing accelerator that may be used to perform ray tracing operations in conjunction with the graphics processor 1408. The accelerator 1412 may also be an AI accelerator or NPU to accelerate neural network training or inference operations. In one embodiment, an external accelerator 1419 may be used instead of or in conjunction with the accelerator 1412.The accelerator 1412 and / or the external accelerator 1419 may have similar functionality to the one or more accelerator devices 130 of FIG. Fig. 1.

[0178] A display device 1411 may be provided that can be connected to the one or more processors 1402. The display device 1411 may be one or more of an internal display device, such as in a mobile electronic device or a laptop device, or an external display device connected via a display interface (e.g., the DisplayPort, etc.). The display device 1411 may be a head-mounted display (HMD), such as a stereoscopic display device for use in VR or AR applications.

[0179] The platform controller hub 1430 may enable peripherals to connect to the storage device 1420 and the processor 1402 via a high-speed I / O bus. The I / O peripherals include, among others, an audio controller 1446, a network controller 1434, a firmware interface 1428, a wireless transceiver 1426, touch-sensitive sensors 1425, and a data storage device 1424 (e.g., non-volatile memory, volatile memory, a hard disk drive, flash memory, NAND, 3D NAND, 3D XPoint / Optane, etc.). The data storage device 1424 may be connected via a storage interface (e.g., SATA) or via a peripheral bus, such as a peripheral component interconnect bus (e.g., PCI, PCI Express). The touch sensors 1425 may include touchscreen sensors, pressure sensors, or fingerprint sensors.The wireless transceiver 1426 may be a Wi-Fi transceiver, a Bluetooth transceiver, or a cellular network transceiver, such as a 3G, 4G, 5G, or LTE (Long-Term Evolution) transceiver. The firmware interface 1428 enables communication with system firmware and may be, for example, a Unified Extensible Firmware Interface (UEFI). The network controller 1434 may enable network connection to a wired network. In some embodiments, a high-performance network controller (not shown) is coupled to the interface bus 1410. The audio controller 1446 may be a multi-channel high-definition audio controller. In some of these embodiments, the system 1400 includes an optional legacy I / O controller 1440 for coupling legacy devices (e.g., the Personal System 2 (PS / 2)) to the system.The platform controller hub 1430 may also be connected to one or more USB (Universal Serial Bus) controllers 1442 to connect to input devices such as keyboard and mouse combinations 1443, a camera 1444, or other USB input devices.

[0180] The processing system 1400 shown is exemplary and not limiting, as other types of data processing systems configured differently may be used. For example, an instance of the memory controller 1416 and the platform controller hub 1430 may be integrated into a discrete external graphics processor, such as the external graphics processor 1418. The platform controller hub 1430 and / or the memory controller 1416 may be external to the one or more processors 1402. For example, the memory controller 1416 and the platform controller hub 1430 may be external to the processing system 1400 and configured as a memory controller hub and a peripheral controller hub within a system chipset in communication with the one or more processors 1402.

[0181] For example, printed circuit boards (“sleds”) can be used to house components such as CPUs, memory, and other components, and are designed for increased thermal performance. Processing components, such as processors, can be located on the top of a sled, while nearby memory, such as DIMMs, is located on the bottom of the sled. Due to the improved airflow provided by this design, the components can operate at higher frequencies and power levels than in typical systems, thereby increasing performance capabilities. Furthermore, the sleds are configured to be blindly connected to power and data communication cables in a rack, enhancing their ability to be quickly removed, upgraded, reinstalled, and / or replaced.Similarly, individual components located on the sleds, such as processors, accelerators, memory, and storage drives, are configured to be easily upgraded due to their increased spacing from one another. In the exemplary embodiment, the components additionally include hardware attestation features to prove their authenticity.

[0182] A data center may use a single network architecture (“fabric”) that supports multiple other network architectures, including Ethernet and Omni-Path. The sleds may be coupled to switches via optical fibers that provide higher bandwidth and lower latency than typical twisted-pair cabling (e.g., Category 5, Category 5e, Category 6, Category 7, Category 8, etc.). Due to the high-bandwidth, low-latency interconnects and network architecture, the data center can operationally pool resources such as memory, accelerators (e.g., GPUs, graphics accelerators, FPGAs, ASICs, neural network and artificial intelligence accelerators, etc.), and data storage drives that are physically disaggregated and distribute them to compute resources (e.g.,processors) as needed, allowing the computing resources to access the pooled resources as if they were local.

[0183] A power supply or source can provide voltage and / or current to the processing system 1400 or any component or system described herein. In one example, the power supply includes an AC-DC adapter for connection to an electrical outlet. This AC power can come from renewable energy sources (e.g., solar power). In one example, the power source includes a DC power source, such as an external AC-DC converter. A power source or source can also include wireless charging hardware for charging via proximity to a charging pad. The power source can include an internal battery, an AC power supply, a motion-based power supply, a solar power supply, or a fuel cell source.

[0184] The Fig. Figures 15A to 15C illustrate computer systems and graphics processors. The elements from the Fig. 15A through 15C, which have the same or similar names as the elements of any other figure described herein, are the same elements as in the other figures and may operate or function in a similar manner, may include the same components, and may be associated with other entities such as, but not limited to, those described elsewhere herein.

[0185] Fig. 15A is a block diagram of a processor 1500, which may be a variant of one of the processors 1402 and may be used in place of one of them. Therefore, the disclosure of any features in combination with the processor 1500 herein also discloses, but is not limited to, a corresponding combination with the one or more processors 1402. The processor 1500 may include one or more processor cores 1502A through 1502N, at least one integrated memory controller 1514, and an integrated graphics processor 1508. The graphics processor 1508 may be integrated with the processor 1500 within a system chipset or it may be coupled via a system bus. The processor 1500 may include additional cores up to and including the additional core 1502N, which are represented by the dashed-line boxes. Each of the processor cores 1502A to 1502N includes one or more internal cache units 1504A to 1504N.In some embodiments, each processor core 1502A-1502N also has access to one or more shared cache units 1506. The internal cache units 1504A-1504N and the shared cache units 1506 represent a cache hierarchy within the processor 1500. The cache hierarchy may include at least one level of instruction and data caches within each processor core and one or more levels of shared mid-level caches, such as Level 2 (L2), Level 3 (L3), Level 4 (L4), or other levels of caches, with the highest cache level prior to external memory being classified as the LLC. In some embodiments, the cache coherency logic maintains coherency between the various cache units (e.g., the shared cache units 1506 and the internal cache units 1504A through 1504N).

[0186] The processor 1500 may also include one or more bus control units 1516 and a system agent core 1510. The one or more bus control units 1516 manage a set of peripheral buses, such as one or more PCI or PCI Express buses. The one or more bus control units 1516 may also manage one or more memory buses to various external storage devices (not shown). The system agent core 1510 provides management functionality for the various processor components and may include the at least one memory control unit 1514.

[0187] For example, one or more of the processor cores 1502A through 1502N may include support for concurrently executing multiple threads. The system agent core 1510 includes components for coordinating and operating the cores 1502A through 1502N during multi-threaded processing. The system agent core 1510 may also include a power control unit (PCU) that includes logic and components for regulating the power state of the processor cores 1502A through 1502N and the graphics processor 1508.

[0188] The processor 1500 may additionally include a graphics processor 1508 for performing graphics processing operations. In some embodiments, the graphics processor 1508 is coupled to the one or more shared cache units 1506 and the system agent core 1510, including the at least one memory controller 1514. The system agent core 1510 may also include a display controller 1511 for driving the graphics processor output to one or more coupled displays. The display controller 1511 may also be a separate module coupled to the graphics processor via at least one interconnect, or it may be integrated into the graphics processor 1508.

[0189] A ring- or network-based interconnect 1512 may be used to couple the internal components of processor 1500 together. However, an alternative interconnect unit, such as a point-to-point interconnect, a switched interconnect, or other techniques, including those known in the art, may be used. In some embodiments with a ring- or network-based interconnect 1512, graphics processor 1508 is coupled to ring- or network-based interconnect 1512 via an I / O connection 1513.

[0190] The example I / O interconnect 1513 represents at least one of several variations of I / O interconnects, including a chassis I / O interconnect that enables communication between various processor components and a high-performance memory module 1518, such as an embedded DRAM (eDRAM) module or a high-bandwidth memory (HBM) module. Optionally, each of the processor cores 1502A-1502N and the graphics processor 1508 may use the high-performance memory module 1518 as unified memory and / or a shared last-level cache if a DRAM memory system is also present. Optionally, one or more accelerators 1515 may also be included within the processor 1500, including, for example, an NPU for accelerating certain neural network operations.The NPU may enable lower-performance inference operations compared to using the graphics processor 1508, or may function in conjunction with the graphics processor 1508 to enable higher inference performance compared to the graphics processor 1508 alone. In one embodiment, an NPU within the one or more accelerators 1515 may include matrix or tensor acceleration logic and may be used to implement at least some of the computational operations described herein as being implementable via the graphics processor 1508.

[0191] For example, processor cores 1502A-1502N may be homogeneous cores executing the same instruction set architecture. Alternatively, processor cores 1502A-1502N may be heterogeneous in instruction set architecture (ISA), where one or more of processor cores 1502A-1502N execute a first instruction set while at least one of the other cores executes a subset of the first instruction set or a different instruction set. Processor cores 1502A-1502N may be heterogeneous in microarchitecture, where one or more relatively higher-power cores are coupled with one or more lower-power performance cores. As another example, processor cores 1502A-1502N are heterogeneous in computational capability.Furthermore, processor 1500 may be implemented on one or more chips or chiplets, or as an integrated circuit (SoC) that includes the illustrated components in addition to other components. The integrated circuit (SoC) may be implemented using multiple chiplets.

[0192] Fig. 15B is a block diagram of hardware logic of a graphics processor core block 1519 according to some embodiments described herein. In some embodiments, elements of Fig. 15B, which have the same reference numerals (or names) as the elements of any other figure described herein, will operate or function in a similar manner to those described elsewhere herein. In one embodiment, the graphics processor core block 1519 is exemplary of a partition of a graphics processor. The graphics processor core block 1519 may be included within the integrated graphics processor 1508. Fig. 15A or a discrete graphics processor, a parallel processor, and / or a compute accelerator. A graphics processor as described herein may include multiple graphics core blocks based on target performance and capability envelopes. Each graphics processor core block 1519 may include a functional block 1530 coupled to multiple graphics cores 1521A through 1521F, which may include modular blocks of fixed function logic and general-purpose programmable logic. The graphics processor core block 1519 also includes a shared cache 1536 accessible by all graphics cores 1521A through 1521F, rasterization logic 1537, and additional fixed function logic 1538.

[0193] In some embodiments, functional block 1530 includes a geometry and fixed function pipeline 1531 that may be shared by all graphics cores in graphics processor core block 1519. In various embodiments, geometry and fixed function pipeline 1531 includes a 3D geometry pipeline, a video front-end unit, a thread creator and a global thread dispatcher, and a unified return buffer manager that manages the unified return buffers. In one embodiment, functional block 1530 also includes a graphics SoC interface 1532, a graphics microcontroller 1533, and a media pipeline 1534. Graphics SoC interface 1532 provides an interface between graphics processor core block 1519 and other core blocks in a graphics processor or compute accelerator SoC.Graphics microcontroller 1533 is a programmable subprocessor configurable to manage numerous functions of graphics processor core block 1519, including thread dispatch, scheduling, and preemption interruption. Media pipeline 1534 includes logic to facilitate decoding, encoding, preprocessing, and / or postprocessing of multimedia data, including image and video data. Media pipeline 1534 implements media operations via compute requests or sampling logic within graphics cores 1521 through 1521F. One or more pixel backends 1535 may also be integrated into functional block 1530. The one or more pixel backends 1535 include a cache for storing pixel color values ​​and may perform blending operations and lossless color compression of rendered pixel data.

[0194] In one embodiment, graphics SoC interface 1532 enables graphics processor core block 1519 to communicate with general-purpose application processor cores (e.g., CPUs) and / or other components within a SoC or a system host CPU coupled to the SoC via a peripheral interface. Graphics SoC interface 1532 also enables communication with off-chip memory hierarchy elements, such as a shared last-level cache, system RAM, and / or embedded on-chip or on-package DRAM. Graphics SoC interface 1532 may also enable communication with fixed-function devices within the SoC, such as camera image acquisition pipelines, and enables the use of and / or implements global memory atomics that may be shared between graphics processor core block 1519 and CPUs within the SoC.The graphics SoC interface 1532 may also implement power management controllers for the graphics processor core block 1519 and enable an interface between a clock domain of the graphics processor core block 1519 and other clock domains within the SoC. In one embodiment, the graphics SoC interface 1532 enables receipt of command buffers from a command streamer and a global thread dispatcher configured to provide commands and instructions to each of one or more graphics cores within a graphics processor. The commands and instructions may be sent to the media pipeline 1534 when media operations are to be performed, or to the geometry pipeline and fixed function pipeline 1531 when graphics processing operations are to be performed.When computational operations need to be performed, the computational dispatch logic can dispatch the instructions to the graphics cores 1521A through 1521F, bypassing the geometry and media pipelines.

[0195] Graphics microcontroller 1533 may be configured to perform numerous scheduling and management tasks for graphics processor core block 1519. In one embodiment, graphics microcontroller 1533 may perform graphics and / or compute workload scheduling in the various vector engines 1522A to 1522F, 1524A to 1524F, and matrix engines 1523A to 1523F, 1525A to 1525F within graphics cores 1521A to 1521F. In this scheduling model, host software executing in a CPU core of an SoC including graphics processor core block 1519 may submit workloads to one of several graphics processor doorbells, which invoke a scheduling operation ai of the corresponding graphics engine.Scheduling operations include determining which workload to execute next, submitting a workload to a command streamer, prioritizing existing workloads executing in an engine, monitoring the progress of a workload, and notifying host software when a workload has completed. In one embodiment, graphics microcontroller 1533 may also facilitate low-power or idle states for graphics processor core block 1519 and provide graphics processor core block 1519 with the ability to save and restore registers within graphics processor core block 1519 across low-power state transitions, independent of the operating system and / or graphics driver software in the system.

[0196] The graphics processor core block 1519 may include more or fewer than the illustrated graphics cores 1521A through 1521F, up to N modular graphics cores. For each set of N graphics cores, the graphics processor core block 1519 may also include a shared cache 1536, which may be configured as shared memory or as a cache, rasterization unit logic 1537, and additional fixed function logic 1538 for accelerating various graphics and compute processing operations.

[0197] Within each graphics core 1521A through 1521F is a set of execution resources that can be used to perform graphics, media, and compute operations in response to requests from a graphics pipeline, media pipeline, or shader program. Graphics cores 1521A through 1521F include multiple vector engines 1522A through 1522F, 1524A through 1524F, matrix acceleration units 1523A through 1523F, 1525A through 1525D, shared local memory (SLM) caches, a sampling unit 1526A through 1526F, and a ray tracing unit 1527A through 1527F.

[0198] The 1522A through 1522F and 1524A through 1524F vector engines are general-purpose graphics processing units capable of performing floating-point and integer / fixed-point logic operations in service of graphics, media, or compute operations, including graphics, media, or compute / GPGPU programs. The 1522A through 1522F and 1524A through 1524F vector engines can operate with variable vector widths using SIMD, SIMT, or SIMT+SIMD. Matrix acceleration units 1523A through 1523F, 1525A through 1525D include matrix-to-matrix and matrix-to-vector acceleration logic that improves performance for matrix operations, particularly very low and mixed-precision matrix operations (e.g., INT8, FP16, BF16, FP8, FP4) used for machine learning. In one embodiment, matrix acceleration units 1523A through 1523F, 1525A through 1525D include support for microscale (MX) formats.In one embodiment, each of the matrix acceleration units 1523A through 1523F, 1525A through 1525D includes one or more systolic arrays of processing elements capable of performing concurrent matrix multiplication or dot product operations on matrix elements.

[0199] The sampling unit 1526A to 1526F may read media or texture data into memory and may sample data differently based on a configured sampler state and the texture / media format being read. Threads executing in the vector engines 1522A to 1522F, 1524A to 1524F, or matrix acceleration units 1523A to 1523F, 1525A to 1525D may use the cache / SLM 1528A to 1528F within each of the graphics cores 1521A to 1521F. The cache / SLM 1528A to 1528F may be configured as cache memory or as a pool of shared memory local to each of the respective graphics cores 1521A to 1521F.Ray tracing units 1527A through 1527F within graphics cores 1521A through 1521F include ray traversal / intersection circuitry for performing ray traversal using bounding volume hierarchies (BVHs) and for identifying intersections between rays and primitives contained within the BVH volumes. In one embodiment, ray tracing units 1527A through 1527F include circuitry for performing depth checking and suppression (e.g., using a depth buffer or similar arrangement). In one implementation, the ray tracing units 1527A to 1527F perform traversal and intersection operations consistent with image denoising, at least a portion of which may be performed using an associated matrix acceleration unit 1523A to 1523F, 1525A to 1525D.

[0200] Fig. 15C is a block diagram of a general-purpose graphics processing unit (GPGPU) 1570, which may be configured as a graphics processor, e.g., graphics processor 1508, and / or as a compute accelerator according to the embodiments described herein. GPGPU 1570 may be connected to host processors (e.g., one or more CPUs 1546) and memories 1571, 1572 via one or more system and / or memory buses. Memory 1571 may be system memory shared with the one or more CPUs 1546, while memory 1572 is device memory dedicated to GPGPU 1570. For example, components within GPGPU 1570 and memory 1572 may be assigned memory addresses accessible by the one or more CPUs 1546. Access to the memories 1571 and 1572 can be simplified via a memory control unit 1568.The memory controller 1568 may include an internal direct memory access (DMA) controller 1569 or may include logic to perform operations that would otherwise be performed by a DMA controller. In one embodiment, at least one of the one or more CPUs 1546 may include one or more accelerators 1545, including, but not limited to, neural network accelerators.

[0201] The GPGPU 1570 includes multiple global caches, including an L2 cache 1553, an L1 cache 1554, an instruction cache 1555, and a shared memory 1556, at least a portion of which may also be partitioned as a cache. The GPGPU 1570 also includes multiple compute units 1560A through 1560N. Each compute unit 1560A through 1560N includes a set of vector registers 1561, scalar registers 1562, vector logic units 1563, scalar logic units 1564, and a scheduler 1584. The compute units 1560A through 1560N may also include a shared local memory 1565 and a local cache 1566. The compute units 1560A to 1560N may be coupled to a constant cache 1567, which may be used to store constant data, which is the data that does not change during the execution of the kernel or shader program executing in the GPGPU 1570.The constant cache 1567 may be a cache for scalar data, and cached data may be retrieved directly from the scalar registers 1562. In one embodiment, the compute unit 1560A-1560N additionally includes at least one matrix unit 1580 for accelerating matrix, tensor, or artificial intelligence operations and at least one ray tracing unit (RT unit 1582) for accelerating ray tracing operations. The at least one matrix unit 1580 and the RT unit 1582 may include similar functionality to other matrix / tensor accelerators and ray tracing accelerators described herein.

[0202] During operation, the one or more CPUs 1546 may write instructions to registers or memory in the GPGPU 1570 that has been mapped to an accessible address space. The instruction processors 1557 may read the instructions from registers or memory and determine how to process those instructions in the GPGPU 1570. A thread dispatcher 1558 may then be used to dispatch threads to the compute units 1560A through 1560N to execute those instructions. Each compute unit 1560A through 1560N may execute threads independently of the other compute units. Additionally, each compute unit 1560A through 1560N may be independently configured for conditional computations, and the results of the computations may be conditionally output to memory. The instruction processors 1557 may interrupt the one or more CPUs 1546 when the submitted instructions have completed.

[0203] Fig. 16 is a block diagram of a graphics processor 1600, which may be a discrete graphics processing unit, or may be a graphics processor integrated with a plurality of processing cores or other semiconductor devices, such as, among others, memory devices or network interfaces. The elements of FIG. 1600 having the same or similar names as the elements of any other figure described herein describe the same elements as in the other figures, may operate or function in a similar manner, may include the same components, and may be associated with different entities than, but are not limited to, those described elsewhere herein. The graphics processor 1600 may be a variant of the graphics processor 1508 and may be used in place of the graphics processor 1508.The graphics processor may communicate with registers in the graphics processor and with instructions placed in the processor memory via an I / O interface associated with the memory. Memory interface 1600 may include a memory interface 1614 for accessing local memory, one or more internal caches, one or more shared external caches, and / or system memory.

[0204] The graphics processor 1600 may optionally also include a display controller 1602 to drive display output data to a display device 1618. The display controller 1602 includes hardware for one or more overlay layers for displaying and composing multiple layers of video or user interface elements. The display device 1618 may be an internal or external display device. In one embodiment, the display device 1618 is a head-mounted display device, such as a VR or AR display device. The graphics processor 1600 may include a video codec engine 1606 for encoding, decoding, or transcoding media to, from, or between one or more media coding formats, including, but not limited to, MPEG (Moving Picture Experts Group) formats such as MPEG-2, AVC (Advanced Video Coding) formats such as H.264 / MPEG-4 AVC, H.265 / HEVC, Alliance for Open Media (AOMedia) VP8, VP9 as well as the SMPTE-421M / VC-1 (SMPTE: Society of Motion Picture & Television Engineers) and JPEG (JPEG: Joint Photographic Experts Group) formats such as JPEG and Motion JPEG (MJPEG) formats.

[0205] The graphics processor 1600 may include a block image transfer engine (BLIT engine) 1603 for performing two-dimensional (2D) rasterization unit operations, including, for example, bit boundary block transfers, or 2D graphics operations may be performed using one or more components of the graphics processing engine (GPE) 1610. The GPE 1610 may include a 3D pipeline 1612 for performing 3D operations, such as rendering three-dimensional images and scenes using processing functions that manipulate 3D primitive shapes (e.g., a rectangle, a triangle, etc.). The 3D pipeline 1612 includes programmable and fixed function elements that perform numerous tasks within the element and / or create execution threads to a 3D / media subsystem 1615.Although the 3D pipeline 1612 may be used to perform media operations, the GPE 1610 may also include a media pipeline 1616 specifically used to perform media operations such as decoding, encoding, post-processing, and enhancing images and videos. The media pipeline 1616 may include fixed function logic units or programmable logic units to perform one or more specialized media operations, such as video decode acceleration, video line descrambling, and video encode acceleration, instead of or on behalf of the video codec engine 1606. The media pipeline 1616 may also include a thread creation unit to create threads for execution in the 3D / media subsystem 1615. The created threads perform computations for the media operations in one or more graphics execution units included in the 3D / media subsystem 1615.

[0206] Fig. 17A illustrates a graphics processor 1720 that is a variant of the graphics processor 1600 of the Fig. 16 and may be used instead of graphics processor 1600, and vice versa. Therefore, the disclosure of any features in combination with graphics processor 1600 herein also discloses, but is not limited to, a corresponding combination with graphics processor 1720. Graphics processor 1720, according to embodiments described herein, has a tiled architecture. Graphics processor 1720 may include a graphics processing engine cluster 1722 having multiple graphics processing engines within a plurality of graphics engine tiles. All graphics engine tiles 1710A-1710D may be interconnected via a set of tile interconnects 1723A-1723F. Each graphics engine tile 1710A-1710D may also be interconnected to a memory module or memory devices 1726A-1726D via memory interconnect 1725A-1725D.Memory devices 1726A-1726D may use any graphics memory technology. For example, memory devices 1726A-1726D may be graphics double data rate (GDDR) memory. Memory devices 1726A-1726D may be high-bandwidth memory modules (HBM modules) that may be co-located with their respective graphics engine tile 1710A-1710D. Memory devices 1726A-1726D may be stacked memory devices that may be stacked on top of their respective graphics processor tile 1710A-1710D. Each of the graphics engine tiles 1710A through 1710D and the associated memory devices 1726A through 1726D may be located on separate chiplets bonded to a base die or substrate, as shown in FIGS. Fig. 25A and Fig. 25B is described in more detail.

[0207] Graphics processor 1720 may be configured with a non-uniform memory access (NUMA) system in which memory devices 1726A through 1726D are coupled to associated graphics engine tiles 1710A through 1710D. A given memory device may be accessed by graphics engine tiles other than the tile to which it is directly connected. However, access latency to memory devices 1726A through 1726D may be lowest when accessing a local tile. In one embodiment, a cache coherent NUMA (ccNUMA) system is enabled using tile interconnects 1723A through 1723F to enable communication between cache controllers within graphics engine tiles 1710A through 1710D to maintain a consistent memory image when more than one cache stores in the same memory location.

[0208] The graphics processing engine cluster 1722 may be connected to an interconnect fabric 1724, which may be connected to an on-chip or on-package fabric. In one embodiment, the fabric interconnect 1724 includes a network processor, a network on a chip (NoC), or other switching processor to enable the fabric interconnect 1724 to function as a packet-switched fabric interconnect that switches data packets between components of the graphics processor 1720. The fabric interconnect 1724 may enable communication between graphics engine tiles 1710A-1710D and components such as the video codec engine 1706 and one or more copy engines 1704. The copy engines 1704 may be used to move data to, from, and between the storage devices 1726A through 1726D and memory external to the graphics processor 1720 (e.g.,the system memory). Fabric interconnect 1724 may also be used to connect graphics engine tiles 1710A through 1710D. Graphics processor 1720 may optionally include a display controller 1702 to enable connection to display device 1718. The graphics processor may also be configured as a graphics or compute accelerator. In the accelerator configuration, display controller 1702 and display device 1718 may be omitted.

[0209] The graphics processor 1720 may connect to a host system via a host interface 1728. The host interface 1728 may enable communication between the graphics processor 1720, system memory, and / or other system components. The host interface 1728 may be, for example, a PCI Express bus or another type of host system interface. The host interface 1728 may be, for example, an NVLink or NVSwitch interface. The host interface 1728 and the fabric interconnect 1724 may cooperate to allow multiple instances of the graphics processor 1720 to act as a single logical device. The cooperation between the host interface 1728 and the fabric interconnect 1724 may also result in the individual graphics engine tiles 1710A through 1710D being presented to the host system as different logical graphics devices.

[0210] Fig. 17B illustrates a compute accelerator 1730 according to embodiments described herein. The compute accelerator 1730 may have architectural similarities to the graphics processor 1720 of Fig. 17B and is optimized for compute acceleration. A compute engine cluster 1732 may include a set of compute engine tiles 1740A-1740D including execution logic optimized for parallel or vector-based general-purpose compute operations. In one embodiment, compute accelerator 1730 may be configured as an AI accelerator or an NPU. In such an embodiment, the execution logic of compute engine tiles 1740A-1740D may be primarily directed toward matrix or tensor operations and may include tensor cores or matrix engines described herein. Compute engine tiles 1740A-1740D may not include fixed-function graphics processing logic, although in some embodiments, one or more of compute engine tiles 1740A-1740D may include logic for performing media acceleration.Compute engine tiles 1740A-1740D may be connected to memory devices 1726A-1726D via memory interconnects 1725A-1725D. Memory devices 1726A-1726D and memory interconnects 1725A-1725D may be similar in technology to graphics processor 1720, or they may be different. Compute engine tiles 1740A-1740D may also be interconnected via a set of tile interconnects 1723A-1723F and may be connected to and / or interconnected by a fabric interconnect 1724. In one embodiment, compute accelerator 1730 includes a large L3 cache 1736, which may be configured as a device-wide cache. The compute accelerator 1730 may also be accessed via a host interface 1728 in a manner similar to the graphics processor 1720 of FIG. Fig. 17B can be connected to a host processor and a memory.

[0211] The compute accelerator 1730 may also include an integrated network interface 1742. In one embodiment, the integrated network interface 1742 includes a network processor and control logic that enables the compute engine cluster 1732 to communicate over a data link layer interconnect 1744 without requiring data to traverse a host system's memory. In one embodiment, one of the compute engine tiles 1740A-1740D is replaced with network processor logic, and the data to be transmitted or received over the data link layer interconnect 1744 may be transmitted directly to or from the memory 1726A-1726D. Multiple instances of the compute accelerator 1730 may be connected to a single logical device over the data link layer interconnect 1744.Alternatively, the various compute engine tiles 1740A through 1740D may also be presented as different network-accessible compute accelerator devices. Graphics processing resources

[0212] The Fig. 18A through 18C illustrate execution logic including an array of processing elements employed in a graphics processor according to the embodiments described herein. Fig. 18A illustrates a graphics core cluster according to one embodiment. Fig. 18B illustrates a vector engine of a graphics core according to one embodiment. Fig. Figure 18C illustrates a matrix engine of a graphics core according to one embodiment. The elements of the Fig. 18A to 18C, which have the same reference numerals as the elements of any other figure described herein, may be operated or function in any manner similar to, but not limited to, those described elsewhere herein. For example, the elements of Fig. 18A to 18C in the context of the graphics processor core block 1519 of Fig. 15B. In one embodiment, the elements of the Fig. 18A to 18C have similar functionality to equivalent components of the 1508 graphics processor from Fig. 15A or the GPGPU 1570 Fig. 15C.

[0213] As in Fig. 18A, in one embodiment, the graphics core cluster 1800 includes a graphics processor core block 1519, which may include any number of graphics cores (e.g., graphics core 1815A, graphics core 1815B, through graphics core 1815N). Multiple instances of the graphics core block 1519 may be included. In one embodiment, the elements of graphics cores 1815A through 1815N have similar or equivalent functionality to the elements of graphics cores 1521A through 1521F. Fig. 15B. In such an embodiment, the graphics cores 1815A-1815N each include circuitry including, among other things, vector engines 1802A-1802N, matrix engines 1803A-1803N, memory load / store units 1804A-1804N, instruction caches 1805A-1805N, data caches / shared local memory 1806A-1806N, ray tracing units 1808A-1808N, and samplers 1810A-1810N. The circuitry of the graphics cores 1815A-1815N may additionally include fixed function logic 1812A-1812N. The number of vector engines 1802A through 1802N and matrix engines 1803A through 1803N within the graphics cores 1815A through 1815N of a design may vary based on the workload, capability, and performance goals for the design.

[0214] With reference to the graphics core 1815A, the vector engine 1802A and the matrix engine 1803A are configurable to perform parallel computational operations on data in a variety of integer and floating-point data formats based on instructions associated with shader programs. Each vector engine 1802A and matrix engine 1803A can function as a programmable general-purpose compute unit capable of concurrently executing multiple hardware threads while processing multiple data elements in parallel for each thread. The vector engine 1802A and the matrix engine 1803A support processing of variable-width vectors with various SIMD widths, including, but not limited to, SIMD8, SIMD16, and SIMD32.The input data elements can be stored as a packed data type in a register, and the 1802A vector engine and 1803A matrix engine can process the different elements based on the data size of the elements. For example, when operating on a 256-bit vector, the bits of the vector are stored in a register, and the vector is processed as four separate packed 64-bit data elements (quad-word (QW) data elements), eight separate packed 32-bit data elements (double-word (DW) data elements), sixteen separate packed 16-bit data elements (word (W) data elements), or thirty-two separate 8-bit data elements (byte (B) data elements). However, other vector widths and register sizes are also possible. In one embodiment, the vector engine 1802A and the matrix engine 1803A are also configured to perform a SIMT operation on chains or thread groups of various sizes (e.g.,B. 8, 16 or 32 threads) can be configured.

[0215] The graphics core 1815A also services memory access requests from a memory load / store unit 1804A issued by the vector engine 1802A, the matrix engine 1803A, and / or other components of the graphics core 1815A that have access to memory. The memory access request may be processed by the memory load / store unit 1804A to load or store the requested data into or from the cache or memory into a register file associated with the vector engine 1802A and / or the matrix engine 1803A. The memory load / store unit 1804A may also perform prefetch operations. With further reference to Fig. 19, in one embodiment, memory load / store unit 1804A is configured to provide a SIMT scatter / gather or block prefetch for data stored in memory 1910 from memory local to other tiles, via tile interconnect 1908, or from system memory. The prefetching may be performed in a dedicated L1 cache (e.g., data cache / shared local memory 1806A), L2 cache 1904, or L3 cache 1906. In one embodiment, a prefetch to L3 cache 1906 automatically results in the data being stored in L2 cache 1904.

[0216] Instruction cache 1805A stores instructions to be executed by graphics core 1815A. In one embodiment, graphics core 1815A also includes instruction fetch and prefetch circuitry that fetches or prefetches instructions into instruction cache 1805A. Graphics core 1815A also includes instruction decode logic for decoding instructions within instruction cache 1805A. Data cache / shared local memory 1806A may be configured as a data cache managed by a cache controller implementing a cache replacement policy and / or configured as explicitly managed shared memory. Ray tracing unit 1808A includes circuitry for accelerating ray tracing operations. Sampling unit 1810A provides texture sampling for 3D operations and media sampling for media operations.Fixed function logic 1812A includes fixed function circuitry shared by the multiple instances of vector engine 1802A and matrix engine 1803A. Graphics cores 1815B through 1815N can operate in a manner similar to graphics core 1815A.

[0217] The functionalities of instruction caches 1805A through 1805N, data caches / shared local memory 1806A through 1806N, ray tracing units 1808A through 1808N, sampling units 1810A through 1810N, and fixed function logic 1812A through 1812N correspond to equivalent functionalities in the graphics processor architectures described herein. For example, instruction caches 1805A through 1805N can be configured in a manner similar to instruction cache 1855 of Fig. 15C. The data caches / shared local memory 1806A to 1806N, the ray tracing units 1808A to 1808N, and the sampling units 1810A to 1810N may be configured in a manner similar to the cache / SLM 1528A to 1528F, the ray tracing units 1527A to 1527F, and the sampling units 1526A to 1526F of Fig. 15B. The fixed function logic 1812A-1812N may include elements of the geometry pipeline / fixed function pipeline 1531 and / or the additional fixed function logic 1538 from Fig. 15B. In one embodiment, the ray tracing units 1808A to 1808N include circuitry for performing ray tracing acceleration operations performed by the ray tracing cores 372 of Fig. 3C can be carried out.

[0218] As in Fig. 18B, in one embodiment, the vector engine 1802 includes an instruction fetch unit 1837, a general purpose register file array (GRF) 1824, an architectural register file array (ARF) 1826, a thread arbitration unit 1822, a dispatch unit 1830, a branch unit 1832, a set of SIMD floating point units (FPUs) 1834, and, in one embodiment, a set of integer SIMD ALUs 1835. The GRF 1824 and the ARF 1826 comprise the set of general purpose register files and architectural register files associated with each hardware thread that may be active in the vector engine 1802. In one embodiment, per-thread architectural state is maintained in the ARF 1826, while data used during thread execution is stored in the GRF 1824. The execution state of each thread, including the instruction pointers for each thread, can be held in thread-specific registers in the ARF 1826.Register renaming can be used to dynamically assign registers to hardware threads.

[0219] In one embodiment, the vector engine 1802 has an architecture that is a combination of simultaneous multi-threading (SMT) and fine-grained interleaved multi-threading (IMT). The architecture has a modular configuration that can be fine-tuned at design time based on a target number of concurrent threads and the number of registers per graphics core, with the graphics core resources being divided among the logic for executing multiple concurrent threads. The number of logical threads that can be executed by the vector engine 1802 is not limited to the number of hardware threads, and each hardware thread can be assigned multiple logical threads.

[0220] In one embodiment, vector engine 1802 may issue multiple instructions concurrently, each of which may be a different instruction. Thread arbitration unit 1822 may forward the instructions to one of dispatch units 1830, branch units 1832, or SIMD FPUs 1834 for execution. Each execution thread may access 128 general-purpose registers within GRF 1824, where each register may store 32 bytes accessible as a variable-width vector of 32-bit data elements. In one embodiment, each thread has access to 4 KB within GRF 1824, although embodiments are not so limited, and other embodiments may provide more or fewer register resources.In one embodiment, the vector engine 1802 is divided into seven hardware threads that can perform computational operations independently, although the number of threads per vector engine 1802 can also vary depending on the embodiment. For example, in one embodiment, up to 16 hardware threads are supported. In an embodiment where seven threads can access 4 kilobytes, the GRF 1824 can store a total of 28 kilobytes. If 16 threads can access 4 kilobytes, the GRF 1824 can store a total of 64 kilobytes. Flexible addressing modes can allow registers to be addressed together to effectively build wider registers or to represent evenly spaced rectangular block data structures.

[0221] In one embodiment, memory operations, scan unit operations, and other longer-latency system communications are dispatched via "send" instructions executed by message-passing send unit 1830. In one embodiment, branch instructions are dispatched to a branch unit 1832 to facilitate SIMD divergence and eventual convergence.

[0222] In one embodiment, the SIMD FPUs 1834 of the vector engine 1802 perform floating-point operations. In one embodiment, the SIMD FPU(s) 1834 also support integer computation. In one embodiment, the SIMD FPUs 1834 can perform up to M 32-bit floating-point (or integer) operations or up to 2M 16-bit integer or 16-bit floating-point operations. In one embodiment, at least one of the FPUs provides extended mathematical capabilities to support high-throughput transcendental mathematical functions and 64-bit double-precision floating-point operations. In some embodiments, SIMD ALUs 1835 are configured to perform 8-bit integer operations, which may be specifically optimized to perform operations associated with machine learning computations.In one embodiment, the SIMD ALUs 1835 are replaced by the SIMD FPUs 1834, which are configurable to perform integer and floating-point operations. In one embodiment, the SIMD FPUs 1834 and the SIMD ALUs 1835 are configurable to execute SIMT programs. In one embodiment, combined SIMD+SIMT operation is supported.

[0223] In one embodiment, arrays of multiple instances of the vector engine 1802 may be instantiated within a graphics core. For scalability, product architects may choose the exact number of vector engines per graphics core grouping. In one embodiment, the vector engine 1802 may execute instructions using a plurality of execution channels. In another embodiment, each thread executed by the vector engine 1802 executes on a different channel.

[0224] As in Fig. 18C, in one embodiment, the matrix engine 1803 includes an array of processing elements configured to perform tensor operations, including vector-matrix and matrix-matrix operations, such as, among others, matrix multiplication and / or dot product operations. The matrix engine 1803 is configured with M rows and N columns of processing elements (1852AA through 1852MN) that include multiplier and adder circuits organized in a pipelined manner.In one embodiment, processing elements 1852AA through 1852MN form the physical pipeline stages of an N-wide and M-deep systolic array that can be used to perform vector / matrix or matrix / matrix operations in a data-parallel manner, including matrix multiplication, fused multiply-add, dot product multiplication, or other general-purpose matrix-matrix multiplication (GEMM) operations. In one embodiment, matrix engine 1803 supports 16-bit and 8-bit floating-point operations, as well as 8-bit, 4-bit, 2-bit, and binary integer operations. Matrix engine 1803 can also be configured to accelerate specific machine learning operations.In such embodiments, the matrix engine 1803 may be configured with support for the 16-bit floating point format of a bfloat (Brain floating point) or a 32-bit floating point format for tensor floating point (TF32), which have different numbers of mantissa and exponent bits compared to the IEEE 754 formats of the Institute of Electrical and Electronics Engineers.

[0225] In one embodiment, each stage may add the result of the operations performed in that stage to the output of the previous stage during each cycle. In other embodiments, the pattern of data movement between processing elements 1852AA through 1852MN after a set of compute cycles may vary based on the instruction or macro operation being executed. For example, in one embodiment, partial sum loopback is enabled, and the processing elements may instead add the output of a current cycle to an output generated in the previous cycle. In one embodiment, the last stage of the systolic array may be configured with a loopback to the beginning stage of the systolic array. In such an embodiment, the number of physical pipeline stages may be decoupled from the number of logical pipeline stages supported by matrix engine 1803.For example, if processing elements 1852AA through 1852MN are configured as a systolic array of M physical stages, a loopback from stage M to the initial pipeline stage may enable processing elements 1852AA through 1852MN to operate as a systolic array of, for example, 2M, 3M, 4M, etc., logical pipeline stages.

[0226] In one embodiment, matrix engine 1803 includes memory 1841A to 1841N, 1842A to 1842M for storing input data in the form of row and column data for input matrices. Memory 1842A to 1842M is configurable to store row elements (A0 to Am) of a first input matrix, and memory 1841A to 1841N is configurable to store column elements (B0 to Bn) of a second input matrix. The row and column elements are provided as input to processing elements 1852AA to 1852MN for processing. In one embodiment, row and column elements of the input matrices may be stored in a systolic register file 1840 within the matrix engine 1803 before providing these elements to the memory 1841A through 1841N, 1842A through 1842M.In one embodiment, the systolic register file 1840 is excluded, and the memory 1841A through 1841N, 1842A through 1842M is generated from registers in an associated vector engine (e.g., the GRF 1824 of the vector engine 1802 of FIG. Fig. 18B) or another memory of the graphics core that includes the matrix engine 1803 (e.g., a data cache / shared local memory 1806A for the matrix engine 1803A of Fig. 18A). The results generated by processing elements 1852AA through 1852MN are then output to an output buffer and / or written to a register file (e.g., systolic register file 1840, GRF 1824, data cache / shared local memory 1806A-1806N) for further processing by other functional units of the graphics processor or for output to memory.

[0227] In some embodiments, the matrix engine 1803 is configured with support for input underpopulation, where multiplication operations for sparse regions of input data can be bypassed by skipping multiplication operations with a zero-valued operand. In one embodiment, the processing elements 1852AA through 1852MN are configured to skip performing certain operations that have a zero input. In one embodiment, underpopulation within input matrices can be detected, and operations that have known zero output values ​​can be bypassed before being submitted to the processing elements 1852AA through 1852MN. Loading zero-valued operands into the processing elements can be bypassed, and the processing elements 1852AA through 1852MN can be configured to perform multiplications on the input elements with non-zero values.Matrix engine 1803 may also be configured with support for output understaffing, such that operations with results predetermined to be zero are bypassed. For input understaffing and / or output understaffing, in one embodiment, metadata is provided to processing elements 1852AA through 1852MN to indicate, for a processing cycle, which processing elements and / or data channels should be active during that cycle.

[0228] In one embodiment, the matrix engine 1803 includes hardware to enable operations on sparse data comprising a compressed representation of a sparse matrix storing non-zero values ​​and metadata identifying the positions of the non-zero values ​​within the matrix. Example compressed representations include, but are not limited to, compressed tensor representations, such as compressed sparse row (CSR), compressed sparse column (CSC), and compressed sparse fiber (CSF) representations. Support for compressed representations enables operations to be performed on inputs in a compressed tensor format without requiring decompression or decoding of the compressed representation.In such an embodiment, operations may be performed only on non-zero input values, and the resulting non-zero output values ​​may be allocated in an output matrix. In some embodiments, hardware support is also provided for machine-specific lossless data compression formats used when transferring data within hardware or across system buses. This data may be maintained in a compressed format for sparse input data, and the matrix engine 1803 may use the compression metadata for the compressed data to enable operations performed only on non-zero values ​​or to allow blocks of zero data inputs to be skipped for multiplication operations.

[0229] In various embodiments, the input data may be provided by a programmer in a compressed tensor representation, or a codec may compress the input data into the compressed tensor representation or another sparse data encoding. In addition to supporting compressed tensor representations, streaming compression of sparse input data may be performed before the data is provided to processing elements 1852AA through 1852MN. In one embodiment, compression is performed on data written to a cache associated with graphics core cluster 1800, where the compression is performed using encoding supported by matrix engine 1803. In one embodiment, matrix engine 1803 supports inputs with structured underpopulation, where a predetermined degree or pattern of underpopulation is imposed on the input data.This data may be compressed to a known compression ratio, with the compressed data being processed by the processing elements 1852AA through 1852MN according to the metadata associated with the compressed data.

[0230] Fig. 19 illustrates a tile 1900 of a multi-tile processor according to one embodiment. In one embodiment, tile 1900 represents one of the graphics engine tiles 1710A through 1710D of Fig. 17A or the compute engine tiles 1740A to 1740D Fig. 17B. The multi-tile graphics processor tile 1900 includes an array of graphics core clusters (e.g., graphics core cluster 1800A, graphics core cluster 1800B, through graphics core cluster 1800N), each graphics core cluster having an array of graphics cores 515A through 515N. The tile 1900 also includes a global arbiter 1902 that allocates threads to the processing resources of the tile 1900.

[0231] Tile 1900 may include or be coupled to an L3 cache 1906 and a memory 1910. In various embodiments, L3 cache 1906 may be excluded, or tile 1900 may include additional cache levels, such as an L4 cache. In one embodiment, each instance of tile 1900 in the multi-tile graphics processor is associated with a memory 1910, such as in Fig. 17A and Fig. 17B. In one embodiment, a multi-tile processor may be configured as a multi-chip module in which the L3 cache 1906 and / or the memory 1910 are packaged on separate chiplets, with the graphics core clusters 1800A-1800N being packaged on other chiplets. In this context, a chiplet is an at least partially packaged integrated circuit that includes different logic units that can be assembled with other chiplets in a larger package. For example, the L3 cache 1906 may be included in a dedicated cache chiplet or may be on the same chiplet as the graphics core clusters 1800A-1800N. In one embodiment, the L3 cache 1906 may be included in an active base die or an active interposer.

[0232] A memory fabric 1903 enables communication between the graphics core clusters 1800A to 1800N, the L3 cache 1906, and the memory 1910. An L2 cache 1904 is coupled to the memory fabric 1903 and is configurable to cache transactions performed via the memory fabric 1903. A tile interconnect 1908 enables communication with other tiles on the graphics processors and may be one of the tile interconnects 1723A to 1723F. Fig. 17A and Fig. 17B. In embodiments where L3 cache 1906 is excluded from tile 1900, L2 cache 1904 may be configured as a combined L2 / L3 cache. Memory fabric 1903 may be configured to forward data to L3 cache 1906 or to memory controllers associated with memory 1910 based on a presence or absence of L3 cache 1906 in a specific implementation. L3 cache 1906 may be configured as a per-tile cache dedicated to processing resources of tile 1900, or it may be a partition of a GPU-wide L3 cache.

[0233] Fig. 20 is a block diagram illustrating graphics processor instruction formats 2000. The graphics processor execution units support an instruction set with instructions in multiple formats. The solid-line boxes represent the components generally included in an execution unit instruction, while the dashed-line boxes contain components that are optional or included only in a subset of the instructions. In some embodiments, the graphics processor instruction formats 2000 described and illustrated are macro-instructions because they are instructions that are fed to the execution unit, as opposed to micro-operations that result from instruction decode once the instruction has been processed. Thus, a single instruction can cause the hardware to perform multiple micro-operations.

[0234] The graphics processor execution units described herein may natively support instructions in a 128-bit instruction format 2010. A compacted 64-bit instruction format 2030 is available for some instructions based on the selected instruction, instruction options, and number of operands. The native 128-bit instruction format 2010 provides access to all instruction options, while some options and operations are restricted in the 64-bit instruction format 2030. The native instructions available in the 64-bit instruction format 2030 vary depending on the embodiment. The instruction is partially compacted using a set of index values ​​in an index field 2013.The execution unit hardware refers to a set of compaction tables based on the index values ​​and uses the compaction table outputs to reconstruct a native instruction in the 128-bit 2010 instruction format. Other instruction sizes and formats can be used.

[0235] For each format, an instruction opcode 2012 defines the operation to be performed by the execution unit. The execution units execute each instruction in parallel across the multiple data elements of each operand. For example, in response to an add instruction, the execution unit performs a concurrent add operation across each color channel representing a texture element or a picture element. By default, the execution unit executes each instruction across all data channels of the operands. The instruction control field 2014 can provide control over certain execution options, such as channel selection (e.g., a predication) and data channel ordering (e.g., reordering). For instructions in the 128-bit instruction format 2010, an execution size field 2016 limits the number of data channels executed in parallel.A 2016 execution size field need not be available for use in the 2030 compact 64-bit instruction format.

[0236] Some execution unit instructions have up to three operands, including two source operands, src0 2020, src1 2022, and one destination operand (dest 2018). Other instructions, such as data manipulation instructions, dot product instructions, multiply-add instructions, or multiply-accumulate instructions, may have a third source operand (e.g., SRC2 2024). The instruction opcode 2012 determines the number of source operands. The last source operand of an instruction can be an immediate value (e.g., a hard-coded value) passed with the instruction. The execution units can also support multiple destination instructions, where one or more destinations are implied or implicit based on the instruction and / or the specified destination.

[0237] The 128-bit instruction format 2010 may include an access / address mode field 2026, which specifies, for example, whether direct register addressing mode or indirect register addressing mode is used. When direct register addressing mode is used, the register address of one or more operands is provided directly by bits in the instruction.

[0238] The 128-bit instruction format 2010 may also include an access / address mode field 2026, which specifies an address mode and / or an access mode for the instruction. The access mode may be used to define a data access alignment for the instruction. Access modes including an aligned 16-bit access mode and an aligned 1-byte access mode may be supported, with the byte alignment of the access mode determining the access alignment of the instruction operands. For example, if the instruction is in a first mode, it may use byte-aligned addressing for source and destination operands, and if it is in a second mode, the instruction may use aligned 16-byte addressing for all source and destination operands.

[0239] The address mode portion of the access / address mode field 2026 can determine whether the instruction should use direct or indirect addressing. When a direct register addressing mode is used, bits in the instruction directly provide the register address of one or more operands. When an indirect register addressing mode is used, the register address of one or more operands can be calculated based on an address register value and a direct address field in the instruction.

[0240] Instructions may be grouped based on the bit fields of the instruction opcode 2012 to simplify opcode decoding 2040. For an 8-bit opcode, bits 4, 5, and 6 allow the execution unit to determine the type of opcode. The exact opcode grouping shown is merely an example. A move-and-logic opcode group 2042 may include data movement and logic instructions (e.g., move (mov), compare (cmp)). The move-and-logic opcode group 2042 may share the five least significant bits (LSB), with move (mov) instructions being in the form 0000xxxxb and logic instructions being in the form 0001xxxxb. A flow control instruction group 2044 (e.g., call, jump (jmp)) contains instructions in the form 0010xxxxb (e.g., 0x20). A miscellaneous instruction group 2046 contains a mix of instructions, including synchronization instructions (e.g., wait, send) in the form 0011xxxxb (e.g., 0x30).A parallel math instruction group 2048 contains component-wise arithmetic instructions (e.g., add, multiply) in the form 0100xxxxb (e.g., Ox40). The parallel math instruction group 2048 performs the arithmetic operations in parallel across data channels. The vector math group 2050 includes arithmetic instructions (e.g., dp4) in the form 0101xxxxb (e.g., Ox50). The vector math group performs arithmetic such as dot product calculations on vector operands. The illustrated opcode decoding 2040 may be used in one embodiment to determine which portion of an execution unit is used to execute a decoded instruction. For example, some instructions may be referred to as systolic instructions, which are executed from a systolic array.Other instructions, such as ray tracing instructions (not shown), may be passed to a ray tracing kernel or ray tracing logic within a section or partition of the execution logic. Graphics pipeline

[0241] Fig. 21 is a block diagram of a graphics processor 2100 according to another embodiment. The elements of Fig. 21 that have the same or similar names as the elements of any other figure described herein, describe the same elements as in the other figures, may operate or function in a similar manner, may include the same components, and may be linked to other entities such as, but are not limited to, those described elsewhere herein.

[0242] The graphics processor 2100 may include various types of graphics processing pipelines, such as a geometry pipeline 2120, a media pipeline 2130, a display engine 2140, thread execution logic 2150, and a rendering output pipeline 2170. The graphics processor 2100 may be a graphics processor within a multi-core processing system that includes one or more general-purpose processing cores. The graphics processor may be controlled by register writes to one or more control registers (not shown) or by commands issued to the graphics processor 2100 via a ring or network interconnect 2102. The ring or network interconnect 2102 may couple the graphics processor 2100 to other processing components, such as other graphics processors or general-purpose processors.The commands from the ring or network interconnect 2102 are interpreted by a command streamer 2103, which delivers commands to individual components of the geometry pipeline 2120 or the media pipeline 2130.

[0243] The command streamer 2103 may direct the operation of a vertex fetcher 2105, which reads vertex data from memory and executes vertex processing instructions provided by the command streamer 2103. The vertex fetcher 2105 may provide the vertex data to a vertex shader 2107, which performs a coordinate space transformation and lighting operations for each vertex. The vertex fetcher 2105 and the vertex shader 2107 may execute vertex processing instructions by dispatching the execution threads to the graphics cores 2152A-2152B via a thread dispatcher 2131.

[0244] Graphics cores 2152A through 2152B may be an array of vector processors with an instruction set for performing graphics and media operations. Graphics cores 2152A through 2152B may have an attached L1 cache 2151 that is specific to each array or shared among the arrays. The cache may be configured as a data cache, an instruction cache, or a single cache partitioned to contain data and instructions in different partitions.

[0245] A geometry pipeline 2120 may include tessellation components to perform hardware-accelerated tessellation of 3D objects. A programmable hull shader 2111 may configure the tessellation operations. A programmable domain shader 2117 may provide back-end evaluation of the tessellation output. A tessellator 2113 may operate under the direction of a programmable hull shader 2111 and may include dedicated logic for generating a set of detailed geometric objects based on a coarse geometric model provided as input to the geometry pipeline 2120. Furthermore, if tessellation is not used, tessellation components (e.g., the hull shader 2111, the tessellator 2113, and the domain shader 2117) may be bypassed.The tessellation components may be operated based on data received from the vertex shader 2107.

[0246] Entire geometric objects may be processed by a geometry shader 2119 via one or more threads allocated to graphics cores 2152A through 2152B, or they may proceed directly to clipper 2129. The geometry shader may operate on entire geometric objects, rather than on vertices or faces of vertices as in previous stages of the graphics pipeline. When tessellation is disabled, geometry shader 2119 receives input from vertex shader 2107. Geometry shader 2119 may be programmable by a geometry shader program to perform geometry tessellation when the tessellation units are disabled.

[0247] Before rasterization, a clipper 2129 processes the vertex data. The clipper 2129 can be a fixed-function clipper or a programmable clipper with clipping and geometry shader functions. A rasterization unit and depth inspection component 2173 can allocate pixel shaders in the rendering output pipeline 2170 to convert the geometric objects into pixel-wise representations. The pixel shader logic can be included in the thread execution logic 2150. Optionally, an application can bypass the rasterization unit and depth inspection components 2173 and access unrasterized vertex data via a stream output unit 2123.

[0248] The graphics processor 2100 includes an interconnect bus, an interconnect fabric, or other interconnect mechanism that allows data and messages to be passed between the main components of the processor. In some embodiments, the graphics cores 2152A-2152B and associated logic units (e.g., the L1 cache 2151, the sampling unit 2154, the texture cache 2158, etc.) are interconnected via a data port 2156 to perform memory access and communicate with components of the processor's rendering output pipeline. A sampling unit 2154, the L1 cache 2151, the texture cache 2158, and the graphics cores 2152A-2152B may each have separate memory access paths. The texture cache 2158 may also optionally be configured as a sampling unit cache.

[0249] The rendering output pipeline 2170 may include a rasterization unit and depth inspection component 2173 that converts vertex-based objects into an associated pixel-based representation. The rasterization unit logic may include a windowing / masking unit to perform fixed-function triangle and line rasterization. An associated rendering cache 2178 and a depth cache 2179 are also available in some embodiments. A pixel operations component 2177 performs pixel-based operations on the data, although in some cases, pixel operations associated with 2D operations (e.g., bit-block image transfers with blending) are performed by the 2D engine 2141 or replaced at display time by the display controller 2143 using overlay display planes.A shared L3 cache 2175 can be available to all graphics components, allowing data sharing without the use of main system memory.

[0250] The media pipeline 2130 may include a media engine 2137 and a video frontend 2134. The video frontend 2134 may receive pipeline commands from the command streamer 2103. The media pipeline 2130 may include a separate command streamer. The video frontend 2134 may process media commands before sending the command to the media engine 2137. The media engine 2137 may include thread creation functionality to create threads for dispatch to the thread execution logic 2150 via the thread dispatcher 2131.

[0251] The graphics processor 2100 may include a display engine 2140. This display engine 2140 may be external to the processor 2100 and may be coupled to the graphics processor via the ring or network interconnect 2102 or any other interconnect bus or fabric. The display engine 2140 may include a 2D engine 2141 and a display controller 2143. The display engine 2140 may include dedicated logic capable of operating independently of the 3D pipeline. The display controller 2143 may be coupled to a display device (not shown), which may be a system-integrated display device, such as in a laptop computer, or an external display device connected via a display device connector.

[0252] The geometry pipeline 2120 and the media pipeline 2130 may be configurable to perform operations based on multiple graphics and media programming interfaces, but they are not specific to any one application programming interface (API). Driver software for the graphics processor may translate API calls specific to a particular graphics or media library into commands that can be processed by the graphics processor. Support for the Open Graphics Library (OpenGL), Open Computing Language (OpenCL), and / or Vulkan graphics and compute API, all from the Khronos group, may be provided. Support may also be provided for the Direct3D library from Microsoft Corporation. A combination of these libraries may be supported. Support for the open source Computer Vision Library (OpenCV) may also be provided.A future API with a compatible 3D pipeline would also be supported if a mapping can be performed from the pipeline of the future API to the pipeline of the graphics processor. Graphics pipeline programming

[0253] Fig. 22A is a block diagram illustrating an instruction format of a graphics processor 2200 used to program graphics processing pipelines, such as those described herein in connection with Fig. 16 and Fig. 21 described pipelines. Fig. Figure 22B is a block diagram illustrating a graphics processor command sequence 2210 according to one embodiment. The solid-line boxes in Fig. 22A represent the components generally included in a graphics instruction, while the dashed boxes contain components that are optional or that are only included in a subset of the graphics instructions. The graphics processor instruction format 2200 from Fig. Figure 22A contains fields for identifying a client 2202, an instruction operation code (opcode) 2204, and a data field 2206 for the instruction. A partial opcode 2205 and an instruction size 2208 are also included in some instructions.

[0254] The client 2202 may specify the client unit of the graphics device that will process the command data. A command parser of a graphics processor may examine the client field of each command to condition further processing of the command and forward the command data to the appropriate client unit. The client units of the graphics processor may include a memory interface unit, a rendering unit, a 2D unit, a 3D unit, and a media unit. Each client unit may have a corresponding processing pipeline that processes the commands. After the command is received by the client unit, the client unit reads the opcode 2204 and, if present, the partial opcode 2205 to determine the operation to be performed. The client unit executes the command using information in the data field 2206.For some instructions, an instruction size of 2208 is expected to explicitly specify the instruction size. The instruction parser can automatically determine the size of at least some of the instructions based on the instruction opcode. Instructions can be aligned across multiples of a double word. Other instruction formats can also be used.

[0255] The flowchart in Fig. Figure 22B illustrates a graphics processor command sequence 2210. Software or firmware of a data processing system highlighting an example graphics processor may use a version of the illustrated command sequence to set up, execute, and complete a set of graphics operations. A sample command sequence is shown and described for example purposes only, but is not limited to these specific commands or to this command sequence. Furthermore, the commands may be issued as a batch of commands in a command sequence such that the graphics processor processes the command sequence at least partially concurrently.

[0256] The graphics processor 2210 instruction sequence may begin with a pipeline flush instruction 2212 to cause all active graphics pipelines to complete the currently pending instructions for the pipeline. Optionally, the 3D pipeline 2222 and the media pipeline 2224 may not operate concurrently. Pipeline flushing is performed to cause the active graphics pipeline to complete all pending instructions. In response to a pipeline flush, the instruction parser for the graphics processor suspends instruction processing until the active drawing engines complete all pending operations and the relevant read caches have been invalidated. Optionally, any data in the rendering cache marked as "dirty" may be flushed to memory. The pipeline flush instruction 2212 may be used for pipeline synchronization or before placing the graphics processor into a low-power state.

[0257] A pipeline select instruction 2213 may be used when an instruction sequence requires the graphics processor to explicitly switch between pipelines. A pipeline select instruction 2213 may be required only once within an execution context before issuing pipeline instructions, unless the context needs to issue instructions for both pipelines. A pipeline flush instruction 2212 may be required immediately before a pipeline switch via the pipeline select instruction 2213.

[0258] A pipeline control instruction 2214 can configure a graphics pipeline for operation and can be used to program the 3D pipeline 2222 and the media pipeline 2224. The pipeline control instruction 2214 can configure the pipeline state for the active pipeline. The pipeline control instruction 2214 can be used for pipeline synchronization and to flush data from one or more caches in the active pipeline before processing an instruction stack.

[0259] Instructions relating to return buffer state 2216 can be used to configure a set of return buffers for the respective pipelines to write data to. Some pipeline operations require the allocation, selection, or configuration of one or more return buffers into which the operations immediately write intermediate data during processing. The graphics processor may also use one or more return buffers to store output data and perform cross-thread communication. Return buffer state 2216 may involve selecting the size and number of return buffers to use for a set of pipeline operations.

[0260] The remaining instructions in the instruction sequence differ based on the active pipeline for operations. Based on a pipeline determination 2220, the instruction sequence is tailored to the 3D pipeline 2222, beginning with the 3D pipeline state 2230, or the media pipeline 2224, beginning with the media pipeline state 2240.

[0261] The 3D pipeline state configuration commands 2230 include 3D state setting commands for vertex buffer state, vertex element state, constant color state, depth buffer state, and other state variables that must be configured before the 3D primitive commands are processed. The values ​​of these commands are determined based at least in part on the specific 3D API being used. The 3D pipeline state commands 2230 may also be capable of selectively disabling or bypassing certain pipeline elements when those elements are not in use.

[0262] A 3D primitive command 2232 may be used to submit 3D primitives to be processed by the 3D pipeline. The commands and associated parameters passed to the graphics processor via the 3D primitive command 2232 are passed to the vertex fetch function in the graphics pipeline. The vertex fetch function uses the data from the 3D primitive command 2232 to create vertex data structures. The vertex data structures are stored in one or more return buffers. The 3D primitive command 2232 may be used to perform vertex operations on 3D primitives via the vertex shaders. To process vertex shaders, the 3D pipeline 2222 dispatches shader execution threads to the graphics processor execution units.

[0263] The 3D pipeline 2222 can be triggered via an execute instruction 2234 or an execute event. A register can write trigger instruction executions. Execution can be triggered via a "Go" or "Kick" instruction in the instruction sequence. Instruction execution can be triggered using a pipeline synchronization instruction to flush the instruction sequence from the graphics pipeline. The 3D pipeline will perform geometry processing on the 3D primitives. Once the operations are complete, the resulting geometric objects are rasterized, and the pixel engine colors the resulting pixels. Additional instructions to control pixel shading and pixel backend operations can also be included for those operations.

[0264] The graphics processor instruction sequence 2210 may follow the path of the media pipeline 2224 when performing media operations. In general, the specific use and programming style for the media pipeline 2224 depend on the media or computational operations to be performed. Specific media decoding operations may be offloaded to the media pipeline during media decoding. The media pipeline may also be bypassed, and media decoding may be performed entirely or partially using resources provided by one or more general-purpose processing cores.The media pipeline may also include elements for general-purpose graphics processor unit (GPGPU) operations, where the graphics processor is used to perform SIMD vector operations using computational shader programs that are not explicitly related to the rendering of graphics primitives.

[0265] The media pipeline 2224 may be configured in a similar manner to the 3D pipeline 2222. A set of instructions to configure the media pipeline state 2240 is dispatched or queued in a command queue before the media object instructions 2242. Instructions for the media pipeline state 2240 may include data to configure the media pipeline elements used to process the media objects. This includes data to configure the video decoding and encoding logic in the media pipeline, such as an encoding or decoding format. The instructions for the media pipeline state 2240 may also support the use of one or more pointers to "indirect" state elements that contain a stack of state settings.

[0266] The media object instructions 2242 may provide pointers to media objects for processing by the media pipeline. The media objects include memory buffers containing video data to be processed. Optionally, all media pipeline states must be valid before a media object instruction 2242 is issued. Once the pipeline state is configured and the media object instructions 2242 are queued, the media pipeline 2224 is triggered via an execute instruction 2244 or an equivalent execution event (e.g., a register write). The output from the media pipeline 2224 may then be post-processed by operations provided by the 3D pipeline 2222 or the media pipeline 2224. The GPGPU operations may be configured and executed in a similar manner to media operations. Graphics software architecture

[0267] Fig. 23 illustrates an exemplary graphics software architecture for a data processing system 2300. Such a software architecture may include a 3D graphics application 2310, an operating system 2320, and a processor 2330. The processor 2330 may include a graphics processor 2332 and one or more general-purpose processor cores 2334. The processor 2330 may be a variant of the one or more processors 1402 or any other processors described herein and may be used instead. The disclosure of any features in combination with the one or more processors 1402 or any other processors described herein therefore also discloses a corresponding combination with the graphics processor 2332, but is not limited thereto. Furthermore, the elements of Fig. 23, which have the same or similar names as the elements of any other figures described herein, the same elements as in the other figures may operate or function in a similar manner, may include the same components, and may be associated with other entities such as, but not limited to, those described elsewhere herein. The 3D graphics application 2310 and the operating system 2320 each execute in the system memory 2350 of the data processing system.

[0268] The 3D graphics application 2310 may include one or more shader programs including shader instructions 2312. The shader language instructions may be in a high-level shader language, such as Direct3D's High-Level Shader Language (HLSL), OpenGL Shader Language (GLSL), and so on. The application may also include executable instructions 2314 in a machine language suitable for execution by the general-purpose processor core 2334. The application may also include graphics objects 2316 defined by vertex data.

[0269] The operating system 2320 may be a Microsoft® Windows® operating system from Microsoft Corporation, a proprietary UNIX-like operating system, or an open-source UNIX-like operating system using a variant of the Linux kernel. The operating system 2320 may support a graphics API 2322, such as the Direct3D API, the OpenGL API, or the Vulkan API. When the Direct3D API is employed, the operating system 2320 uses a front-end shader compiler 2324 to compile arbitrary shader instructions 2312 in HLSL into a low-level shader language. The compilation may be just-in-time (JIT) compilation, or the application may perform shader precompilation. Higher-level shaders may be compiled as lower-level shaders during compilation of the 3D graphics application 2310.The shading instructions 2312 may be provided in an intermediate form, such as a version of the Standard Portable Intermediate Representation (SPIR) used by the Vulkan API.

[0270] A user-mode graphics driver 2326 may include a backend shader compiler 2327 to convert the shader instructions 2312 into a hardware-specific representation. When the OpenGL API is used, the shader instructions 2312 are passed in the high-level GLSL language to a user-mode graphics driver 2326 for compilation. The user-mode graphics driver 2326 may use operating system kernel-mode functions 2328 to communicate with a kernel-mode graphics driver 2329. The kernel-mode graphics driver 2329 may communicate with the graphics processor 2332 to dispatch commands and instructions. IP core implementations

[0271] One or more aspects may be implemented by representative code stored on a machine-readable medium that represents and / or defines logic in an integrated circuit, such as a processor. For example, the machine-readable medium may include instructions that represent numerous logics within the processor. When read by a machine, the instructions may cause the machine to fabricate the logics to perform the techniques described herein. Such representations, known as "IP cores," are reusable logic units for an integrated circuit that are stored on a tangible machine-readable medium as a hardware model that describes the structure of the integrated circuit.The hardware model can be delivered to various customers or manufacturing facilities, which load the hardware model into manufacturing machines that manufacture the integrated circuit. The integrated circuit can be manufactured such that the circuit performs operations described in connection with any of the embodiments described herein.

[0272] Fig. 24 is a block diagram illustrating an IP core development system 2400 according to an embodiment that can be used to manufacture an integrated circuit to perform the operations according to an embodiment. The IP core development system 2400 can be used to create modular, reusable designs that can be integrated into a larger design or used to build an entire integrated circuit (e.g., an SoC integrated circuit). A design facility 2430 can create a software simulation 2410 of an IP core design in a high-level programming language (e.g., C / C++). The software simulation 2410 can be used to design, test, and verify the behavior of the IP core using a simulation model 2412. The simulation model 2412 can include functional, behavioral, and / or timing simulations.A register transfer level design (RTL design) 2415 can then be generated or synthesized from the simulation model 2412. The RTL design 2415 is an abstraction of the integrated circuit's behavior that models the flow of digital signals between the hardware registers, including the associated logic performed using the modeled digital signals. In addition to the RTL design 2415, lower-level designs at the logic level or the transistor level can also be generated, designed, or synthesized. Thus, the specific details of the initial design and simulation can vary.

[0273] The RTL design 2415 or an equivalent thereof may be further synthesized by the design facility into a hardware model 2420, which may be in a hardware description language (HDL) or other representation of physical design data. The HDL may be further simulated or tested to verify the IP core design. The IP core design may be stored for delivery to a third-party manufacturing facility 2465 using non-volatile memory 2440 (e.g., hard disk, flash memory, or any non-volatile storage medium). The manufacturing facility 2465 may be a third-party manufacturing facility. Alternatively, the IP core design may be transmitted (e.g., over the Internet) via a wired connection 2450 or wireless connection 2460.The manufacturing facility 2465 may then fabricate an integrated circuit based at least in part on the IP core design. The fabricated integrated circuit may be configured to perform operations in accordance with at least one embodiment described herein.

[0274] Fig. 25A illustrates a cross-sectional side view of a package assembly 2590 for an integrated circuit comprising multiple units of hardware logic chiplets coupled to a substrate 2580 (e.g., a base die). A graphics processing unit, a parallel processor, and / or a compute accelerator, as described herein, may be assembled from diverse silicon chiplets that are separately fabricated. In this context, a chiplet is an at least partially packaged integrated circuit that includes different logic units that can be assembled with other chiplets in a larger package. A diverse set of chiplets with different IP core logic may be assembled into a single device. Furthermore, the chiplets may be integrated into a base die or a base chiplet using active interposer technology.The concepts described here enable the interconnection and communication between the various forms of IP within the GPU. IP cores can be manufactured using different process technologies and assembled during manufacturing, avoiding the complexity of converging multiple IPs onto the same manufacturing process, especially for a large SoC with multiple IPs in different forms. The ability to use multiple process technologies improves time to market and provides a cost-effective way to create multiple product SKUs. Furthermore, the disaggregated IPs can be more easily powered independently, and components not used for a given workload can be turned off, reducing overall power consumption.

[0275] In numerous embodiments, a package assembly 2590 may include a smaller or larger number of components and chiplets interconnected by an interconnect fabric 2585 or a bridge structure 2587. The bridge structures 2587 may be used to facilitate a point-to-point interconnect between, for example, logic or I / O chiplets 2574 and memory chiplets 2575. In some implementations, the bridge structures 2587 may also be embedded in the substrate 2580.The chiplets within the package assembly 2590 may have a 2.5D arrangement using chip-on-wafer-on-substrate (CoWoS) stacking, in which multiple dies are stacked side by side on a silicon interposer that includes through-silicon vias (TSVs) to couple the chiplets to the substrate 2580, which includes electrical connections to the package interconnect 2583.

[0276] In one embodiment, the silicon interposer is an active interposer 2589 that includes embedded logic in addition to the TSVs. In such an embodiment, the chiplets are arranged within the package assembly 2590 using 3D front-to-front die stacking on the active interposer 2589. The active interposer 2589 may include, in addition to the interconnect fabric 2585 and the bridge structure 2587, I / O hardware logic 2591, a cache memory 2592, and other hardware logic 2593. The interconnect fabric 2585 enables communication between the numerous logic chiplets within the active interposer 2589. The fabric 2585 may be a NoC interconnect or another form of packet-switched fabric that switches data packets between components of the package assembly.For complex assemblies, the 2585 fabric can be a dedicated chiplet that enables communication between the various hardware logic of the 2590 package assembly.

[0277] The hardware logic chiplets may include dedicated hardware logic chiplets 2572, a logic or I / O chiplet 2574, and / or memory chiplets 2575. The dedicated hardware logic chiplets 2572 and the logic or I / O chiplet 2574 may be implemented at least partially in configurable logic or in fixed-functionality logic hardware and may include one or more portions of one or more processor cores, graphics processors, parallel processors, or other acceleration devices described herein. The memory chiplets 2575 may be DRAM memory (e.g., GDDR, HBM) or cache memory (SRAM). The cache memory 2592 within the active interposer 2589 (or the substrate 2580) can function as a global cache for the chassis assembly 2590, as part of a distributed global cache, or as a dedicated cache for the fabric 2585.

[0278] Each chiplet may be fabricated as a separate semiconductor chip and coupled to a base die that is embedded in or coupled to the substrate 2580. Coupling to the substrate 2580 may be performed via an interconnect structure 2573. The interconnect structure 2573 may be configured to carry electrical signals between the numerous chiplets and the logic within the substrate 2580. The interconnect structure 2573 may include interconnects such as, but not limited to, bumps or pillars. In some embodiments, the interconnect structure 2573 may be configured to carry electrical signals, such as input / output (I / O) signals and / or power or ground signals, associated with the operation of the logic, I / O, and memory chiplets.In one embodiment, an additional interconnect structure couples the active interposer 2589 to the substrate 2580.

[0279] The substrate 2580 may be an epoxy-based laminate substrate and / or may include other suitable types of substrates. The package assembly 2590 may be connected to other electrical devices via a package interconnect 2583. The package interconnect 2583 may be coupled to a surface of the substrate 2580 to conduct electrical signals to other electrical devices, such as a motherboard, another chipset, or a multi-chip module.

[0280] A logic or I / O chiplet 2574 and a memory chiplet 2575 may be electrically coupled via a bridge 2587 configured to route electrical signals between the logic or I / O chiplet 2574 and a memory chiplet 2575. The bridge 2587 may be a dense interconnect structure that provides a route for electrical signals. The bridge structure 2587 may include a bridge substrate composed of glass or a suitable semiconductor material. Electrical routing features may be formed on the bridge substrate to provide a chip-to-chip connection between the logic or I / O chiplet 2574 and a memory chiplet 2575. The bridge structure 2587 may also be referred to as a silicon bridge or an interconnect bridge. For example, the 2587 bridge structure is an embedded multi-die interconnect bridge (EMIB).Alternatively, the 2587 bridge structure can simply be a direct connection from one single chip to another single chip.

[0281] Fig. Figure 25B illustrates a package assembly 2594 including replaceable chiplets 2595, according to one embodiment. The replaceable chiplets 2595 can be mounted in standardized chiplet slots or chiplet sockets on base chiplets 2596, 2598. The base chiplets 2596, 2598 can be coupled via a bridge interconnect 2597, which can be similar to the other bridge interconnects described herein and can be, for example, an EMIB. The memory chiplets can also be connected to logic or I / O chiplets via a bridge interconnect. The I / O and logic chiplets can communicate via an interconnect fabric. The base chiplets can each carry one or more slots in a standardized format for logic, I / O, or memory / caches.

[0282] SRAM and power supply circuitry may be mounted in one or more of the base chiplets 2596, 2598, which may be manufactured using a different process technology than the replaceable chiplets 2595 stacked on top of the base chiplets. For example, the base chiplets 2596, 2598 may be manufactured using a larger process technology, while the replaceable chiplets may be manufactured using a smaller process technology. One or more of the replaceable chiplets 2595 may be memory chiplets (e.g., a DRAM). Different memory densities may be selected for the package assembly 2594, depending on the desired performance and / or capability for the product using the package assembly 2594.Additionally, logic chiplets with different numbers or types of functional units can be selected at assembly time based on the product's target performance and / or capability. Furthermore, chiplets containing different types of IP logic cores can be inserted into the interchangeable chiplet slots, enabling hybrid processor designs where IP blocks from different technologies can be mixed and matched. Example integrated system-on-chip circuit

[0283] Fig. Figure 26 illustrates an example integrated circuit that may be manufactured using one or more IP cores. In addition to what is illustrated, other logic and circuitry may be incorporated, including additional graphics processors / cores, peripheral interface controllers, or general-purpose processor cores. The elements of Fig. 26 have the same or similar names as the elements of any other figure described herein, describe the same elements as in the other figures, may operate or function in a similar manner, may include the same components, and may be associated with entities other than, but are not limited to, those described elsewhere herein.

[0284] The example integrated circuit 2600 includes one or more application processors 2605 (e.g., CPUs), a graphics processor 2610, which may be a variant of the graphics processor 1408 or any graphics processor described herein, and may be used in place of any graphics processor described herein. Therefore, the disclosure of any features in combination with a graphics processor herein also discloses, but is not limited to, a corresponding combination with the graphics processor 2610. The integrated circuit 2600 may additionally include an image processor 2615 and / or a video processor 2620, each of which may be a modular IP core from the same or several different design devices. The system-on-chip integrated circuit 2600 may include peripheral or bus logic, including a USB controller 2625, a UART controller 2630, an SPI / SDIO controller 2635, and an I2S / I 2C controller 2640. Additionally, the integrated circuit may include a display device 2645 coupled to an HDMI (High Definition Multimedia Interface) controller 2650 and / or a reliability, availability, and serviceability (RAS) engine 2655. The RAS engine 2655 is used to identify potential errors that may occur during device runtime to minimize the downtime that would result if these potential errors were to occur. Data storage may be provided by a flash memory subsystem 2660, which includes flash memory and a flash memory controller. A memory interface may be provided via a memory controller 2665 for accessing SDRAM or SRAM memory devices. Some integrated circuits additionally include an integrated security engine 2670. Fast path CPU mode for offloading accelerator devices

[0285] Parallel processing accelerator devices, such as a GPU or an AI accelerator such as a neural processing unit (NPU), are configured to perform parallel data operations using SIMD or SIMT using parallel processing units, vector processing units, and / or matrix or tensor accelerators. Such devices may be capable of performing individual scalar operations, but these operations may not be executed efficiently. Additionally, some scalar operations may not be executable depending on the ISA supported by the accelerator.Here, we describe a different approach to solving the problem of heterogeneous computations by allowing an accelerator device, such as an NPU or GPU, to offload scalar operations scattered among its parallel operations back to the CPU for a computation. Such operations may previously have been able to be offloaded to an accelerator software stack executed via the CPU. However, the round-trip between a kernel executed by an accelerator and the OS and accelerator software stack can introduce unacceptable latency. In contrast, the fast-path CPU offload mode allows scalar operations to be executed in the CPU without incurring latency introduced by the OS or accelerator software stack.

[0286] In one embodiment, fast-path CPU swapping is implemented via an additional CPU execution mode located outside the normal CPU execution domain. This additional CPU execution mode is analogous to system management mode (SMM), but with a much lower execution privilege. When SMM is enabled, an alternative software system in the computer's firmware or a hardware-assisted debugger executes with high privilege. Unlike SMM, this new execution mode (e.g., GPU swapping mode) operates independently of the system firmware or other alternative software systems and executes with low privilege. The CPU executes the GPU swapping instructions without requiring any software environment or operating system control. During GPU swapping mode, the CPU operates with limited access to CPU memory.In one embodiment, access may be restricted to only the address of a scalar binary to be executed. In one embodiment, the instructions of the scalar binary may be allowed only restricted access to an address space in the memory of the graphics processor. The accessible address space may be restricted to the address space where data used to perform the swapped operation is stored. In one embodiment, the accessible address space may be specified during swapping. The accessible address space may be restricted to memory accessible to the GPU program code performing the swapping operation.

[0287] Fig. Figure 27 illustrates a CPU mode transition for offloading from a GPU. The CPU mode transition is illustrated in the context of an execution timeline 2701. A GPU 2720 may execute a workload dispatched for execution by program code. The workload may be dispatched for execution by an application executing by the CPU 2710 while the CPU 2710 is under OS control 2711. In an embodiment where the CPU 2710 is capable of self-dispatching workloads, self-dispatched workloads may also be allowed to offload to the CPU. For selected scalar operations within a GPU workload, the execution resources within the GPU may trigger a GPU offload interrupt, interrupting OS-controlled operation in the CPU.In one embodiment, the interrupt is a non-maskable interrupt (NMI 2712), which cannot be ignored by the OS or the CPU. The NMI 2712 causes the CPU 2710 to enter a GPU swap mode 2713. During the GPU swap mode 2713, the CPU 2710 will execute a GPU swap binary independently of the OS. Upon completion (done 2714), the CPU 2710 transitions back to OS control 2711.

[0288] Although a GPU offload mode is described in which a CPU accepts offloading of a scalar workload from a GPU, fast-path offloading for scalar operations can be enabled to allow a variety of highly parallel accelerators to offload scalar operations to a CPU. Scalar workload offloading is particularly useful for, but not necessarily limited to, processor architectures in which a single processor includes cores or accelerators with multiple different types of hardware architectures. These architectures from different hardware vendors may, for example, be referred to as XPU or APU processors, which are processors in which a CPU is integrated on-chip or on-package with one or more additional hardware accelerators, including GPUs, FPGAs, or ASICs with specialized accelerators.

[0289] Fig. Figure 28 illustrates a system 2800 in which a fast-path CPU mode is enabled for mixed XPU workloads. Fast-path operations are described here in the context of offloading operations from a GPU to a CPU. A fast-path mechanism may also be enabled in a similar manner by other types of accelerators, particularly highly parallel accelerators that wish to offload scalar workloads. Furthermore, while GPU offload mode is described in the context of XPU support for discrete GPU offload mode execution, discrete GPU offload mode may also be implemented in a secure computing environment where the GPU is within the CPU's trusted execution environment and / or a low-latency, high-bandwidth interconnect is enabled between the CPU and the discrete GPU.Fast-path offloading allows the GPU to offload the performance of scalar workloads to a CPU while bypassing overhead associated with the GPU software stack. The GPU is enabled with a low-latency interrupt path in a dedicated, low-privilege mode of the CPU, with no OS overhead.

[0290] The system 2800 includes a memory 2802, a CPU 2810, and a GPU 2820. The CPU 2810 and the GPU 2820 are coupled to the memory 2802 via memory interconnects and / or via a system interconnect. The CPU 2810 may include the CPU 2710 from Fig. 27 or any CPU described here. The GPU 2720 can replace the GPU 2820 from Fig. 27 or any GPU described herein. In one embodiment, memory 2802 is a physically unified memory system in which CPU 2810 and GPU 2820 physically share the memory devices. In one embodiment, memory 2802 is a virtually unified memory system in which different physical memory devices share a common virtual memory space. CPU 2810 is configured to execute an application 2811 and a driver 2812 for GPU 2820.

[0291] The application 2811 is configured to submit workloads to the GPU 2820 for execution, where the workloads may be graphics workloads, compute workloads, or a mixture of graphics and compute workloads. A workload submission (1) may be performed by the application 2811 to the driver 2812 to request the GPU 2820 to perform a graphics or compute operation on behalf of the application 2811. The driver 2812 will create a batch buffer (2) for submission via a workload submission interface of the GPU 2820 to the GPU 2820. The batch buffer contains instructions for execution by the GPU, which include instructions for executing program code via a processing resource array 2824 of the GPU 2820. The instructions may be processed by an instruction streamer 2822, and a job (3) may be submitted to the processing resource array 2724 for execution.

[0292] The processing resource array 2824 is an array of processing resources that may include processing resources in the form of graphics multiprocessors, compute units, or graphics cores described herein. The graphics multiprocessors, compute units, or graphics cores within the processing resource array 2824 may be, for example, GPGPU cores, tensor cores, vector engines, and / or matrix engines that are configurable to execute instructions. The processing resource array 2824 may perform computational operations (4) using data 2806 stored in memory 2802. The computational operations are generally parallel vector or matrix operations. Although in some implementations, the processing resource array 2824 may be configured to perform scalar operations, these operations are not performed efficiently.Additionally, some scalar operations may not be executable via processing resource array 2824 due to a lack of support for complex scalar operations in the ISA supported by GPU 2820. In some implementations, it may be possible to request driver 2812 to perform some preconfigured and / or predetermined operations via CPU 2810, with this request being performed over a high-latency path that may have limited functionality.

[0293] Instead, program code executing via processing resource array 2824 is enabled to follow a fast-path mode to enable offloading of a scalar workload from GPU 2820 to CPU 2810. When fast-path mode is enabled, GPU 2820 is enhanced with the ability to trigger an interrupt in CPU 2810 that triggers the transition to offload mode 2813. Processing resource array 2824 may trigger a fast-path request that includes a fence operation (5a) to facilitate ordering of the data and an NMI (5b), which is a non-maskable interrupt handled by CPU 2810 entering offload mode 2813. In one embodiment, the fast path request is initiated via a command that causes both the fence operation and the offload NMI to be performed.The fence operation will ensure that memory accesses to the data 2806 performed by the GPU 2820 are visible to the CPU 2810 before any memory operations are performed by the scalar binary 2804. When the swap mode 2813 is operating, cycles are stolen from the OS to perform scalar operations based on a small piece of binary code (the scalar binary 2804) located in memory 2802. A processor core under the control of an OS executing in the CPU may be selected to execute the swapped instructions. Execution of the instructions in the processor core may pause, and the context currently executing in the selected processor core may be saved before the processor core enters the swap mode 2813.The context may be restored to the processor core after the operations in the swap mode 2813 have been completed.

[0294] In one embodiment, the scalar binary 2804 is specified as part of the GPU job description in the workload submission initially submitted by the CPU 2810 on behalf of the application 2811 to the GPU 2820. In one embodiment, the scalar binary 2804 is loaded into memory by the driver 2812, and the address of the scalar binary 2804 is provided to the GPU 2820 by the driver 2812. For example, the driver 2812 may pass the address of the scalar binary 2804 within the batch buffer as a batch buffer command, or it may configure the address as a GPU state variable via a set state command.

[0295] In one embodiment, when swap mode 2813 is triggered, CPU 2810 retrieves scalar binary file 2804 from memory 2802 (6) and performs the swap mode computations (7). In one embodiment, the swap mode computations may be performed based in part on data 2806 stored in memory 2802. In one embodiment, the computations performed in swap mode 2813 are performed by one core of CPU 2810 while other cores continue to perform OS and application computations, thereby allowing the swap mode computations to be performed transparently to the system, even if cycles are stolen from the operating system to perform the computations.In one embodiment, CPU 2810 may be configured to allocate cycles for paged-mode compute operations to a core that is currently performing non-interactive background processes or otherwise low-priority operations for an operating system, while avoiding cores used for interactive processes.

[0296] In one embodiment, the scalar binary 2804 includes firmware instructions that define a standalone operation to be performed in swap mode 2813. In one embodiment, the scalar binary includes CPU-executable microcode or extended microcode to perform the standalone operation in swap mode 2813. The scalar binary 2804 may be limited to operations that affect the state of the specific CPU core selected to operate in swap mode 2813 and is incapable of affecting CPU operations outside of that core. In one embodiment, the scalar binary 2804 is limited to executing within the interrupt handler and operates independently of and without relying on the OS.The execution environment's access scope of scalar binary 2804 may be limited to the data 2806 that scalar binary 2804 will use to perform its operations, with the address of that data 2806 defined prior to execution. For example, in one embodiment, the data 2806 to be accessed by scalar binary 2804 is defined by application 2811 in the workload submission to driver 2812. In one embodiment, the address of data 2806 may be predefined by application 2811 or specified by the GPU program executing in processing resource array 2824 before the NMI is triggered to initiate swapping.In one embodiment, the address and / or address range in which the scalar binary file 2804 and the data 2806 to be accessed by the scalar binary file 2804 are stored is predefined by the CPU 2810.

[0297] The operations performed on CPU 2810 in swap mode 2813 are performed asynchronously relative to operations on GPU 2820. Independent threads in GPU 2820 may continue executing without blocking. Threads with a dependency on the output of the scalar workload executing in CPU 2810 may perform independent operations unless and until forward progress can no longer continue without the output of the scalar workload executing in swap mode 2813.

[0298] After CPU 2810 completes the swap mode operations, CPU 2810 may perform a fence operation (8) to ensure data ordering between the operations performed by CPU 2810 in swap mode 2813 and the operations performed by GPU 2820. Processing resource array 2824 may then perform any additional computational operations (9) that may depend on the outcome of the scalar operations performed by the CPU. When the job submitted to processing resource array 2824 by instruction streamer 2822 is completed, a completion signal (e.g., done (10)) is transmitted to instruction streamer 2822. When all batch buffer operations are completed, a batch buffer completion signal (e.g., done (11)) is transmitted to driver 2812.When all activities for the workload submitted to driver 2812 are completed, the driver may send a workload completion signal (e.g., done (11)) to application 2811. The workload completion signal indicates to application 2811 that the submitted workload is ready for exploitation or use by application 2811.

[0299] Although the techniques described here are not explicitly limited to use within a hybrid processor architecture, the techniques are best suited for use within a heterogeneous processor, such as an XPU that includes CPU and GPU processors. CPU swap mode may also be initiated for device-launched workloads in GPU devices where accelerator workloads are capable of directly launching other accelerator workloads. If swap mode 2813 is to be triggered by a device-launched workload, GPU 2820 may load scalar binary 2804 into memory 2802 before asserting the CPU swap signal if scalar binary 2804 has not been previously written by application 2811 or driver 2812.

[0300] Fig. Figure 29 illustrates a processor 2900 having a CPU and a GPU, according to one embodiment. The processor 2900 includes a CPU 2902, a GPU 2904, and a plurality of memory devices 2906A through 2906H. In one embodiment, the processor 2900 is a multi-chip module (MCM), and both the CPU 2902 and the GPU 2904 are workstation- or server-class processors. The CPU 2902 and the GPU 2904 may each be soldered to the MCM, or the MCM may include sockets for the CPU 2902 and the GPU 2904. In one embodiment, the memory devices are high-bandwidth memory devices (HBM devices) located in the MCM package and shared by the CPU 2902 and the GPU 2904. The

[0301] The GPU 2904 may include a plurality of graphics core clusters 2914A through 2914H, each including a plurality of graphics cores. The graphics core clusters 2914A through 2914H may each include, for example, a processing cluster 214 as shown in Fig. 2C or a graphics core cluster 1800 as in Fig. 18A. The graphics cores within the graphics core clusters 2914A to 2914H may each comprise, for example, a graphics multiprocessor 234 of Fig. 2C, a graphics core 1521A from Fig. 15B or a Matrix Engine 1803A from Fig. 18A included.

[0302] The CPU 2902 may include a plurality of cores 2912A-2912H, which may be architecturally homogeneous or heterogeneous. In one embodiment, all cores 2912A-2912H of the CPU 2902 may include the swap mode described herein. In one embodiment, a subset of the cores 2912A-2912H of the CPU 2902 may include the swap mode. For example, the cores 2912A-2912H may include a first group of high-performance cores and a second group of more efficient cores with a lower clock speed and lower power consumption compared to the high-performance cores. In one embodiment, one or more of the high-performance cores may support the swap mode, while another group of high-performance cores may not support the swap mode.In one embodiment, a subset of the efficient cores is configured to support swap mode, and these cores are considered high priority for scheduling background operating system processes from which cycles for swap operations can be stolen transparently without visible disruption.

[0303] Fig. 30 illustrates a chiplet-based processor 3000 with CPU, GPU, and accelerator execution resources, according to one embodiment. The chiplet-based processor 3000 is a modular heterogeneous parallel processor including multiple chiplets in the form of discrete integrated circuits assembled on an active base die having multiple standardized chiplet slots. In one embodiment, the chiplet-based processor 3000 includes a global logic unit 3001, an interface 3002, a scheduler 3003, a media unit 3004, a CPU 3005, a set of parallel compute units 3006A through 3006E, an accelerator 3007, a DMA engine 3008, and a set of cache / memory units 301A, 301B.The set of parallel compute units 3006A through 3006E may be dedicated general-purpose compute units or graphics and compute units operating in conjunction with a fixed function pipeline and a programmable graphics pipeline. CPU 3005 includes a swap mode 2813 as described herein. Each of the parallel compute units 3006A through 3006E and the accelerator 3007 are configured to utilize the swap mode 2813 of CPU 3005.

[0304] In one embodiment, each of the parallel computing units 3006A-3006E includes a set of vector engines 3012A-3012N, a set of matrix engines 3014A-3014N, and at least one cache 3016. The vector engines 3012A-3012N may include GPGPU cores 262, graphics cores 370, vector engines 1522A-1522F, 1524A-1524F, or compute units 1560A-1560N, or similar parallel or vector computing units. Matrix engines 3014A through 3014N may include Tensor Cores 264, Tensor Cores 371, Matrix engines 1523A through 1523F, 1525A through 1525F, or similar matrix or tensor accelerators. Chiplet-based processor 3000 may generally implement any of the graphics processor or general-purpose parallel computing processor architectures described herein.In various embodiments and configurations, the components of the chiplet-based processor 3000 may be arranged, stacked, or positioned in two dimensions (2D), two and a half dimensions (2.5D), or three dimensions (3D).

[0305] Global logic unit 3001, in one embodiment, includes global functionality for parallel processor 3000, including device configuration registers, global schedulers, power management logic, and the like. Global logic unit 3001 may also include clock control logic and power control logic to perform dynamic voltage and frequency scaling for chiplet-based processor 3000. In one embodiment, global logic unit 3001 includes one-time programmable memory, such as eFuses or antifuses, for hardware key storage, device identifiers, etc. Global logic unit 3001 may also include reset logic, test logic, error handling, and error recovery logic.

[0306] Interface 3002 may include a front-end interface for chiplet-based processor 3000 and may include system interconnect logic for connecting chiplet-based processor 3000 via a system interconnect protocol, such as, but not limited to, the PCIe and / or CXL protocols. Thread dispatcher 3003 may receive workloads from interface 3002 and dispatch threads for the workload to compute units 3005A through 3005E. If the workload includes any media operations, at least a portion of these operations may be performed by media unit 3004. The media unit may also offload some operations to compute units 3005A through 3005E. The cache / memory units 3010A to 3010B may include an SRAM for cache memory (e.g., the L3 cache), a DRAM for device-internal random access memory (e.g.,HBM, GDDR), or a single cache / memory unit may include both SRAM and DRAM devices. The accelerator 3007 may be any type of accelerator device, including a dedicated artificial intelligence accelerator or neural accelerator, a digital signal processor, or other types of accelerators. In one embodiment, the accelerator 3007 may be or include programmable logic, such as an FPGA. The DMA engine 3008 includes copy engines to move data from one portion of the memory of the chiplet-based processor 3000 to another portion of the memory of the chiplet-based processor 3000. The DMA engine 3008 may include codecs for encoding and decoding and / or compressing and decompressing data during transfer.

[0307] The modular nature of the chiplet-based processor 3000 enables improved design and manufacturing capability compared to monolithic designs. As the size of integrated circuit dies increases to meet performance requirements, designs begin to approach the photolithography reticle limit, which restricts the size of a single integrated circuit. If a die can fit within the reticle limit, multiple smaller dies connected via in-package interconnects may be preferred for yield optimization and die reuse across multiple market segments. The modular nature of the chiplet-based processor 3000 also enables the use of several different manufacturing technologies for different components.For example, the compute units 3005A through 3005E can be implemented in an advanced process node to enable power-efficient performance, while the memory and I / O control functionality associated with the cache / memory 3010A through 3010B and the interface 3002 can be reused from a design already deployed in an established process node. Such partitioning can result in smaller die, lower manufacturing costs, reduced design time, and reduced design costs. Chiplet integration into the package also allows designers to make different tradeoffs for different market segments by selecting different numbers and types of die. For example, a designer can select different numbers of compute, memory, and I / O dies depending on the needs of the market segment.Since different die designs do not have to be produced for different market segments, lower product SKU costs can be realized.

[0308] In various embodiments, chiplet-based processor 3000 may be implemented using both homogeneous and heterogeneous chiplet tiles. From a software perspective, the compute model may be partitioned so that chiplet-based processor 3000 may be partitioned into multiple sub-devices with separate instruction schedulers. Although the instruction schedulers may be separate, the scheduling model and a unified scheduling model may be used across all chiplet functions and / or sub-devices. In one embodiment, the cache memory located within cache / memory units 3010A-3010E may be distributed across multiple chiplets, which may minimize the crossbar bandwidth required to connect the cache memory to compute units 3005A-3005E.

[0309] Fig. Figure 31 illustrates a graph-based algorithm 3100 with path determination computed via an offloaded operation. The graph-based algorithm 3100 can be implemented using graph-based programming techniques, where programs are represented as graphs, where nodes represent operations or data, and edges represent the relationships or data flow between these nodes. Graph-based programming allows for explicit modeling of data dependencies and flows, which can help optimize memory usage and data transfer. Furthermore, machine learning frameworks can use computational graphs to define neural networks. The computational graphs are then used to optimize and execute operations on GPUs.

[0310] In one embodiment, a GPU implementing a graph-based algorithm may perform a first GPU compute operation 3102, represented by a node of the graph-based algorithm 3100. Once the first GPU compute operation 3102, or another operation executing in parallel, has computed sufficient data to determine which path the graph-based algorithm 3100 will take, a path computation operation 3103 may be performed via a scalar workload offloaded to the CPU. The path computation operation 3103 may involve complex scalar operations that are not efficiently computable on the GPU. Rather than performing an inefficient operation on the GPU, the path computation operation 3103 may be efficiently offloaded to the CPU via the fast-path CPU offload described herein.Based on the result of the offloaded scalar workload, the GPU then performs either a second GPU compute operation 3104A or a third GPU compute operation 3104B after the first GPU compute operation 3102 has completed.

[0311] Fig. 32 illustrates a method 3200 for performing fast-path CPU offloading from an accelerator. The method 3200 may be performed by a heterogeneous processor, such as an XPU, APU, or a multi-chip module including a CPU and a GPU and / or other accelerator devices. The method 3200 enables the accelerator device to offload scalar operations to the CPU. The CPU includes circuitry for entering a offload mode in which offloaded scalar instructions may be executed independently of the operating system.

[0312] In one embodiment, method 3200 includes specifying a workload for execution that includes a scalar operation that can be offloaded to a CPU (3202). The workload will primarily be a parallel workload for execution in an accelerator, such as, but not limited to, a GPU or an NPU, where some of the operations of the parallel workload will have a dependency on the scalar operation to be performed in the CPU. The scalar operation, in turn, will have a dependency on data that must be generated during at least a portion of the parallel workload. The workload specification may define, include, or identify instructions for performing the scalar operation that are offloaded to the CPU.The instructions may be contained in a binary object that is loaded into a memory location accessible to the accelerator and the CPU. In one embodiment, the binary object is loaded into the memory location by the application prior to workload submission. In one embodiment, the binary object may be loaded into the memory location by the accelerator prior to or in conjunction with triggering swap mode in the CPU. Method 3200 also includes triggering execution of the workload in an accelerator coupled to the CPU (3204). Execution of the workload may be triggered by a CPU-executing application by submitting the workload to an operating system API or to an API provided by a graphics, compute, or neural network framework.The workload is processed through the API and ultimately passed to the accelerator driver for submission to the accelerator. The accelerator driver creates a command buffer to execute the workload and submits the command buffer to the accelerator.

[0313] After receiving an instruction buffer from the driver, the accelerator may execute the workload until the scalar operation must be performed via CPU swap (3206). Data-parallel portions of the workload may be executed by the accelerator to produce an output. At least a portion of this output is used to perform the scalar operation. The scalar operation may be performed once the output upon which the scalar operation depends has been produced by the accelerator. The accelerator may dispatch a swap interrupt to the CPU, causing the CPU to enter swap mode (3208). In one embodiment, the swap interrupt is non-maskable. The swap interrupt is a specialized interrupt that causes a processor core of the CPU to enter swap mode.Swap mode allows a batch of instructions to be executed outside the software environment of the operating system or any applications assigned to the CPU. Swap mode is a non-privileged mode in which executed instructions have restricted access to system resources. In one embodiment, while in swap mode, the CPU core is restricted to accessing only the binary object containing the scalar instructions to be executed and the data the instructions will operate on. In one embodiment, the binary object may contain only the non-privileged instructions. In one embodiment, the CPU core is restricted to accessing a predefined address range in memory, which may contain the binary object and its associated data.In such an embodiment, the address range may be defined by the architecture of the CPU or CPU core that supports swap mode. When the binary object is loaded into memory by the application, the workload specification may indicate an address within the predefined address range in memory where the binary object is stored. The CPU core entering swap mode may retrieve the binary object from the memory address upon entering swap mode.

[0314] The CPU core entering swap mode may then execute instructions to perform the scalar operation while in swap mode (3210). In one embodiment, the instructions are standard ISA scalar instructions supported by the CPU core. In one embodiment, the instructions are microcode instructions that may bypass one or more instruction decode operations prior to execution by the CPU core. The instructions may read data generated by the parallel operations performed in the accelerator and perform the specified scalar operations to produce an output. The output may be written back to memory.Memory ordering instructions, such as fence instructions, can be used to ensure that writes performed by the accelerator to the dependent data are globally visible before that data is read by the CPU core in swap mode. Memory ordering instructions can also be used to ensure that data written by the CPU core in swap mode is globally visible before it is consumed by the accelerator. In one embodiment, the memory ordering instructions are executed by the accelerator before swapping to the CPU is initiated and after swapping returns to the CPU. The execution of swap mode on the CPU can be performed asynchronously relative to the accelerator thread that initiates swapping.The accelerator thread that initiates swap mode is not blocked by the accelerator. While a CPU core is executing swapped instructions, accelerator threads that depend on the issue of the swapped instructions can perform independent operations until the result of the scalar operation is available.

[0315] The accelerator can perform operations that depend on the output of the scalar operation as soon as the output is available (3212). For example, the accelerator can execute a dynamic algorithm in which one of several parallel data operations is performed based on the output of a scalar operation. Instead of inefficiently performing the scalar operations in the accelerator, these scalar operations can be asynchronously offloaded to the CPU. The result, used to determine which parallel data routine to perform next, can be computed asynchronously via CPU offload while the accelerator computes a parallel predecessor data routine. Additional exemplary computing device

[0316] Fig. 33 is a block diagram of a computing device including a graphics processor, according to one embodiment. Versions of computing device 3300 may be or be included in a communications device, such as a set-top box (e.g., internet-based cable television set-top boxes, etc.), GPS-based devices (GPS: Global Positioning System), etc. Computing device 3300 may also be or be included in mobile computing devices, such as cellular phones, smartphones, personal digital assistants (PDAs), tablet computers, laptop computers, e-readers, smart televisions, television platforms, wearable devices (e.g., eyeglasses, watches, bracelets, smart cards, jewelry, clothing, etc.), media players, etc.For example, in one embodiment, computing device 3300 includes a mobile computing device employing an integrated circuit ("IC"), such as a system-on-chip ("SoC" or "SOC"), in which various hardware and / or software components of computing device 3300 are integrated into a single chip.

[0317] Computing device 3300 includes a graphics processor 3304. Graphics processor 3304 represents any graphics processor described herein. In one embodiment, graphics processor 3304 includes a cache 3314, which may be a single cache or a cache segmented into multiple segments, including, but not limited to, L1, L2, L3, or L4 caches, rendering caches, depth caches, sampling caches, and / or shader unit caches. In one embodiment, cache 3314 may be a last-level cache shared with application processor 3306. Computing device 3300, in one embodiment, includes CXL logic 3312 to facilitate the sharing and transfer of data between application processor 3306 and graphics processor 3304.Computing device 3300 may also include hardware and software logic 3313 to enable scalable I / O virtualization (S-IOV) or single-root I / O virtualization (SRIOV) to provide virtual instances of graphics processor 3304 to software domains executed by application processor 3306.

[0318] In one embodiment, graphics processor 3304 includes a graphics microcontroller that implements control and scheduling logic for the graphics processor. The control and scheduling logic may be firmware executed by graphics microcontroller 3315. The firmware may be loaded at boot time by graphics driver logic 3322. The firmware may also be programmed into an electronically erasable programmable random access memory or loaded from a flash memory device within graphics microcontroller 3315. The firmware may enable a GPU OS 3316, which includes device management logic 3317, driver logic 3318, and a scheduler 3319. GPU OS 3316 may also include a graphics memory manager 3320, which may supplement or replace graphics memory manager 3321 within graphics driver logic 3322.

[0319] The graphics processor 3304 also includes a GPGPU engine 3344, which includes one or more graphics engines, graphics processor cores, and other graphics execution resources as described herein. These graphics execution resources may be presented in forms including, but not limited to, execution units, shader engines, fragme...

Claims

[1] Processor device comprising: a general-purpose processor including circuitry for providing a swap mode for executing a swapped instruction; and an accelerator device configured to: Executing a first portion of a parallel data workload, wherein at least a second portion of the parallel data workload has a dependency on a scalar operation, and wherein the scalar operation has a dependency on the first portion of the parallel data workload; Causing the general-purpose processor to enter swap mode after completion of at least the first portion of the parallel data workload, wherein the general-purpose processor enters swap mode to perform the scalar operation; and Executing the second part of the parallel data workload based at least in part on an output of the scalar operation. [2] The processor device of claim 1, wherein the accelerator device is required to trigger an interrupt in the general purpose processor to cause the general purpose processor to enter the swap mode. [3] The processor device of claim 2, wherein the interrupt is a non-maskable interrupt. [4] The processor device of any one of claims 1 to 3, wherein the general-purpose processor is configured to enter the swap mode by: Transferring at least a portion of the general-purpose processor from control by the operating system; Retrieving a binary object from memory, the binary object containing one or more instructions for performing the scalar operation; executing the one or more instructions to perform the scalar operation; Writing a result to memory; and Transferring the part of the general-purpose processor to the control of the operating system. [5] The processor device of claim 4, wherein the one or more instructions include microcode executable by the general purpose processor. [6] The processor device of claim 4 or 5, wherein the swap mode is a non-privileged execution mode in which the one or more instructions have restricted access to the system resources. [7] The processor device of any one of claims 1 to 6, wherein the accelerator device is configured to execute the second portion of the parallel data workload to select a parallel data routine from a plurality of parallel data routines, the accelerator device selecting the parallel data routine based on the output of the scalar operation. [8] The processor device of any of claims 1 to 7, wherein the second portion of the parallel data workload has a data dependency on the output of the scalar operation. [9] The processor device of any of claims 1 to 8, wherein a general-purpose processor is configured to enter swap mode via a first processor core without affecting the operations of a second processor core. [10] The processor device of any one of claims 1 to 9, wherein the accelerator device includes a graphics processor. [11] A method for enabling a swap mode in a general-purpose processor, the method comprising: specifying a workload for execution in an accelerator device that includes a scalar operation that is offloaded by the accelerator device to a general-purpose processor; executing a first parallel data portion of the workload in the accelerator device to generate first output data; causing the general-purpose processor to enter a swap mode in which instructions for performing the scalar operation are executable by the general-purpose processor independently of an operating system executing on the general-purpose processor; Executing instructions for performing the scalar operation in the general-purpose processor while in the swap mode to generate second output data, the second output data being based at least in part on the first output data; and Executing a second parallel data portion of the workload in the accelerator device based at least in part on the second output data. [12] The method of claim 11, comprising triggering execution of the workload in the accelerator device by the general purpose processor on behalf of an application executed by the general purpose processor. [13] The method of claim 11 or 12, comprising causing the general purpose processor to enter swap mode by an interrupt to the general purpose processor triggered by the accelerator device. [14] A method according to claim 13, comprising: Receiving, in the general-purpose processor, the interrupt to enter swap mode; Transferring at least a portion of the general-purpose processor from control by the operating system; Retrieving a binary object from memory, the binary object containing one or more instructions for performing the scalar operation; executing the one or more instructions to perform the scalar operation; Writing a result to memory; and Transferring the part of the general-purpose processor to the control of the operating system. [15] The method of claim 11, wherein executing the second parallel data portion of the workload includes selecting, in the accelerator device, a parallel data routine from a plurality of parallel data routines based at least in part on the second output data, and wherein the accelerator device includes a graphics processor. [16] Heterogeneous data processing system, comprising: a base die containing a plurality of chiplet sockets; and a plurality of chiplets coupled to the plurality of chiplet sockets, the plurality of chiplets including: a first chiplet including a general-purpose processor including circuitry for providing a swap mode for executing a swapped instruction; and a second chiplet including an accelerator device configured to: Executing a first portion of a parallel data workload, wherein at least a second portion of the parallel data workload has a dependency on a scalar operation, and wherein the scalar operation has a dependency on the first portion of the parallel data workload; Causing the general-purpose processor to enter swap mode after completing at least the first portion of the parallel data workload, wherein the general-purpose processor enters swap mode to perform the scalar operation; and Executing the second part of the parallel data workload based at least in part on an output of the scalar operation. [17] The heterogeneous data processing system of claim 16, wherein the accelerator device is required to trigger a non-maskable interrupt in the general purpose processor to cause the general purpose processor to enter the swap mode. [18] The heterogeneous data processing system of claim 17, wherein the general purpose processor is configured to enter the swap mode by: Transferring at least a portion of the general-purpose processor from control by the operating system; Retrieving a binary object from memory, the binary object containing one or more instructions for performing the scalar operation; executing the one or more instructions to perform the scalar operation; Writing a result to memory; and Transferring the part of the general-purpose processor to the control of the operating system. [19] The heterogeneous data processing system of claim 18, wherein the accelerator device for executing the second part of the parallel data workload is configured to select a parallel data routine from a plurality of parallel data routines, wherein the accelerator device is to select the parallel data routine based on the output of the scalar operation. [20] The heterogeneous data processing system of claim 19, wherein the accelerator device includes a graphics processor.

Images