Apparatus and method for double-precision ray traversal in a ray tracing pipeline

Abstract

An apparatus and method for double-precision traversal and intersection. For example, one embodiment of a device includes: a bounding volume hierarchy (BVH) generator for constructing a BVH comprising a plurality of hierarchically arranged BVH nodes; a ray memory for storing rays that will traverse one or more traversals of the BVH nodes; a ray traversal circuit comprising a first plurality of 64-bit arithmetic logic units (ALUs) natively supporting double-precision floating-point operations, the ray traversal circuit being configured to use at least the first ALU of the one or more ALUs to traverse the first ray through the first BVH node with double-precision floating-point precision to produce a double-precision floating-point traversal result; and a plurality of execution units (EUs) coupled to the ray traversal circuit, at least one of the plurality of EUs comprising a second plurality of 64-bit ALUs capable of natively performing double-precision floating-point operations, the at least one of the plurality of EUs being configured to execute one or more intersection shaders to perform a ray primitive intersection test with double-precision floating-point precision based on the double-precision floating-point traversal result.

Description

[Technical Field]

[0001] This invention generally relates to the field of graphics processors. More specifically, this invention relates to an apparatus and method for performing double-precision traversal in a ray-tracing pipeline. [Previous Technology]

[0002] Ray tracing is a technique that simulates light transmission through entity-based rendering. Until a few years ago, it was widely used in film rendering, and therefore it was too resource-intensive for real-time performance. One of the key operations in ray tracing is handling the visibility query of ray-scene intersections, called "ray traversal," which calculates ray-scene intersections by traversing and nodes in the intersection bounding volume hierarchy (BVH). [Summary of the Invention]

[0003] and

Implementation Method

[0004] In the following description, numerous specific details are set forth for purposes of explanation in order to provide a thorough understanding of embodiments of the invention described below. However, it will be apparent to those skilled in the art that embodiments of the invention may be practiced without some of these specific details. In other instances, well-known structures and apparatuses are shown in block diagram form to avoid obscuring the basic principles of embodiments of the invention. Exemplary Graphics Processor Architecture and Data Type System Overview

[0005] FIG1 is a block diagram of a processing system 100 according to an embodiment. The system 100 can be used in a single-processor desktop computer system, a multi-processor workstation system, or a server system having a large number of processors 102 or processor cores 107. In one embodiment, the system 100 is a processing platform incorporated within a system-on-a-chip (SoC) integrated circuit for use in mobile, handheld, or embedded devices, such as in wired or wireless Internet of Things (IoT) devices having connections to a local area network or a wide area network.

[0006] In one embodiment, system 100 may include a server-based gaming platform; a game console including a game and media console; a mobile game console, handheld game console, or online game console, coupled thereto or integrated therein. In some embodiments, system 100 is part of a mobile phone, smartphone, tablet computing device, or mobile internet-connected device such as a notebook computer with low internal storage. Processing system 100 may also include wearable devices such as smartwatches; smart glasses or clothing with augmented reality (AR) or virtual reality (VR) capabilities to provide visual, audio, or haptic output to supplement real-world visual, audio, or haptic experiences, or to provide text, audio, graphics, video, holograms, or video or haptic feedback; other augmented reality (AR) devices; or other virtual reality (VR) devices, coupled thereto or integrated therein. In some embodiments, processing system 100 includes, or is part of, a television or set-top box device. In one embodiment, system 100 may include, be coupled to, or be integrated with an autonomous vehicle such as a bus, tractor, car, locomotive or electric motorcycle, airplane or glider (or any combination thereof). The autonomous vehicle can use system 100 to process the perceived environment around the vehicle.

[0007] In some embodiments, one or more processors 102 each include one or more processor cores 107 for processing instructions that, when run, perform operations of the system and user software. In some embodiments, at least one of the one or more processor cores 107 is configured to process a particular instruction set 109. In some embodiments, the instruction set 109 may facilitate Complex Instruction Set Computing (CISC), Reduced Instruction Set Computing (RISC), or computation via Very Long Instruction Word (VLIW). The one or more processor cores 107 may process different instruction sets 109, and may include instructions that facilitate the emulation of other instruction sets. The processor core 107 may also include other processing means, such as a digital signal processor (DSP).

[0008] In some embodiments, processor 102 includes cache memory 104. Depending on the architecture, processor 102 may have a single internal cache or multiple levels of internal cache. In some embodiments, cache memory is shared among various components of processor 102. In some embodiments, processor 102 also uses external caches (e.g., Level 3 (L3) cache or final-level cache (LLC)) (not shown), which may be shared among processor cores 107 using known cache coherence techniques. Register file 106 may be additionally included in processor 102 and may include different types of registers for storing different types of data (e.g., integer registers, floating-point registers, status registers, and instruction pointer registers). Some registers may be general-purpose registers, while others may be specific to the design of processor 102.

[0009] In some embodiments, one or more processors 102 are coupled to one or more interface buses 110 to transmit communication signals (such as address, data, or control signals) between the processors 102 and other components in the system 100. In one embodiment, the interface bus 110 may be a processor bus, such as a version of a Direct Media Interface (DMI) bus. However, the processor bus is not limited to a DMI bus, but may include one or more peripheral component interconnect buses (e.g., PCI, PCI express), memory buses, or other types of interface buses. In one embodiment, (one or more) processors 102 include an integrated memory controller 116 and a platform controller hub 130. The memory controller 116 facilitates communication between memory devices and other components of the system 100, while the platform controller hub (PCH) 130 provides connectivity to I / O devices via a regional I / O bus.

[0010] Memory device 120 may be a dynamic random access memory (DRAM) device, a static random access memory (SRAM) device, a flash memory device, a phase-change memory device, or some other memory device with suitable performance for use as process memory. In one embodiment, memory device 120 is capable of operating as system memory of system 100 to store data 122 and instructions 121 for use by one or more processors 102 when running applications or processes. Memory controller 116 is also coupled to an optional external graphics processor 118, which may communicate with one or more graphics processors 108 in processor 102 to perform graphics and media operations. In some embodiments, graphics, media, and / or computational operations may be assisted by accelerator 112, which is a coprocessor that can be configured to perform a dedicated set of graphics, media, or computational operations. For example, in one embodiment, accelerator 112 is a matrix multiplication accelerator for optimizing machine learning or computational operations. In one embodiment, accelerator 112 is a ray tracing accelerator that can be used to perform ray tracing operations in conjunction with graphics processor 108. In one embodiment, external accelerator 119 may replace accelerator 112 or be used in conjunction with accelerator 112.

[0011] In some embodiments, the display device 111 may be connected to one or more processors 102. The display device 111 may be one or more of an internal display device (as in mobile electronic devices or laptop devices) or an external display device attached via a display interface (e.g., DisplayPort). In one embodiment, the display device 111 may be a head-mounted display (HMD), such as a stereoscopic display device for use in virtual reality (VR) or augmented reality (AR) applications.

[0012] In some embodiments, the platform controller hub 130 enables peripheral devices to connect to the memory device 120 and the processor 102 via a high-speed I / O bus. I / O peripheral devices include, but are not limited to, an audio controller 146, a network controller 134, a firmware interface 128, a wireless transceiver 126, a touch sensor 125, and a data storage device 124 (e.g., non-volatile memory, volatile memory, hard disk drive, flash memory, NAND, 3D NAND, 3D XPoint, etc.). The data storage device 124 can connect via a storage interface (e.g., SATA) or via a peripheral bus (such as a peripheral component interconnect bus (e.g., PCI, PCIe). The touch sensor 125 may include a touch screen sensor, pressure sensor, or fingerprint sensor. The wireless transceiver 126 may be a Wi-Fi transceiver, Bluetooth transceiver, or mobile network transceiver, such as a 3G, 4G, 5G, or LTE transceiver. The firmware interface 128 enables communication with the system firmware and may be, for example, a Unified Extended Firmware Interface (UEFI). The network controller 134 enables network connectivity to a wired network. In some embodiments, the high-performance network controller... The controller (not shown) is coupled to the interface bus 110. In one embodiment, the audio controller 146 is a multi-channel high-resolution audio controller. In one embodiment, the system 100 includes an optional legacy I / O controller 140 for coupling legacy devices (e.g., Personal System 2 (PS / 2)) to the system. The platform controller hub 130 can also be connected to one or more Universal Serial Bus (USB) controllers 142 to connect input devices such as a keyboard and mouse combination 143, a camera 144, or other USB input devices.

[0013] It will be understood that the illustrated system 100 is exemplary and not limiting, and other types of data processing systems with different configurations may be used. For example, instances of the memory controller 116 and platform controller hub 130 may be integrated into a discrete external graphics processor (such as external graphics processor 118). In one embodiment, the platform controller hub 130 and / or the memory controller 116 may be external to one or more processors 102. For example, system 100 may include an external memory controller 116 and a platform controller hub 130, which may be configured as a memory controller hub and a peripheral controller hub within a system chipset to communicate with (one or more) processors 102.

[0014] For example, a circuit board ("base") may be used on which components such as the CPU, memory, and other parts are mounted to improve heat dissipation performance. In some examples, processing components such as processors are located on the top side of the base, while near-end memory such as DIMMs is located on the bottom side. This design provides enhanced airflow, allowing these components to operate at higher frequencies and power levels than typical systems, thus improving performance. Furthermore, the base is configured to blindly overlap with power and data communication cables in the rack, enhancing its ability to be quickly removed, upgraded, reinstalled, and / or replaced. Similarly, the various components located on the base (such as processors, accelerators, memory, and data storage drives) are configured for easy upgrades due to increased spacing between them. In illustrative embodiments, the components additionally include hardware verification features to prove their authenticity.

[0015] The data center can utilize a single network architecture ("architecture") that supports a variety of other network architectures, including Ethernet and Omni-Path. The base can be coupled to a switch via fiber optic cable, which offers higher bandwidth and lower latency compared to typical twisted-pair cables (e.g., Category 5, Category 5e, Category 6, etc.). Due to the high bandwidth and low latency interconnect and network architecture, the data center may pool resources (such as memory, accelerators (e.g., GPUs, graphics accelerators, FPGAs, ASICs, neural networks and / or artificial intelligence accelerators, etc.) and physically decomposed data storage drives, and provide them to computing resources (e.g., processors) as needed, allowing computing resources to access resources in the pool like local resources.

[0016] The power supply or power source may provide voltage and / or current to System 100 or any component or system described herein. In one example, the power supply includes an AC to DC adapter for plugging into a wall-mounted power outlet. This AC power source may be a renewable energy source (e.g., solar energy). In one example, the power source includes a DC power source, such as an external AC to DC converter. In one example, the power source or power supply includes wireless charging hardware for charging via proximity to a charging site. In one example, the power source may include an internal battery, an AC power supply, a motion-based power supply, a solar power supply, or a fuel cell power source.

[0017] Figures 2A-2D show the computing system and graphics processor provided in the embodiments described herein. Elements in Figures 2A-2D that have the same element symbols (or names) as elements in any other figures herein can operate or function in any manner similar to, but are not limited to, those described in other parts of this document.

[0018] FIG2A is a block diagram of an embodiment of processor 200, which has one or more processor cores 202A-202N, an integrated memory controller 214, and an integrated graphics processor 208. Processor 200 may include up to and including additional cores 202N, indicated by dashed boxes. Each processor core 202A-202N includes one or more internal cache units 204A-204N. In some embodiments, each processor core may also access one or more shared cache units 206. Internal cache units 204A-204N and shared cache units 206 represent cache memory hierarchies within processor 200. The cache memory hierarchy may include at least one level of instruction and data cache within each processor core and one or more levels of shared intermediate level caches, such as level 2 (L2), level 3 (L3), level 4 (L4), or other levels of cache, wherein the highest-order cache prior to external memory is ordered LLC. In some embodiments, cache consistency logic maintains consistency between various cache units 206 and 204A-204N.

[0019] In some embodiments, the processor 200 may further include a collection of one or more bus controller units 216 and a system agent core 210. The one or more bus controller units 216 manage a collection of peripheral buses, such as one or more PCI or PCI express buses. The system agent core 210 provides management functionality for various processor components. In some embodiments, the system agent core 210 includes one or more integrated memory controllers 214 to manage access to various external memory devices (not shown).

[0020] In some embodiments, one or more of the processor cores 202A-202N include support for simultaneous multithreading. In such an embodiment, the system agent core 210 includes components for coordinating and operating the cores 202A-202N during multithreaded processing. The system agent core 210 may also include a power control unit (PCU) including logic and components for regulating the power states of the processor cores 202A-202N and the graphics processor 208.

[0021] In some embodiments, the processor 200 further includes a graphics processor 208 for performing graphics processing operations. In some embodiments, the graphics processor 208 is coupled to a collection of shared cache units 206 and a system proxy core 210 including one or more integrated memory controllers 214. In some embodiments, the system proxy core 210 further includes a display controller 211 for driving the graphics processor output to one or more coupled displays. In some embodiments, the display controller 211 may also be a separate module coupled to the graphics processor via at least one interconnect, or it may be integrated within the graphics processor 208.

[0022] In some embodiments, the ring interconnect unit 212 is used to couple internal components of the processor 200. However, alternative interconnect units, such as point-to-point interconnects, switched interconnects, or other technologies (including those well known in the art), may be used. In some embodiments, the graphics processor 208 is coupled to the ring interconnect 212 via I / O link 213.

[0023] The exemplary I / O link 213 represents at least one of a variety of I / O interconnects, including on-package I / O interconnects, which facilitate communication between various processor components and high-performance embedded memory modules 218 (such as eDRAM modules). In some embodiments, each of the processor cores 202A-202N and the graphics processor 208 may use the embedded memory module 218 as a shared final-level cache.

[0024] In some embodiments, processor cores 202A-202N are homogeneous cores running the same instruction set architecture. In another embodiment, processor cores 202A-202N are heterogeneous in terms of instruction set architecture (ISA), wherein one or more of processor cores 202A-202N run a first instruction set, while at least one of the other cores runs a subset of the first instruction set or a different instruction set. In one embodiment, processor cores 202A-202N are heterogeneous in terms of microarchitecture, wherein one or more cores with relatively high power consumption are coupled to one or more power cores with lower power consumption. In one embodiment, processor cores 202A-202N are heterogeneous in terms of computing power. Furthermore, processor 200 can be implemented on one or more chips or as a SoC integrated circuit having the shown components and other components.

[0025] FIG. 2B is a block diagram of the hardware logic of a graphics processor core 219 according to some embodiments described herein. Elements in FIG. 2B having the same element symbols (or names) as elements in any other figures herein are capable of operating or functioning in any manner similar to, but not limited to, those described in other parts of this document. The graphics processor core 219 (sometimes referred to as a core chip) may be one or more graphics cores within a modular graphics processor. The graphics processor core 219 is an example of a graphics core chip, and the graphics processor described herein may include multiple graphics core chips based on a target power and performance envelope. Each graphics processor core 219 may include a fixed-function block 230 coupled to multiple sub-cores 221A-221F (also referred to as sub-chips), which contains modular blocks of general and fixed-function logic.

[0026] In some embodiments, the fixed-function block 230 includes a geometry / fixed-function pipeline 231, which can be shared by all sub-cores of the graphics processor core 219, for example, in a lower-performance and / or lower-power graphics processor implementation. In various embodiments, the geometry / fixed-function pipeline 231 includes a 3D fixed-function pipeline (e.g., 3D pipeline 312 as shown below in Figures 3 and 4), a video front-end unit, a thread generator and a thread dispatcher, and a unified return buffer manager that manages the unified return buffer (e.g., unified return buffer 418 as shown below in Figure 4).

[0027] In one embodiment, the fixed function block 230 further includes a graphics SoC interface 232, a graphics microcontroller 233, and a media pipeline 234. The graphics SoC interface 232 provides an interface between the graphics processor core 219 and other processor cores in the system-on-a-chip integrated circuit. The graphics microcontroller 233 is a programmable subprocessor that can be configured to manage various functions of the graphics processor core 219, including thread dispatch, scheduling, and preemption. The media pipeline 234 (e.g., media pipeline 316 of Figures 3 and 4) includes logic that facilitates the decoding, encoding, preprocessing, and / or post-processing of multimedia data containing image and video data. The media pipeline 234 implements media operations via logic within request computation or sampling cores 221A-221F.

[0028] In one embodiment, the SoC interface 232 causes the graphics processor core 219 to communicate with a general-purpose application processor core (e.g., CPU) and / or other components within the SoC, including memory hierarchy elements such as shared last-level cache memory, system RAM, and / or DRAM embedded in the die or package. The SoC interface 232 may also cause communication with fixed-function devices (such as camera imaging pipelines) within the SoC and cause the use and / or implementation of global memory atoms that can be shared between the graphics processor core 219 and the CPU within the SoC. The SoC interface 232 may also implement power management control for the graphics processor core 219 and enable the interface between the clock domain of the graphics core 219 and other clock domains within the SoC. In one embodiment, the SoC interface 232 causes command buffers to be received from a command streamer and a global thread dispatcher, the command streamer and global thread dispatcher being configured to provide commands and instructions to one or more graphics cores within the graphics processor. When a media operation is to be performed, the commands and instructions can be dispatched to the media pipeline 234, or when a graphics processing operation is to be performed, the commands and instructions can be dispatched to the geometry and fixed function pipelines (e.g., geometry and fixed function pipeline 231, geometry and fixed function pipeline 237).

[0029] The graphics microcontroller 233 can be configured to perform various scheduling and management tasks for the graphics processor core 219. In one embodiment, the graphics microcontroller 233 can perform graphics and / or compute workload scheduling on various graphics parallel engines in the execution unit (EU) arrays 222A-222F, 224A-224F within the subcores 221A-221F. In this scheduling model, host software executing on the CPU core of the SoC containing the graphics processor core 219 can submit a workload of one of a plurality of graphics processor doorbells, which invokes scheduling operations on the appropriate graphics engine. The scheduling operations include determining the workload to run next, submitting the workload to a command streamer, preempting an existing workload running on the engine, monitoring the progress of the workload, and notifying the host software when the workload is completed. In one embodiment, the graphics microcontroller 233 may also facilitate a low-power or idle state of the graphics processor core 219, providing the graphics processor core 219 with the ability to save and restore registers within the graphics processor core 219 independently of the operating system and / or graphics driver software across low-power state transitions.

[0030] The graphics processing unit (GPU) core 219 may have more or fewer sub-cores 221A-221F as shown, up to N modular sub-cores. For each group of N sub-cores, the GPU core 219 may also include shared functional logic 235, shared and / or cache memory 236, geometry / fixed-function pipeline 237, and additional fixed-function logic 238 to accelerate various graphics and computational processing operations. The shared functional logic 235 may include logic units (e.g., samplers, mathematical and / or execution inter-thread communication logic) associated with the shared functional logic 420 of FIG4, which may be shared by each of the N sub-cores within the GPU core 219. The shared and / or cache memory 236 may be the last-level cache of the groups of N sub-cores 221A-221F in the GPU core 219, and may also be used as shared memory accessible by multiple sub-cores. The geometry / fixed function pipeline 237 can be included to replace the geometry / fixed function pipeline 231 within the fixed function block 230 and can contain the same or similar logic units.

[0031] In one embodiment, the graphics processor core 219 includes additional fixed-function logic 238, which may contain various fixed-function acceleration logics for use by the graphics processor core 219. In one embodiment, the additional fixed-function logic 238 includes an additional geometry pipeline for location-only shading. In location-only shading, there are two geometry pipelines: the geometry / fixed-function pipeline 238, the full geometry pipeline within 231, and the culling pipeline, which are additional geometry pipelines that can be included within the additional fixed-function logic 238. In one embodiment, the culling pipeline is a trimmed version of the full geometry pipeline. The full pipeline and the culling pipeline can execute different instances of the same application, each with a separate context. Location-only shading can hide long culling runs of discarded triangles, thereby allowing shading to be completed earlier in some cases. For example, in one embodiment, the culling pipeline logic within additional fixed-function logic 238 can execute the position shader in parallel with the main application and typically produces critical results faster than the full pipeline because the culling pipeline only acquires and colors the position attributes of vertices without performing rasterization and drawing of pixels on the frame buffer. The culling pipeline can use the resulting critical results to compute visibility information for all triangles, regardless of whether those triangles were culled. The full pipeline (which in this instance might be called the replay pipeline) can use the visibility information to skip culled triangles, coloring only the visible triangles that ultimately pass to the rasterization stage.

[0032] In one embodiment, the additional fixed-function logic 238 may also include machine learning acceleration logic, such as fixed-function matrix multiplication logic, to implement optimizations for machine learning training or inference.

[0033] Each graphics subcore 221A-221F contains a set of execution resources that can be used to perform graphics, media, and computational operations in response to requests from the graphics pipeline, media pipeline, or shader program. The graphics subcore 221A-221F includes multiple EU arrays 222A-222F, 224A-224F, thread dispatch and inter-thread communication (TD / IC) logic 223A-223F, 3D (e.g., texture) samplers 225A-225F, media samplers 206A-206F, shader processors 227A-227F, and shared area memory (SLM) 228A-228F. Each EU array 222A-222F, 224A-224F contains multiple execution units, which are general-purpose graphics processing units capable of serving graphics, media, or computational operations (including graphics, media, or computational shader programs) to perform floating-point and integer / fixed-point logic operations. TD / IC logic units 223A-223F perform region thread allocation and thread control operations on execution units within the subcore, and facilitate communication between threads executing on execution units within the subcore. 3D samplers 225A-225F can read data associated with textures or other 3D graphics into memory. The 3D samplers can read texture data differently depending on the configured sampling state and the texture format associated with a given texture. Media samplers 206A-206F can perform similar read operations based on the type and format associated with the media data. In one embodiment, each graphics subcore 221A-221F may alternatively include a uniform 3D and media sampler. Threads executing on execution units within each subcore 221A-221F can utilize shared region memory 228A-228F within each subcore, enabling threads executing in a thread group to use a shared on-chip memory pool.

[0034] Figure 2C shows a graphics processing unit (GPU) 239, which includes a dedicated set of graphics processing resources arranged in multi-core groups 240A-240N. Although only details of a single multi-core group 240A are provided, it should be understood that other multi-core groups 240B-240N may be equipped with the same or similar sets of graphics processing resources.

[0035] As shown in the figure, the multi-core group 240A may include a set of graphics cores 243, a set of tensor cores 244, and a set of ray tracing cores 245. A scheduler / dispatcher 241 schedules and assigns graphics threads to execute on the various cores 243, 244, and 245. When a graphics thread is executed, a set of register files 242 stores operand values ​​used by the cores 243, 244, and 245. These may include, for example, integer registers for storing integer values, floating-point registers for storing floating-point values, vector registers for storing compressed data elements (integer and / or floating-point data elements), and slice registers for storing tensor / matrix values. In one embodiment, the slice registers are implemented as a combined set of vector registers.

[0036] One or more combinations of Level 1 (L1) cache and shared memory units 247 regionally store graphics data such as texture data, vertex data, pixel data, ray data, bounding volume data, etc., within each multi-core group 240A. One or more texture units 247 may also be used to perform texture operations, such as texture mapping and sampling. A Level 2 (L2) cache 253, shared by all or a subset of the multi-core groups 240A-240N, stores graphics data and / or instructions for multiple concurrent graphics threads. As shown, the L2 cache 253 may be shared across multiple multi-core groups 240A-240N. One or more memory controllers 248 couple the GPU 239 to memory 249, which may be system memory (e.g., DRAM) and / or dedicated graphics memory (e.g., GDDR6 memory).

[0037] Input / output (I / O) circuitry 250 couples GPU 239 to one or more I / O devices 252, such as digital signal processors (DSPs), network controllers, or user input devices. On-chip interconnects can be used to couple I / O devices 252 to GPU 239 and memory 249. One or more I / O memory management units (IOMMUs) 251 of I / O circuitry 250 directly couple I / O devices 252 to system memory 249. In one embodiment, IOMMU 251 manages multiple sets of page tables to map virtual addresses to physical addresses in system memory 249. In this embodiment, I / O devices 252, CPU 246, and GPU 239 may share the same virtual address space.

[0038] In one implementation, IOMMU 251 supports virtualization. In this case, a first set of page tables can be managed for mapping guest / graphics virtual addresses to guest / graphics entity addresses, and a second set of page tables (e.g., within system memory 249) for mapping guest / graphics entity addresses to system / host entity addresses. The base address of each of the first and second sets of page tables can be stored in a control register and swapped out on a context switcher (e.g., to provide access to the associated set of page tables to a new context). Although not shown in Figure 2C, each of cores 243, 244, 245 and / or multi-core groups 240A-240N can include a translation back buffer (TLB) for caching guest virtual addresses to guest entity translations, caching guest entity addresses to host entity translations, and caching guest virtual addresses to host entity translations.

[0039] In one embodiment, the CPU 246, GPU 239, and I / O device 252 are integrated on a single semiconductor wafer and / or wafer package. The memory 249 shown may be integrated on the same wafer or may be coupled to the memory controller 248 via an off-chip interface. In one implementation, the memory 249 includes GDDR6 memory that shares the same virtual address space with other physical system-level memory, although the basic principles of the invention are not limited to the specific implementation described.

[0040] In one embodiment, the tensor core 244 includes a plurality of execution units specifically designed to perform matrix operations, which are fundamental computational operations used to perform deep learning operations. For example, matrix multiplication can be used for both neural network training and inference. The tensor core 244 can perform matrix processing using various operand precisions, including single-precision floating-point numbers (e.g., 32-bit), half-precision floating-point numbers (e.g., 16-bit), integer arrays (16-bit), bit arrays (8-bit), and half-bit arrays (4-bit). In one embodiment, the neural network implements the extraction of features from each rendered scene, potentially combining details from multiple frames to construct a high-quality final image.

[0041] In deep learning implementations, parallel matrix multiplication operations can be scheduled to be performed on the tensor core 244. Training neural networks, in particular, requires a large number of matrix dot product operations. To handle the inner product formulas of N×N×N matrix multiplications, the tensor core 244 can contain at least N dot product processing elements. Before matrix multiplication begins, a complete matrix is ​​loaded into a slice register, and at least one row of a second matrix is ​​loaded in each of the N cycles. N dot products are processed in each cycle.

[0042] Depending on the specific implementation, matrix elements can be stored with different precisions, including 16-bit words, 8-bit bytes (e.g., INT8), and 4-bit half-bytes (e.g., INT4). Different precision modes can be specified for Tensor Core 244 to ensure that the most efficient precision is used for different workloads (e.g., inference workloads that can tolerate quantization to bits and half-bytes).

[0043] In one embodiment, the ray tracing core 245 accelerates ray tracing operations for both real-time and non-real-time ray tracing implementations. Specifically, the ray tracing core 245 includes ray traversal / intersection circuitry for performing ray traversal using a bounded volume hierarchy (BVH) and identifying intersections between rays and primitives enclosed within the BVH volume. The ray tracing core 245 may also include circuitry for performing depth testing and culling (e.g., using a Z-buffer or similar arrangement). In one implementation, the ray tracing core 245 performs traversal and intersection operations consistent with the image denoising techniques described herein, with at least a portion of these operations performed on the tensor core 244. For example, in one embodiment, the tensor core 244 implements a deep learning-like neural network to perform denoising on frames generated by the ray tracing core 245. However, the CPU 246, graphics core 243, and / or ray tracing core 245 may also implement all or part of the denoising and / or deep learning algorithms.

[0044] Furthermore, as described above, a distributed denoising method can be employed, wherein the GPU 239 is located in a computing device coupled to other computing devices via a network or high-speed interconnect. In this embodiment, the interconnected computing devices share neural network learning / training data to improve the speed at which the entire system learns to perform denoising for different types of image frames and / or different graphics applications.

[0045] In one embodiment, the ray tracing core 245 handles all BVH traversals and ray-primitive intersections, thereby saving the graphics core 243 from being overloaded by thousands of instructions per ray. In one embodiment, each ray tracing core 245 includes a first set of dedicated circuitry for performing bounding box tests (e.g., for traversal operations) and a second set of dedicated circuitry for performing ray-triangle intersection tests (e.g., intersection of traversed rays). Thus, in one embodiment, the multi-core group 240A can simply fire ray detectors, and the ray tracing core 245 independently performs ray traversal and intersection and returns hit data (e.g., hit, miss, multiple hits, etc.) to the execution thread context. Other cores 243, 244 are freed up to perform other graphics or computational work while the ray tracing core 245 performs traversal and intersection operations.

[0046] In one embodiment, each ray tracing core 245 includes a traversal unit that performs BVH test operations and an intersection unit that performs ray-primitive intersection tests. The intersection unit generates a "hit," "miss," or "multiple hit" response and provides it to the appropriate thread. During traversal and intersection operations, execution resources of other cores (e.g., graphics core 243 and tensor core 244) are released to perform other forms of graphics work.

[0047] In a particular embodiment described below, a hybrid rasterization / ray tracing method is used, wherein the operation is distributed between the graphics core 243 and the ray tracing core 245.

[0048] In one embodiment, the ray tracing core 245 (and / or other cores 243, 244) includes hardware support for ray tracing instruction sets, such as Microsoft's DirectX Raytracing (DXR), which includes DispatchRays commands, and shaders for ray generation, nearest hit, any hit, and miss, enabling a unique set of shaders and texture assignments for each object. Another ray tracing platform that may be supported by the ray tracing core 245, graphics core 243, and tensor core 244 is Vulkan 1.1.85. However, it should be noted that the basic principles of the invention are not limited to any particular ray tracing ISA.

[0049] Typically, various cores 245, 244, and 243 can support a set of ray tracing instructions, including instructions / functions for ray generation, nearest hit, any hit, ray primitive intersection, per-primitive and hierarchical bounding box structure, miss, access, and anomalies. More specifically, one embodiment includes ray tracing instructions that perform the following functions: Ray generation - Ray generation instructions can be executed for each pixel, sample, or other user-defined job assignment. Nearest hit - Nearest hit instructions can be executed to locate the nearest intersection point of a ray with a primitive in the scene. Any hit - Any hit instructions can identify multiple intersection points between a ray and a primitive in the scene, potentially allowing the identification of new nearest intersection points. Intersection - Intersection instructions perform a ray-primitive intersection test and output the result. Per-primitive bounding box structure - This instruction establishes a bounding box around a given primitive or set of primitives (e.g., when establishing a new BVH or other acceleration data structure). Miss - Indicates that the ray misses all geometry within the scene, or a specified region within the scene. Access - indicates the sub-volume that the ray will traverse. Exception - contains various types of exception handlers (e.g., calls for various error conditions).

[0050] FIG. 2D is a block diagram of a general-purpose graphics processing unit (GPGPU) 270 that can be configured as a graphics processor and / or computing accelerator according to embodiments described herein. The GPGPU 270 can be interconnected with a host processor (e.g., one or more CPUs 246) and memories 271, 272 via one or more system and / or memory buses. In one embodiment, memory 271 is system memory that can be shared with one or more CPUs 246, while memory 272 is device memory dedicated to the GPGPU 270. In one embodiment, components and device memory 272 within the GPGPU 270 can be mapped to memory addresses accessible by one or more CPUs 246. Access to memories 271 and 272 can be facilitated through a memory controller 268. In one embodiment, the memory controller 268 includes an internal direct memory access (DMA) controller 269, or may include logic for performing operations otherwise performed by the DMA controller.

[0051] The GPGPU 270 includes multiple cache memories, including an L2 cache 253, an L1 cache 254, an instruction cache 255, and shared memory 256, at least a portion of which can also be partitioned into cache memories. The GPGPU 270 also includes multiple compute units 260A-260N. Each compute unit 260A-260N includes a set of vector registers 261, scalar registers 262, vector logic units 263, and scalar logic units 264. Compute units 260A-260N may also include a region shared memory 265 and a program counter 266. Compute units 260A-260N may be coupled to a constant cache 267, which can be used to store constant data that does not change during the execution of the kernel or shader program running on the GPGPU 270. In one embodiment, the constant cache 267 is a scalar data cache, and the cached data can be directly extracted to the scalar register 262.

[0052] During operation, one or more CPUs 246 can write commands to temporary registers or memory mapped into the accessible address space of the GPGPU 270. The command processor 257 can read commands from the temporary registers or memory and determine how to process these commands within the GPGPU 270. The thread dispatcher 258 can then dispatch threads to compute units 260A-260N to execute these commands. Each compute unit 260A-260N can execute threads independently of other compute units. Furthermore, each compute unit 260A-260N can be independently configured for conditional computation and can conditionally output computation results to memory. When a submitted command completes, the command processor 257 can interrupt one or more CPUs 246.

[0053] Figures 3A-3C show block diagrams of additional graphics processor and computing accelerator architectures provided by the embodiments described herein. The elements of Figures 3A-3C, having the same element symbols (or names) as those in any other figures herein, may operate or function in any manner similar to, but not limited to, those described elsewhere herein.

[0054] FIG3A is a block diagram of a graphics processor 300, which may be a discrete graphics processing unit, or a graphics processor integrated with a plurality of processing cores or other semiconductor devices (such as, but not limited to, memory devices or network interfaces). In some embodiments, the graphics processor communicates via a memory-mapped I / O interface to registers on the graphics processor and using commands to put data into processor memory. In some embodiments, the graphics processor 300 includes a memory interface 314 for accessing memory. The memory interface 314 may be an interface to local memory, one or more internal caches, one or more shared external caches, and / or to system memory.

[0055] In some embodiments, the graphics processor 300 further includes a display controller 302 to drive display output data to the display device 318. The display controller 302 includes hardware for a multi-layered display and one or more overlay planes for video or user interface elements. The display device 318 may be an internal or external display device. In one embodiment, the display device 318 is a head-mounted display device, such as a virtual reality (VR) display device or an augmented reality (AR) display device. In some embodiments, the graphics processor 300 includes a video codec engine 306 for encoding, decoding, or transcoding media to, from, or between one or more media encoding formats, including but not limited to Moving Picture Experts Group (MPEG) formats (such as MPEG-2), High-Order Video Coding (AVC) formats (such as H.264 / MPEG-4 AVC, H.265 / HEVC, AOMedia VP8, VP9), and SMPTE 421M / VC-1 and Joint Picture Experts Group (JPEG) formats (such as JPEG and Motion Picture JPEG (MJPEG)).

[0056] In some embodiments, the graphics processor 300 includes a block image transfer (BLIT) engine 304 to perform two-dimensional (2D) rasterizer operations, including, for example, bit-boundary block transfer. However, in one embodiment, 2D graphics operations are performed using one or more components of a graphics processing engine (GPE) 310. In some embodiments, the GPE 310 is a computing engine for performing graphics operations, including three-dimensional (3D) graphics operations and media operations.

[0057] In some embodiments, GPE 310 includes a 3D pipeline 312 for performing 3D operations, such as rendering 3D images and scenes using processing functions acting on 3D primitive shapes (e.g., rectangles, triangles, etc.). The 3D pipeline 312 includes programmable and fixed functional elements that perform various tasks within the elements and / or generate threads to the 3D / media subsystem 315. While the 3D pipeline 312 can be used to perform media operations, embodiments of GPE 310 also include a media pipeline 316, which is specifically used to perform media operations such as video post-processing and image enhancement.

[0058] In some embodiments, the media pipeline 316 includes fixed-function or programmable logic units to perform one or more specialized media operations, such as video decoding acceleration, video de-interlacing, and video encoding acceleration (replacing or representing the video codec engine 306). In some embodiments, the media pipeline 316 additionally includes a thread generation unit to generate threads for execution on the 3D / media subsystem 315. The generated threads perform calculations for media operations on one or more graphics execution units included in the 3D / media subsystem 315.

[0059] In some embodiments, the 3D / media subsystem 315 includes logic for executing threads generated by the 3D pipeline 312 and the media pipeline 316. In one embodiment, the pipelines send thread execution requests to the 3D / media subsystem 315, including thread delivery logic for arbitrating and delivering various requests to available thread execution resources. Execution resources include an array of graphics execution units to process 3D and media threads. In some embodiments, the 3D / media subsystem 315 includes one or more internal caches for thread instructions and data. In some embodiments, the subsystem also includes shared memory (including registers and addressable memory) to share data between threads and store output data.

[0060] FIG. 3B shows a graphics processor 320 with a tiled architecture according to an embodiment described herein. In one embodiment, the graphics processor 320 includes a graphics processing engine cluster 322 having multiple instances of the graphics processing engine 310 of FIG. 3A within graphics engine blocks 310A-310D. Each graphics engine block 310A-310D may be interconnected via a set of block interconnects 323A-323F. Each graphics engine block 310A-310D may also be connected to a memory module or memory device 326A-326D via memory interconnects 325A-325D. The memory devices 326A-326D may use any graphics memory technology. For example, the memory devices 326A-326D may be graphics double data rate (GDDR) memory. In one embodiment, the memory devices 326A-326D are high bandwidth memory (HBM) modules that may be on-die with their respective graphics engine blocks 310A-310D. In one embodiment, memory devices 326A-326D may be stacked memory devices stacked on top of their respective graphics engine blocks 310A-310D. In one embodiment, each graphics engine block 310A-310D and its associated memory 326A-326D reside on an individual chiplet, which is bonded to a base die or base substrate, as further described in detail in Figures 11B-11D.

[0061] The graphics processing engine cluster 322 can be connected to on-chip or package-based interconnects 324. The interconnects 324 enable communication between the graphics engine blocks 310A-310D and components such as the video codec 306 and one or more copy engines 304. The copy engines 304 can be used to move data in and out of memory devices 326A-326D and between memory devices 326A-326D and memory external to the graphics processor 320 (e.g., system memory). The interconnects 324 can also be used to interconnect the graphics engine blocks 310A-310D. The graphics processor 320 may optionally include a display controller 302 to enable connectivity with an external display device 318. The graphics processor can also be configured as a graphics or computing accelerator. In an accelerator configuration, the display controller 302 and the display device 318 can be omitted.

[0062] The graphics processor 320 can be connected to the host system via a host interface 328. The host interface 328 can enable communication between the graphics processor 320, system memory, and / or other system components. The host interface 328 can be, for example, a PCI express bus or another type of host system interface.

[0063] FIG. 3C shows a computation accelerator 330 according to an embodiment described herein. The computation accelerator 330 may include an architecture similar to the graphics processor 320 of FIG. 3B and is optimized for computation acceleration. The computation engine cluster 332 may include a set of computation engine blocks 340A-340D, which contain execution logic optimized for parallel or vector-based general computational operations. In some embodiments, the computation engine blocks 340A-340D do not include fixed-function graphics processing logic, although in one embodiment, one or more computation engine blocks 340A-340D may include logic for performing media acceleration. The computation engine blocks 340A-340D may be connected to memory 326A-326D via memory interconnects 325A-325D. The technology of memory 326A-326D and memory interconnects 325A-325D may be similar to or different from that of graphics processor 320. The graphics processing engine blocks 340A-340D can also be interconnected via a set of block interconnects 323A-323F, and can be connected to and / or interconnected via the structural interconnect 324. In one embodiment, the computing accelerator 330 includes a large L3 cache 336, which can be configured as a device-wide cache. The computing accelerator 330 can also be connected to the host processor and memory via the host interface 328 in a manner similar to that of the graphics processor 320 of FIG. 3B. Graphics Processing Engine

[0064] FIG4 is a block diagram of a graphics processing engine 410 of a graphics processor according to some embodiments. In one embodiment, the graphics processing engine (GPE) 410 is a version of the GPE 310 shown in FIG3A, and may also represent graphics engine blocks 310A-310D of FIG3B. Elements in FIG4 having the same element symbols (or names) as elements in any other figures herein are capable of operating or functioning in any manner similar to, but not limited to, those described in other parts of this document. For example, the 3D pipeline 312 and media pipeline 316 of FIG3A are shown. The media pipeline 316 is optional in some embodiments of the GPE 410 and may not be explicitly included in the GPE 410. For example, and in at least one embodiment, a separate media and / or image processor is coupled to the GPE 410.

[0065] In some embodiments, GPE 410 is coupled to or includes command streamer 403, which provides command streams to 3D pipeline 312 and / or media pipeline 316. In some embodiments, command streamer 403 is coupled to memory, which may be system memory or one or more of internal cache memory and shared cache memory. In some embodiments, command streamer 403 receives commands from memory and sends the commands to 3D pipeline 312 and / or media pipeline 316. Commands are instructions obtained from a ring buffer that stores commands for 3D pipeline 312 and media pipeline 316. In one embodiment, the ring buffer may further include a batch command buffer that stores multiple commands in batches. Commands for 3D pipeline 312 may also include references to data stored in memory, such as, but not limited to, vertex and geometry data of 3D pipeline 312 and / or image data and memory objects of media pipeline 316. The 3D pipeline 312 and media pipeline 316 process commands and data by performing operations via logic in their respective pipelines or by dispatching one or more execution threads to the graphics core array 414. In one embodiment, the graphics core array 414 includes one or more graphics core blocks (e.g., one or more graphics cores 415A, one or more graphics cores 415B), each block including one or more graphics cores. Each graphics core includes a set of graphics execution resources, including general and graphics-specific execution logic for performing graphics and computational operations, as well as fixed-function texture processing and / or machine learning and artificial intelligence acceleration logic.

[0066] In various embodiments, the 3D pipeline 312 includes fixed functions and programmable logic to process one or more shader programs, such as vertex shaders, geometry shaders, pixel shaders, fragment shaders, compute shaders, or other shader programs, by processing instructions and dispatching execution threads to the graphics core array 414. The graphics core array 414 provides a unified execution resource block for use in processing these shader programs. The multipurpose execution logic (e.g., execution units) within the one or more graphics cores 415A-415B of the graphics core array 414 includes support for various 3D API shader languages ​​and is capable of running multiple concurrently executing threads associated with multiple shaders.

[0067] In some embodiments, the graphics core array 414 includes execution logic to perform media functions, such as video and / or image processing. In one embodiment, the execution unit includes general-purpose logic that is programmable to perform parallel general-purpose computing operations in addition to graphics processing operations. The general-purpose logic is capable of performing processing operations in parallel with or in combination with the general-purpose logic within the processor cores 107 of FIG1 or within cores 202A-202N of FIG2A.

[0068] Output data generated by the threads running on the graphics core array 414 can be output to memory in a unified return buffer (URB) 418. The URB 418 can store data for multiple threads. In some embodiments, the URB 418 can be used to send data between different threads running on the graphics core array 414. In some embodiments, the URB 418 can also be used for synchronization between threads on the graphics core array and the fixed-function logic within the shared-function logic 420.

[0069] In some embodiments, the graphics core array 414 is scalable, such that the array includes a variable number of graphics cores, each having a variable number of execution units based on the target power and performance level of the GPE 410. In one embodiment, the execution resources are dynamically scalable, such that the execution resources can be enabled or disabled as needed.

[0070] The graphics core array 414 is coupled to shared function logic 420, which includes multiple resources shared among the graphics cores in the graphics core array. The shared functions within the shared function logic 420 are hardware logic units that provide dedicated supplemental functionality to the graphics core array 414. In various embodiments, the shared function logic 420 includes, but is not limited to, a sampler 421, mathematical logic 422, and inter-thread communication (ITC) logic 423. Additionally, some embodiments implement one or more caches 425 within the shared function logic 420.

[0071] A shared function is implemented at least when the demand for a given dedicated function is insufficient to be contained within the graphics core array 414. Alternatively, a single instance of the dedicated function is implemented as a separate entity in the shared function logic 420 and shared among execution resources within the graphics core array 414. The exact set of functions shared between and contained within the graphics core array 414 varies across embodiments. In some embodiments, a particular shared function widely used by the graphics core array 414 within the shared function logic 420 may be contained within the shared function logic 416 within the graphics core array 414. In various embodiments, the shared function logic 416 within the graphics core array 414 may include some or all of the logic within the shared function logic 420. In one embodiment, all logic elements within the shared function logic 420 may be replicated within the shared function logic 416 of the graphics core array 414. In one embodiment, the shared function logic 420 is not included for the sake of the shared function logic 416 within the graphics core array 414. Execution Unit

[0072] Figures 5A-5B show thread execution logic 500 including an array of processing elements employed in a graphics processor core according to the embodiments described herein. Elements in Figures 5A-5B having the same element symbols (or names) as elements in any other figures herein can operate or function in any manner similar to, but not limited to, those described in other parts of this document. Figures 5A-5B show an overview of thread execution logic 500, which may represent the hardware logic shown as each sub-core 221A-221F of Figure 2B. Figure 5A represents an execution unit within a general-purpose graphics processor, while Figure 5B represents an execution unit that can be used within a computing accelerator.

[0073] As shown in FIG5A, in some embodiments, the thread execution logic 500 includes a shader processor 502, a thread dispatcher 504, an instruction cache 506, a scalable execution unit array including a plurality of execution units 508A-508N, a sampler 510, a shared region memory 511, a data cache 512, and a data port 514. In one embodiment, the scalable execution unit array is capable of dynamically scaling by enabling or disabling one or more execution units (e.g., any one of execution units 508A, 508B, 508C, 508D to 508N-1 and 508N) based on workload computing needs. In one embodiment, the included components are interconnected via an interconnect structure (linked to each of the components). In some embodiments, the thread execution logic 500 includes one or more connections to memory (such as system memory or cache memory) via one or more of the instruction cache 506, data port 514, sampler 510, and execution units 508A-508N. In some embodiments, each execution unit (e.g., 508A) is an independently programmable general-purpose computing unit capable of running multiple concurrent hardware threads, processing multiple data elements in parallel for each thread. In various embodiments, the array of execution units 508A-508N can be scaled to include any number of individual execution units.

[0074] In some embodiments, execution units 508A-508N are primarily used to run shader programs. Shader processor 502 is capable of handling various shader programs and dispatching execution threads associated with shader programs via thread dispatcher 504. In one embodiment, the thread dispatcher includes logic for arbitrating thread initiation requests from the graphics and media pipeline and instantiating the requested thread on one or more execution units in execution units 508A-508N. For example, the geometry pipeline can dispatch vertex, tessellation, or geometry shaders to the thread execution logic for processing. In some embodiments, thread dispatcher 504 is also capable of handling runtime thread generation requests from running shader programs.

[0075] In some embodiments, execution units 508A-508N support instruction sets that include native support for many standard 3D graphics shader instructions, enabling shader programs from graphics libraries (e.g., Direct3D and OpenGL) to run with minimal transformation. The execution units support vertex and geometry processing (e.g., vertex programs, geometry programs, vertex shaders), pixel processing (e.g., pixel shaders, fragment shaders), and general processing (e.g., computation and media shaders). Each execution unit 508A-508N is capable of single instruction multiple data (SIMD) execution, and multiple thread operations provide an efficient execution environment in the face of higher time-delayed memory access. Each hardware thread in each execution unit has a dedicated high-bandwidth register file and associated independent thread state. Execution is clock-to-pipeline multiple issuance, enabling integer, single and double-precision floating-point operations, SIMD branching capabilities, logical operations, transcendental operations, and other miscellaneous operations. While waiting for data from either memory or a shared function, the dependency logic within execution units 508A-508N causes the waiting thread to sleep until the requested data has been returned. While the waiting thread is sleeping, hardware resources may be dedicated to processing other threads. For example, during a delay associated with vertex shader operations, the execution unit may execute operations of a pixel shader, fragment shader, or another type of shader program that includes different vertex shaders. Various embodiments can be adapted to execute using Single Instruction Multiple Threads (SIMT), as an alternative to SIMD or in addition to SIMD. Reference to the SIMD core or operations can also be applied to SIMD or a combination of SIMD and SIMD.

[0076] Each execution unit in the execution units 508A-508N operates on an array of data elements. The number of data elements is the "execution size," or the number of channels used for instructions. An execution channel is a logic unit used for data element access, masking, and flow control within instructions. The number of channels may be independent of the number of physical arithmetic logic units (ALUs) or floating-point units (FPUs) for a particular graphics processor. In some embodiments, the execution units 508A-508N support integer and floating-point data types.

[0077] The execution unit instruction set includes SIMD instructions. Various data elements can be stored as compressed data types in registers, and the execution unit will process various elements based on the data size of the elements. For example, when operating on a 256-bit wide vector, the 256-bit vector is stored in a register, and the execution unit operates on a vector of four independent 54-bit compressed data elements (Quad-Word (QW) size data elements), eight independent 32-bit compressed data elements (Double Word (DW) size data elements), sixteen independent 16-bit compressed data elements (Word (W) size data elements), or thirty-two independent 8-bit data elements (byte (B) size data elements). However, different vector widths and register sizes are possible.

[0078] In one embodiment, one or more execution units can be combined into fused execution units 509A-509N, which have common thread control logic (507A-507N) shared by the fused EUs. Multiple EUs can be fused into an EU group. Each EU in the fused EU group can be configured to run an independent SIMD hardware thread. The number of EUs in the fused EU group can be varied according to embodiments. Additionally, various SIMD widths can be implemented per EU, including but not limited to SIMD8, SIMD16, and SIMD32. Each fused graphics execution unit 509A-509N includes at least two execution units. For example, fused execution unit 509A includes a first EU 508A, a second EU 508B, and thread control logic 507A, which is shared by the first EU 508A and the second EU 508B. Thread control logic 507A controls the threads running on the fused graphics execution unit 509A, thereby allowing each EU within the fused execution units 509A-509N to run using a common instruction index register.

[0079] One or more internal instruction caches (e.g., 506) are included in the thread execution logic 500 to cache the thread instructions of the execution unit. In some embodiments, one or more data caches (e.g., 512) are included to cache thread data during thread execution. Threads executing on the execution logic 500 may also store explicitly managed data in shared area memory 511. In some embodiments, a sampler 510 is included to provide texture sampling for 3D operations and media sampling for media operations. In some embodiments, the sampler 510 includes dedicated texture or media sampling functionality to process texture or media data during a sampling process prior to providing sampled data to the execution unit.

[0080] During execution, the graphics and media pipeline sends a thread initiation request to the thread execution logic 500 via the thread generation and dispatch logic. Once a set of geometric objects has been processed and rasterized into pixel data, the pixel processor logic (e.g., pixel shader logic, fragment shader logic, etc.) in the shader processor 502 is invoked to further compute output information and write the results to an output surface (e.g., color buffer, depth buffer, stencil buffer, etc.). In some embodiments, the pixel shader or fragment shader computes values ​​of various vertex attributes, which are interpolated across the rasterized objects. In some embodiments, the pixel processor logic in the shader processor 502 runs a pixel or fragment shader program provided by an application programming interface (API). To run the shader program, the shader processor 502 dispatches a thread to an execution unit (e.g., 508A) via the thread dispatcher 504. In some embodiments, the shader processor 502 uses texture sampling logic in the sampler 510 to access texture data in a texture map stored in memory. Arithmetic operations are performed on the texture data and input geometry data to calculate the pixel color data of each geometric segment, or to discard one or more pixels to avoid further processing.

[0081] In some embodiments, data port 514 provides a memory access mechanism to enable thread execution logic 500 to output processed data to memory for further processing on the graphics processor output pipeline. In some embodiments, data port 514 includes or is coupled to one or more cache memories (e.g., data cache 512) to cache data for memory access via the data port.

[0082] In one embodiment, the execution logic 500 may further include a ray tracer 505 that can provide ray tracing acceleration capabilities. The ray tracer 505 may support a ray tracing instruction set that includes instructions / functions for ray generation. The ray tracing instruction set may be similar to or different from the ray tracing instruction set supported by the ray tracing core 245 in FIG2C.

[0083] Figure 5B shows exemplary internal details of the execution unit 508 according to an embodiment. The graphics execution unit 508 may include an instruction fetching unit 537, a general register file array (GRF) 524, an architecture register file array (ARF) 526, a thread arbiter 522, a sending unit 530, a branching unit 532, a set of SIMD floating-point units (FPU) 534, and a set of dedicated integer SIMD ALUs 535 in one embodiment. The GRF 524 and ARF 526 include sets of general register files and architecture register files associated with each concurrent hardware thread (which may be active in the graphics execution unit 508). In one embodiment, the architecture state of each thread is maintained in the ARF 526, while the data used during thread execution is stored in the GRF 524. The execution state of each thread (including the instruction index of each thread) can be stored in a thread-specific register in the ARF 526.

[0084] In one embodiment, the graphics execution unit 508 has an architecture that is a combination of simultaneous multithreading (SMT) and fine-grained interleaved multithreading (IMT). The architecture has a modular configuration that can be fine-tuned at design time based on a target number of simultaneous threads and the number of registers per execution unit, wherein execution unit resources are partitioned across logic used to run multiple simultaneous threads. The number of thread logics that can be executed by the graphics execution unit 508 is not limited to the number of hardware threads, and multiple logical threads can be assigned to each hardware thread.

[0085] In one embodiment, the graphics execution unit 508 can issue multiple instructions, each of which can be different instructions. The thread arbiter 522 of the graphics execution unit thread 508 can dispatch instructions to one of the sending unit 530, the branching unit 532, or one or more SIMD FPUs 534 for execution. Each execution thread can access 128 general-purpose registers within the GRF 524, where each register can store 32 bytes accessible as a SIMD 8 element vector of 32-bit data elements. In one embodiment, each execution unit thread has access to 4 Kbytes within the GRF 524, but the embodiment is not limited to this, and more or fewer register resources may be provided in other embodiments. In one embodiment, the graphics execution unit 508 is divided into seven hardware threads that can independently perform computational operations, but the number of threads per execution unit may vary depending on the embodiment. For example, in one embodiment, up to 16 hardware threads are supported. In an embodiment where seven threads can access 4 Kbytes, the GRF 524 can store a total of 28 Kbytes. With 16 threads each capable of accessing 4 Kbytes, the GRF 524 can store a total of 64 Kbytes. The flexible addressing modes allow registers to be co-addressed to efficiently construct wider registers or represent strided rectangular block data structures.

[0086] In one embodiment, memory operations, sampler operations, and other long-delayed system communications are dispatched via "send" instructions (run by message sending unit 530). In one embodiment, branch instructions are dispatched to dedicated branch unit 532 to facilitate SIMD divergence and eventual convergence.

[0087] In one embodiment, the graphics execution unit 508 includes one or more SIMD floating-point units (FPUs) 534 to perform floating-point operations. In one embodiment, the (one or more) FPUs 534 also support integer computation. In one embodiment, the (one or more) FPUs 534 are capable of performing up to M 32-bit floating-point (or integer) operations in SIMD, or up to 2M 16-bit integer or 16-bit floating-point operations in SIMD. In one embodiment, at least one of the (one or more) FPUs provides extended mathematical capabilities to support high-yield beyond mathematical functions and double-precision 54-bit floating-point operations. In some embodiments, a set of 8-bit integer SIMD ALUs 535 is also present and may be specifically optimized to perform operations associated with machine learning computations.

[0088] In one embodiment, an array of multiple instances of the graphics execution unit 508 can be exemplified in a graphics sub-core group (e.g., a sub-slice). For scalability, the product architect can select the exact number of execution units per sub-core group. In one embodiment, the execution unit 508 can run instructions across multiple execution channels. In another embodiment, each thread running on the graphics execution unit 508 runs on a different channel.

[0089] FIG6 shows an additional execution unit 600 according to an embodiment. Execution unit 600 may be, but is not limited to, an execution unit used for computation optimization in computing engine blocks 340A-340D as shown in FIG3C. Variations of execution unit 600 may also be used in graphics engine blocks 310A-310D as shown in FIG3B. In one embodiment, execution unit 600 includes a thread control unit 601, a thread status unit 602, an instruction fetch / prefetch unit 603, and an instruction decoding unit 604. Execution unit 600 also includes a register file 606, which stores registers that can be allocated to hardware threads within the execution unit. Execution unit 600 also includes a sending unit 607 and a branching unit 608. In one embodiment, sending unit 607 and branching unit 608 may operate similarly to sending unit 530 and branching unit 532 of graphics execution unit 508 of FIG5B.

[0090] The execution unit 600 further includes a computing unit 610 containing multiple functional units of different types. In one embodiment, the computing unit 610 includes an ALU unit 611 containing an array of arithmetic logic units. The ALU unit 611 can be configured to perform 64-bit, 32-bit, and 16-bit integer and floating-point operations. Integer and floating-point operations can be performed simultaneously. The computing unit 610 may also include a systolic array 612 and a math unit 613. The systolic array 612 includes a W-width and D-depth network of data processing units that can be used to perform vector or other data parallel operations in a systolic manner. In one embodiment, the systolic array 612 can be configured to perform matrix operations, such as matrix dot product operations. In one embodiment, the systolic array 612 supports 16-bit floating-point operations, as well as 8-bit and 4-bit integer operations. In one embodiment, the systolic array 612 can be configured to accelerate machine learning operations. In this embodiment, the systolic array 612 can be configured to support the bfloat 16-bit floating-point format. In one embodiment, a math unit 613 may be included to perform a specific subset of mathematical operations in a more efficient and lower-power-consumption manner than the ALU unit 611. The math unit 613 may include variations of mathematical logic that can be found in the shared functional logic of a graphics processing engine provided by other embodiments (e.g., the math logic 422 of shared functional logic 420 of FIG4). In one embodiment, the math unit 613 may be configured to perform 32-bit and 64-bit floating-point operations.

[0091] The thread control unit 601 includes logic for controlling the execution of threads within the execution unit. The thread control unit 601 may include thread arbitration logic for starting, stopping, and preemptively executing threads within the execution unit 600. The thread status unit 602 may be used to store the thread status of threads allocated for execution on the execution unit 600. When a thread is blocked or idle, the thread status is stored in the execution unit 600 to enable rapid thread preemption. The instruction fetch / prefetch unit 603 may fetch instructions from the instruction cache of higher-order execution logic (e.g., instruction cache 506 in FIG. 5A). The instruction fetch / prefetch unit 603 may also issue a prefetch request for instructions to be loaded into the instruction cache based on analysis of the currently executing thread. The instruction decoding unit 604 may be used to decode instructions to be executed by the computing unit. In one embodiment, the instruction decoding unit 604 may be used as an auxiliary decoder to decode complex instructions into constituent micro-operations.

[0092] The execution unit 600 also includes a register file 606, which can be used by hardware threads executing on the execution unit 600. Registers in the register file 606 can be logically partitioned among multiple concurrent threads within the computing unit 610 used to execute the execution unit 600. The number of logical threads that can be executed by the graphics execution unit 600 is not limited to the number of hardware threads, and multiple logical threads can be assigned to each hardware thread. The size of the register file 606 can vary in different embodiments based on the number of supported hardware threads. In one embodiment, register renaming can be used to dynamically allocate registers to hardware threads.

[0093] FIG7 is a block diagram illustrating a graphics processor instruction format 700 according to some embodiments. In one or more embodiments, the graphics processor execution unit supports an instruction set having multiple instruction formats. Solid boxes show components generally included in the execution unit instructions, while dashed lines include components that are optional or included only in a subset of the instructions. In some embodiments, the instruction format 700 described and shown is a macro instruction because it is an instruction provided to the execution unit, as opposed to the micro-operations generated from instruction decoding once the instruction is processed.

[0094] In some embodiments, the graphics processor execution unit natively supports instructions according to the 128-bit instruction format 710. A 64-bit compressed instruction format 730 may be available for some instructions based on the selected instruction, instruction options, and the number of operands. The native 128-bit instruction format 710 provides access to all instruction options, while some options and operations are limited to the 64-bit format 730. The native instructions available according to the 64-bit format 730 vary depending on the embodiment. In some embodiments, the set of index values ​​in the index field 713 is used in part to compress the instructions. The execution unit hardware references a set of compression tables based on the index values ​​and uses the output of the compression tables to reconstruct the native instructions employing the 128-bit instruction format 710. Instructions of other sizes and formats may be used.

[0095] For each format, the instruction opcode 712 defines the operation to be performed by the execution unit. The execution unit executes each instruction in parallel across multiple data elements of each operand. For example, in response to an addition instruction, the execution unit performs simultaneous addition across each color channel representing a texture element or image element. By default, the execution unit executes each instruction across all data channels of the operand. In some embodiments, the instruction control field 714 implements control over certain execution options such as channel selection (e.g., assertion) and data channel order (e.g., blending). For instructions using the 128-bit instruction format 710, the execution size field 716 limits the number of data channels that will be run in parallel. In some embodiments, the execution size field 716 is not available for the 64-bit compressed instruction format 730.

[0096] Some execution unit instructions have a total of three operands, including two source operands src0 720 and src1 722 and a destination 718. In some embodiments, the execution unit supports dual-destination instructions, where one of the destinations is implied. Data manipulation instructions can have a third source operand (e.g., SRC2 724), where the instruction operand 712 determines the number of source operands. The last source operand of the instruction can be an immediate (e.g., hard-coded) value passed with the instruction.

[0097] In some embodiments, the 128-bit instruction format 710 includes an access / address mode field 726, which specifies, for example, whether to use a direct register addressing mode or an indirect register addressing mode. When using the direct register addressing mode, the register addresses of one or more operands are provided directly through bits in the instruction.

[0098] In some embodiments, the 128-bit instruction format 710 includes an access / address mode field 726, which specifies the address mode and / or access mode of the instruction. In one embodiment, the access mode is used to define the data access alignment of the instruction. Some embodiments support access modes including 16-byte aligned access modes and 1-byte aligned access modes, wherein the byte alignment of the access mode determines the access alignment of the instruction operands. For example, when in a first mode, the instruction can use byte-aligned addressing for both source and destination operands, while when in a second mode, the instruction can use 16-byte aligned addressing for all source and destination operands.

[0099] In one embodiment, the address mode portion of the access / address mode field 726 determines whether the instruction uses direct or indirect addressing. When using direct register addressing mode, bits in the instruction directly provide the register addresses of one or more operands. When using indirect register addressing mode, the register addresses of one or more operands can be calculated based on the address register values ​​in the instruction and the address immediate field.

[0100] In some embodiments, instructions are grouped based on the 712-bit fields of the operand code to simplify operand code decoding 740. For 8-bit operand codes, bits 4, 5, and 6 allow the execution unit to determine the type of operand code. The exact operand code grouping shown is merely an example. In some embodiments, move and logic operand code grouping 742 includes data move and logic instructions (e.g., move (mov), compare (cmp)). In some embodiments, move and logic grouping 742 shares five most significant bits (MSB), where move (mov) instructions take the form 0000xxxxb, and logic instructions take the form 0001xxxxb. Flow control instruction grouping 744 (e.g., call, jump (jmp)) includes instructions taking the form 0010xxxxb (e.g., 0x20). Miscellaneous instruction grouping 746 includes a mixture of instructions, which includes synchronous instructions (e.g., wait, send) taking the form 0011xxxxb (e.g., 0x30). Parallel math instruction group 748 includes component-by-component arithmetic instructions (e.g., addition, multiplication (mul)) in the form of 0100xxxxb (e.g., 0x40). Parallel math group 748 performs arithmetic operations in parallel across data channels. Vector math group 750 includes arithmetic instructions (e.g., dp4) in the form of 0101xxxxb (e.g., 0x50). Vector math group performs arithmetic on vector operands, such as dot product calculations. In one embodiment, the shown arithmetic code decoder 740 can be used to determine which part of the execution unit will be used to execute the decoded instructions. For example, some instructions can be designated as systolic instructions to be executed by the systolic array. Other instructions, such as ray tracing instructions (not shown), can be routed to the ray tracing core or ray tracing logic within a slice or partition of the execution logic. Graphics Pipeline

[0101] FIG8 is a block diagram of another embodiment of the graphics processor 800. Elements in FIG8 having the same element symbols (or names) as elements in any other figures herein are capable of operating or functioning in any manner similar to those described in other parts of this document, but are not limited thereto.

[0102] In some embodiments, the graphics processor 800 includes a geometry pipeline 820, a media pipeline 830, a display engine 840, thread execution logic 850, and a rendering output pipeline 870. In some embodiments, the graphics processor 800 is a graphics processor in a multi-core processing system, which includes one or more general-purpose processing cores. The graphics processor is controlled by writing to registers of one or more control registers (not shown) or by commands issued to the graphics processor 800 via a ring interconnect 802. In some embodiments, the ring interconnect 802 couples the graphics processor 800 to other processing components (e.g., other graphics processors or general-purpose processors). Commands from the ring interconnect 802 are interpreted by a command streamer 803, which provides instructions to individual components of the geometry pipeline 820 or the media pipeline 830.

[0103] In some embodiments, command streamer 803 directs the operation of vertex extractor 805, which reads vertex data from memory and executes vertex processing commands provided by command streamer 803. In some embodiments, vertex extractor 805 provides vertex data to vertex shader 807, which performs coordinate space transformation and lighting operations on each vertex. In some embodiments, vertex extractor 805 and vertex shader 807 execute vertex processing instructions by dispatching execution threads to execution units 852A-852B via thread dispatcher 831.

[0104] In some embodiments, execution units 852A-852B are vector processor arrays having an instruction set for performing graphics and media operations. In some embodiments, execution units 852A-852B have an attached L1 cache 851, which is specific to each array or shared between arrays. The cache can be configured as a data cache, an instruction cache, or a single cache, which is divided into different partitions containing data and instructions.

[0105] In some embodiments, the geometry pipeline 820 includes tessellation components to perform hardware-accelerated tessellation of 3D objects. In some embodiments, a programmable shell shader 811 configures the tessellation operation. A programmable domain shader 817 provides back-end evaluation of the tessellation output. A tessell 813 operates in the direction of the shell shader 811 and includes dedicated logic to generate a detailed set of geometric objects based on a coarse geometry model (which is provided as input to the geometry pipeline 820). In some embodiments, tessellation components (e.g., shell shader 811, tessell 813, and domain shader 817) can be bypassed if tessellation is not used.

[0106] In some embodiments, the complete geometry object can be processed by the geometry shader 819 via one or more threads dispatched to execution units 852A-852B, or can be directly processed by the clipper 829. In some embodiments, the geometry shader operates on the entire geometry object rather than on vertices or vertex patches as in previous stages of the graphics pipeline. If tessellation is disabled, the geometry shader 819 receives input from the vertex shader 807. In some embodiments, the geometry shader 819 is geometry shader programmable to perform geometry tessellation when the tessellation unit is disabled.

[0107] Before rasterization, clipper 829 is capable of processing vertex data. Clipper 829 may be a fixed-function clipper or a programmable clipper with clipping and geometry shader capabilities. In some embodiments, the rasterizer and depth testing unit 873 in the draw output pipeline 870 dispatches pixel shaders to convert the geometry into a per-pixel representation. In some embodiments, pixel shader logic is included in thread execution logic 850. In some embodiments, the application is able to bypass the rasterizer and depth testing unit 873 and access unrasterized vertex data via stream output unit 823.

[0108] The graphics processor 800 has an interconnect bus, interconnect structure, or some other interconnect mechanism that allows data and messages to be transferred between the main components of the processor. In some embodiments, execution units 852A-852B and associated logic units (e.g., L1 cache 851, sampler 854, texture cache 858, etc.) are interconnected via data port 856 to perform memory access and communicate with the processor's rendering output pipeline components. In some embodiments, sampler 854, caches 851 and 858, and execution units 852A-852B each have independent memory access paths. In one embodiment, texture cache 858 can also be configured as a sampler cache.

[0109] In some embodiments, the rendering output pipeline 870 includes a rasterizer and a depth testing unit 873, which converts vertex-based objects into associated pixel-based representations. In some embodiments, the rasterizer logic includes a windower / mask unit to perform fixed-function triangulation and line rasterization. Associated rendering cache 878 and depth cache 879 are also available in some embodiments. Pixel manipulation unit 877 performs pixel-based operations on the data, but in some instances, pixel operations associated with 2D operations (e.g., using mixed bit-block image transfer) are performed by the 2D engine 841, or by the display controller 843 using an overlay display plane at display time. In some embodiments, a shared L3 cache 875 is available to all graphics components, thereby allowing data sharing without using main system memory.

[0110] In some embodiments, the graphics processor media pipeline 830 includes a media engine 837 and a video front-end 834. In some embodiments, the video front-end 834 receives pipeline commands from a command streamer 803. In some embodiments, the media pipeline 830 includes a separate command streamer. In some embodiments, the video front-end 834 processes media commands before sending them to the media engine 837. In some embodiments, the media engine 837 includes thread generation functionality to generate threads for dispatch to thread execution logic 850 via a thread dispatcher 831.

[0111] In some embodiments, the graphics processor 800 includes a display engine 840. In some embodiments, the display engine 840 is external to the processor 800 and coupled to the graphics processor via a ring interconnect 802 or some other interconnect bus or structure. In some embodiments, the display engine 840 includes a 2D engine 841 and a display controller 843. In some embodiments, the display engine 840 includes dedicated logic that can operate independently of the 3D pipeline. In some embodiments, the display controller 843 is coupled to a display device (not shown), which may be a system-integrated display device such as in a laptop computer or an external display device attached via a display device connector.

[0112] In some embodiments, the geometry pipeline 820 and the media pipeline 830 may be configured to perform operations based on multiple graphics and media staging interfaces, rather than being specific to any one application programming interface (API). In some embodiments, driver software for the graphics processor translates API calls specific to a particular graphics or media library into commands that can be processed by the graphics processor. In some embodiments, support is provided for OpenGL, OpenCL, and / or Vulkan graphics and computing APIs, all from the Khronos Group. In some embodiments, support may also be provided for the Direct3D library from Microsoft Corporation. In some embodiments, a combination of these libraries may be supported. Support may also be provided for the open-source computer vision library (OpenCV). Future APIs with compatible 3D pipelines will also be supported if a pipeline mapping from future APIs to the graphics processor's pipeline can be made. Graphics pipeline execution staging

[0113] FIG9A is a block diagram showing a graphics processor command format 900 according to some embodiments. FIG9B is a block diagram showing a graphics processor command sequence 910 according to an embodiment. Solid boxes in FIG9A show components generally included in graphics commands, while dashed lines include components that are optional or included only in a subset of graphics commands. The exemplary graphics processor command format 900 of FIG9A includes a data field identifying client 902, command operation code (operation code) 904, and command data 906. Suboperation code 905 and command size 908 are also included in some commands.

[0114] In some embodiments, client 902 specifies a client unit of a graphics device that processes command data. In some embodiments, a graphics processor command parser examines the client field of each command to adjust further processing of the command and routes the command data to the appropriate client unit. In some embodiments, the graphics processor client unit includes a memory interface unit, a rendering unit, a 2D unit, a 3D unit, and a media unit. Each client unit has a corresponding processing pipeline for processing commands. Once a command is received by a client unit, the client unit reads the operation code 904 and the sub-operation code 905 (if present) to determine the operation to be performed. The client unit uses information in the data field 906 to execute the command. For some commands, the command size 908 is expected to explicitly specify the size of the command. In some embodiments, the command parser automatically determines the size of at least a portion of the command based on the command operation code. In some embodiments, commands are aligned via multiples of double words, and other command formats may be used.

[0115] Figure 9B shows an exemplary graphics processor command sequence 910. In some embodiments, software or firmware of a data processing system featuring an embodiment of the graphics processor uses a version of the shown command sequence to establish, execute, and terminate a set of graphics operations. Sample command sequences are shown and described for exemplary purposes only, as embodiments are not limited to these specific commands or this command sequence. Furthermore, the commands may be issued as a batch of commands in a command sequence, such that the graphics processor will process the sequence of commands at least partially simultaneously.

[0116] In some embodiments, the graphics processor command sequence 910 may begin with a pipeline clearing command 912 to allow any active graphics pipeline to complete its current pending commands. In some embodiments, the 3D pipeline 922 and the media pipeline 924 do not operate in parallel. Pipeline clearing is performed to allow any pending commands in the active graphics pipeline to complete. In response to pipeline clearing, the graphics processor's command parser will stop command processing until the active graphics engine completes its pending operations and invalidates the relevant read cache. Optionally, any data labeled 'dirty' in the draw cache can be cleared to memory. In some embodiments, the pipeline clearing command 912 can be used for pipeline synchronization or before putting the graphics processor into a low-power state.

[0117] In some embodiments, a pipeline selection command 913 is used when a sequence of commands requires the graphics processor to explicitly switch between pipelines. In some embodiments, the pipeline selection command 913 is requested only once within the execution context prior to issuing a pipeline command, unless the context is issuing commands for both pipelines. In some embodiments, a pipeline clearing command 912 is requested just before the pipeline is switched via the pipeline selection command 913.

[0118] In some embodiments, pipeline control command 914 configures the graphics pipeline for operation and is used to program the 3D pipeline 922 and the media pipeline 924. In some embodiments, pipeline control command 914 configures the pipeline state of the active pipeline. In one embodiment, pipeline control command 914 is used for pipeline synchronization and to clear data from one or more caches within the active pipeline before processing a batch of commands.

[0119] In some embodiments, the return buffer state command 916 is used to configure a set of return buffers to write data to a corresponding pipeline. Some pipeline operations require the allocation, selection, or configuration of one or more return buffers (in which intermediate data is written during processing). In some embodiments, the graphics processor also uses one or more return buffers to store output data and perform intersecting thread communication. In some embodiments, the return buffer state 916 includes selecting the size and number of return buffers for the pipeline operation set.

[0120] The remaining commands in the command sequence differ based on the active pipeline used for the operation. Based on pipeline determination 920, the command sequence is suitable for either 3D pipeline 922 starting at 3D pipeline state 930 or media pipeline 924 starting at media pipeline state 940.

[0121] Commands configuring 3D pipeline states 930 include 3D state setting commands for vertex buffer states, vertex element states, constant color states, depth buffer states, and other state variables configured prior to processing 3D primitive commands. The values ​​of these commands are determined at least in part based on the specific 3D API in use. In some embodiments, 3D pipeline state 930 commands can also selectively disable or bypass certain pipeline elements when they will not be used.

[0122] In some embodiments, the 3D primitive 932 command is used to submit 3D primitives to be processed by the 3D pipeline. The command and associated parameters passed to the graphics processor via the 3D primitive 932 command are forwarded to the vertex acquisition function in the graphics pipeline. The vertex acquisition function uses the 3D primitive 932 command data to generate a vertex data structure. The vertex data structure is stored in one or more return buffers. In some embodiments, the 3D primitive 932 command is used to perform vertex operations on the 3D primitives via a vertex shader. To process the vertex shader, the 3D pipeline 922 dispatches a shader execution thread to the graphics processor execution unit.

[0123] In some embodiments, the 3D pipeline 922 is triggered via a run command 934 or an event. In some embodiments, a register is written to trigger command execution. In some embodiments, execution is triggered via a "go" or "kick" command in a command sequence. In one embodiment, a pipeline synchronization command is used to trigger command execution so that the command sequence is cleared through the graphics pipeline. The 3D pipeline performs geometry processing on 3D primitives. Once the operation is complete, the resulting geometry is rasterized, and the pixel engine shades the resulting pixels. Additional commands controlling pixel shading and pixel backend operations may also be included for those operations.

[0124] In some embodiments, the graphics processor command sequence 910 follows the media pipeline 924 pathway when performing media operations. Generally, the specific use and manner of programming the media pipeline 924 depends on the media or computational operation to be performed. Specific media decoding operations may be offloaded to the media pipeline during media decoding. In some embodiments, it is also possible to bypass the media pipeline, and media decoding can be performed wholly or partially using the resources provided by one or more general-purpose processing cores. In one embodiment, the media pipeline also includes elements for general-purpose graphics processing unit (GPGPU) operations, wherein the graphics processor is used to perform SIMD vector operations using computational shader programs (which are not explicitly related to the drawing of graphics primitives).

[0125] In some embodiments, the media pipeline 924 is configured in a manner similar to that of the 3D pipeline 922. The set of commands configuring the media pipeline state 940 is dispatched or placed in a command queue prior to the media object command 942. In some embodiments, the commands for the media pipeline state 940 include information configuring media pipeline elements (which will be used to process media objects). This includes information configuring video decoding and video encoding logic in the media pipeline, such as encoding or decoding formats. In some embodiments, the commands for the media pipeline state 940 also support the use of one or more indicators that include a set of state settings for "indirect" state elements.

[0126] In some embodiments, media object command 942 provides indicators to media objects for media pipeline processing. Media objects include memory buffers containing video data to be processed. In some embodiments, all media pipeline states must be valid before issuing media object command 942. Once the pipeline states are configured and media object command 942 is queued, media pipeline 924 is triggered via run command 944 or an equivalent run event (e.g., register write). Output from media pipeline 924 can then be post-processed through operations provided by 3D pipeline 922 or media pipeline 924. In some embodiments, GPGPU operations are configured and run in a manner similar to media operations. Graphics Software Architecture

[0127] Figure 10 shows an exemplary graphics software architecture for a data processing system 1000 according to some embodiments. In some embodiments, the software architecture includes a 3D graphics application 1010, an operating system 1020, and at least one processor 1030. In some embodiments, the processor 1030 includes a graphics processor 1032 and one or more general-purpose processor cores 1034. The graphics application 1010 and the operating system 1020 each run in the system memory 1050 of the data processing system.

[0128] In some embodiments, the 3D graphics application 1010 includes one or more shader programs, which include shader instructions 1012. The shader language instructions may be in a high-level shader language, such as the High-Level Shading Language (HLSL) of Direct3D, the OpenGL Shading Language (GLSL), etc. The application also includes executable instructions 1014 in machine language suitable for execution by a general-purpose processor core 1034. The application also includes graphics objects 1016 defined by vertex data.

[0129] In some embodiments, the operating system 1020 is a Microsoft® Windows® operating system, a proprietary Unix-like operating system, or an open-source Unix-like operating system using a variant of the Linux kernel from Microsoft Corporation. The operating system 1020 supports graphics APIs 1022, such as the Direct3D API, OpenGL API, or Vulkan API. When the Direct3D API is used, the operating system 1020 uses a front-end shader compiler 1024 to compile any shader instructions 1012 of HLSL into a low-level shader language. Compilation can be just-in-time (JIT) compilation, or the application can perform shader pre-compilation. In some embodiments, high-level shaders are compiled into low-level shaders during the compilation of the 3D graphics application 1010. In some embodiments, shader instructions 1012 are provided in an intermediate form (e.g., a version of the Standard Portable Intermediate Representation (SPIR) used by the Vulkan API).

[0130] In some embodiments, the user-mode graphics driver 1026 includes a back-end shader compiler 1027 to translate shader instructions 1012 into a hardware-specific representation. When the OpenGL API is used, shader instructions 1012 in the GLSL high-level language are passed to the user-mode graphics driver 1026 for compilation. In some embodiments, the user-mode graphics driver 1026 communicates with the core-mode graphics driver 1029 using the operating system kernel-mode functionality 1028. In some embodiments, the kernel-mode graphics driver 1029 communicates with the graphics processor 1032 to dispatch commands and instructions. IP Core Implementation

[0131] One or more embodiments of at least one example can be implemented via representative code stored on a machine-readable medium, which represents and / or defines logic in an integrated circuit (e.g., a processor). For example, the machine-readable medium may include instructions representing various logics in a processor. When read by a machine, the instructions can cause the machine to create logic that performs the techniques described herein. Such representations (referred to as "IP cores") are reusable units of logic in an integrated circuit that can be stored on a tangible machine-readable medium as a hardware model describing the structure of the integrated circuit. The hardware model can be provided to various customers or manufacturing organizations that load the hardware model onto manufacturing machines that produce integrated circuits. Integrated circuits can be manufactured such that the circuits perform the operations described in any of the embodiments described herein.

[0132] Figure 11A is a block diagram showing an IP core development system 1100, which can be used to manufacture integrated circuits that perform operations according to an embodiment. The IP core development system 1100 can be used to generate modular, reusable designs that can be incorporated into larger designs or used to form an entire integrated circuit (e.g., a SOC integrated circuit). The design unit 1130 can use a high-level programming language (e.g., C / C++) to generate a software simulation 1110 of the IP core design. The software simulation 1110 can be used to design, test, and verify the behavior of the IP core using a simulation model 1112. The simulation model 1112 can include functional, behavioral, and / or timing simulations. A register transfer level (RTL) design 1115 can then be built or synthesized from the simulation model 1112. The RTL design 1115 is an abstraction of the behavior of the integrated circuit, which modularizes the flow of digital signals between hardware registers, including the associated logic performed using the modularized digital signals. In addition to RTL design 1115, low-order designs at the logic or transistor level can be built, designed, or synthesized. Therefore, the specific details of the initial design and simulation can be changed.

[0133] The RTL design 1115 or its equivalent can be further synthesized by the design agency into a hardware model 1120 (which may be expressed in a Hardware Description Language (HDL) or some other representation of physical design data). The HDL can also be simulated or tested to verify the IP core design. The IP core design can be stored for transfer to a third-party manufacturing agency 1165 using non-volatile memory 1140 (e.g., a hard drive, flash memory, or any non-volatile storage medium). Alternatively, the IP core design can be transmitted via a wired connection 1150 or a wireless connection 1160 (e.g., via the Internet). The manufacturing agency 1165 then manufactures an integrated circuit at least partially based on the IP core design. The manufactured integrated circuit can be configured to perform operations according to at least one embodiment described herein.

[0134] FIG11B shows a cross-sectional side view of an integrated circuit package assembly 1170 according to some embodiments described herein. The integrated circuit package assembly 1170 shows an implementation of one or more processor or accelerator devices as described herein. The package assembly 1170 includes a plurality of cells of hardware logic 1172, 1174 connected to a substrate 1180. The logic 1172, 1174 may be implemented at least in part using configurable logic or fixed-function logic hardware, and may include one or more portions of any of a processor core, a graphics processor, or other accelerator devices described herein. Each cell of the logic 1172, 1174 can be implemented within a semiconductor die and coupled to the substrate 1180 via an interconnect structure 1173. The interconnect structure 1173 may be configured to route electrical signals between the logic 1172, 1174 and the substrate 1180, and may include interconnects such as, but not limited to, bumps or pillars. In some embodiments, interconnect structure 1173 may be configured to route electrical signals associated with the operation of logic 1172, 1174, such as, for example, input / output (I / O) signals and / or power or ground signals. In some embodiments, substrate 1180 is an epoxy-based laminate substrate. In other embodiments, substrate 1180 may include other suitable types of substrates. Package assembly 1170 is capable of being connected to other electronic devices via package interconnect 1183. Package interconnect 1183 may be coupled to the surface of substrate 1180 to route electrical signals to other electronic devices, such as motherboards, another chipset, or multi-chip module.

[0135] In some embodiments, the cells of logic 1172, 1174 are electrically coupled to bridge 1182, which is configured to route electrical signals between logic 1172, 1174. Bridge 1182 may be a dense interconnect structure that provides routing of electrical signals. Bridge 1182 may include a bridge substrate made of glass or a suitable semiconductor material. Electrical wiring features can be formed on the bridge substrate to provide chip-to-chip connectivity between logic 1172, 1174.

[0136] Although two units of logic 1172 and 1174 and bridge 1182 are shown, the embodiments described herein may include more or fewer logic units on one or more dies. One or more dies may be connected via zero or more bridges, because bridge 1182 may not be included when the logic is contained on a single die. Alternatively, multiple dies or units of logic can be connected via one or more bridges. Furthermore, multiple logic units, dies, and bridges can be interconnected according to other possible configurations, including three-dimensional configurations.

[0137] Figure 11C shows a package assembly 1190 comprising multiple units (e.g., base dies) of hardware logic wafers connected to a substrate 1180. Graphics processing units, parallel processors, and / or computing accelerators, as described herein, can be composed of various silicon wafers manufactured separately. In this context, a wafer is at least partially packaged integrated circuit containing different units of logic that can be assembled with other wafers into a larger package. Various wafers with different IP core logic can be assembled into a single device. Furthermore, active interposer technology can be used to integrate wafers into base dies or base wafers. The concepts described herein enable interconnection and communication between different forms of IP within a GPU. IP cores can be manufactured using different process technologies and combined during the manufacturing process, thus avoiding the complexity of integrating multiple IPs (especially on large SoCs with multiple IPs) into the same manufacturing process. The use of multiple process technologies can shorten time-to-market and provide a cost-effective way to build multiple product SKUs. Furthermore, the decomposed IP is better suited for independent power control, allowing unused components to be powered off under a given workload, thereby reducing overall power consumption.

[0138] The hardware logic chiplet may include a dedicated hardware logic chiplet 1172, a logic or I / O chiplet 1174, and / or a memory chiplet 1175. The hardware logic chiplet 1172 and the logic or I / O chiplet 1174 may be implemented at least partially as configurable logic or fixed-function logic hardware, and may include one or more portions of any of the processor core, graphics processor, parallel processor, or other accelerator device described herein. The memory chiplet 1175 may be DRAM (e.g., GDDR, HBM) memory or SRAM memory.

[0139] Each chiplet can be fabricated as an individual semiconductor die and coupled to the substrate 1180 via an interconnect structure 1173. The interconnect structure 1173 can be configured to route electrical signals between the various chiplets and the logic within the substrate 1180. The interconnect structure 1173 may include interconnects, such as, but not limited to, bumps or pillars. In some embodiments, the interconnect structure 1173 can be configured to route electrical signals, such as, for example, input / output (I / O) signals and / or power or ground signals associated with the operation of logic, I / O, and memory chiplets.

[0140] In some embodiments, substrate 1180 is an epoxy-based laminate. In other embodiments, substrate 1180 may comprise other suitable types of substrates. Package assembly 1190 may be connected to other electronic devices via package interconnect 1183. Package interconnect 1183 may be coupled to the surface of substrate 1180 to route electrical signals to other electronic devices, such as motherboards, other chipsets, or multi-chip modules.

[0141] In some embodiments, the logic or I / O chip 1174 and the memory chip 1175 may be electrically coupled via a bridge 1187 configured to route electrical signals between the logic or I / O chip 1174 and the memory chip 1175. The bridge 1187 may be a dense interconnect structure providing routing of electrical signals. The bridge 1187 may include a bridge substrate made of glass or a suitable semiconductor material. Circuitry features may be formed on the bridge substrate to provide chip-to-chip connectivity between the logic or I / O chip 1174 and the memory chip 1175. The bridge 1187 may also be referred to as a silicon bridge or an interconnect bridge. For example, in some embodiments, the bridge 1187 is an embedded multi-die interconnect bridge (EMIB). In some embodiments, the bridge 1187 may simply be a direct connection from one chip to another.

[0142] The substrate 1180 may include hardware components for I / O 1191, cache memory 1192, and other hardware logic 1193. Structure 1185 may be embedded in the substrate 1180 to enable communication between various logic chips within the substrate 1180 and the logic 1191, 1193. In one embodiment, the I / O 1191, structure 1185, cache, bridge, and other hardware logic 1193 may be integrated into a base die layered on top of the substrate 1180.

[0143] In various embodiments, package assembly 1190 may include fewer or more components and wafers interconnected by structure 1185 or one or more bridges 1187. The wafers within package assembly 1190 may be arranged in 3D or 2.5D. Generally, bridge structure 1187 can be used to facilitate point-to-point interconnections, for example, between logic or I / O wafers and memory wafers. Structure 1185 can be used to interconnect various logic and / or I / O wafers (e.g., wafers 1172, 1174, 1191, 1193) with other logic and / or I / O wafers. In one embodiment, cache memory 1192 within the substrate may be used as part of a global cache for package assembly 1190, a distributed global cache, or as a dedicated cache for structure 1185.

[0144] Figure 11D shows a package assembly 1194 including interchangeable chiplets 1195 according to one embodiment. The interchangeable chiplets 1195 can be assembled into standardized slots for one or more base chiplets 1196, 1198. The base chiplets 1196, 1198 can be coupled via bridge interconnects 1197, which can be similar to other bridge interconnects described herein and can be, for example, EMIB. Memory chiplets can also be connected to logic or I / O chiplets via bridge interconnects. I / O and logic chiplets can communicate through interconnect structures. Each base chiplet can support one or more slots in a standardized format for either logic or I / O or memory / cache.

[0145] In one embodiment, the SRAM and power delivery circuitry can be fabricated as one or more of the base wafers 1196, 1198, which can be manufactured using different process technologies associated with the interchangeable wafers 1195 stacked on top of the base wafers. For example, the base wafers 1196, 1198 can be manufactured using a larger process technology, while the interchangeable wafers can be manufactured using a smaller process technology. One or more of the interchangeable wafers 1195 can be memory (e.g., DRAM) wafers. Different memory densities can be selected for the package assembly 1194 based on the power and / or performance of the product using the package assembly 1194. Furthermore, logic wafers with different numbers of different types of functional units can be selected during assembly based on the power and / or performance of the product. Additionally, wafers containing different types of IP logic cores can be inserted into slots for the interchangeable wafers, thereby enabling hybrid processor designs that can mix and match different technology IP blocks. Exemplary System on Chip Integrated Circuit

[0146] Figures 12-14 show exemplary integrated circuits and associated graphics processors that can be fabricated using one or more IP cores according to the various embodiments described herein. In addition to those shown, other logic and circuitry may also be included, including additional graphics processors / cores, peripheral interface controllers, or general-purpose processor cores.

[0147] Figure 12 is a block diagram showing an exemplary system-on-a-chip integrated circuit 1200, which can be fabricated using one or more IP cores according to an embodiment. The exemplary integrated circuit 1200 includes one or more application processors 1205 (e.g., CPUs), at least one graphics processor 1210, and may also include an image processor 1215 and / or a video processor 1220, any of which may be modular IP cores from the same or more different design houses. The integrated circuit 1200 includes peripheral or bus logic, including a USB controller 1225, a UART controller 1230, an SPI / SDIO controller 1235, and an I2S / I2C controller 1240. Additionally, the integrated circuit may include a display device 1245 coupled to one or more of a high-resolution multimedia interface (HDMI) controller 1250 and a mobile industry processor interface (MIPI) display interface 1255. The storage device may be provided by a flash memory subsystem 1260 (including flash memory and a flash memory controller). A memory interface may be provided via a memory controller 1265 for accessing SDRAM or SRAM memory devices. Some integrated circuits also include an embedded security engine 1270.

[0148] Figures 13-14 are block diagrams showing exemplary graphics processors for use within a SoC according to embodiments described herein. Figure 13A shows an exemplary graphics processor 1310 of a system-on-a-chip integrated circuit that can be fabricated using one or more IP cores according to an embodiment. Figure 13B shows an additional exemplary graphics processor 1340 of a system-on-a-chip integrated circuit that can be fabricated using one or more IP cores according to an embodiment. The graphics processor 1310 of Figure 13A is an example of a low-power graphics processor core. The graphics processor 1340 of Figure 13B is an example of a higher-performance graphics processor core. Each of the graphics processors 1310 and 1340 may be a variant of the graphics processor 1210 of Figure 12.

[0149] As shown in FIG13, the graphics processor 1310 includes a vertex processor 1305 and one or more fragment processors 1315A-1315N (e.g., 1315A, 1315B, 1315C, 1315D to 1315N-1 and 1315N). The graphics processor 1310 is capable of running different shader programs via independent logic, such that the vertex processor 1305 is optimized to run the operations of the vertex shader program, while the one or more fragment processors 1315A-1315N run the fragment (e.g., pixel) shading operations of the fragment or pixel shader program. The vertex processor 1305 executes the vertex processing stage of the 3D graphics pipeline and generates primitive and vertex data. The (one or more) fragment processors 1315A-1315N use the primitive and vertex data generated by the vertex processor 1305 to generate a frame buffer, which is displayed on a display device. In one embodiment, (one or more) fragment processors 1315A-1315N are optimized to run fragment shader programs as provided in the OpenGL API, which can be used to perform operations similar to those of pixel shader programs as provided in the Direct 3D API.

[0150] The graphics processor 1310 also includes one or more memory management units (MMUs) 1320A-1320B, (one or more) caches 1325A-1325B, and (one or more) circuit interconnects 1330A-1330B. The one or more MMUs 1320A-1320B are graphics processors 1310, including providing virtual-to-physical address mapping for vertex processors 1305 and / or (one or more) fragment processors 1315A-1315N. The virtual-to-physical address mapping can refer to vertex or image / texture data stored in memory in addition to vertex or image / texture data stored in the one or more caches 1325A-1325B. In one embodiment, the one or more MMUs 1320A-1320B can be synchronized with other MMUs within the system, including one or more MMUs associated with the one or more application processors 1205, image processor 1215, and / or video processor 1220 of FIG. 12, such that each processor 1205-1220 can participate in a shared or unified virtual memory system. According to an embodiment, the one or more circuit interconnects 1330A-1330B enable the graphics processor 1310 to interface with other IP cores within the SoC via the SoC's internal bus or via direct connections.

[0151] As shown in Figure 14, the graphics processor 1340 includes one or more MMUs 1320A-1320B, caches 1325A-1325B, and circuit interconnects 1330A-1330B of the graphics processor 1310 of Figure 13. The graphics processor 1340 includes one or more shader cores 1355A-1355N (e.g., 1355A, 1355B, 1355C, 1355D, 1355E, 1355F to 1355N-1 and 1355N) that provide a unified shader core architecture, wherein a single core or type of core is capable of running all types of programmable shader code, including shader programmable code, to implement vertex shaders, fragment shaders, and / or compute shaders. The exact number of shader cores present can vary between embodiments and implementations. Additionally, the graphics processor 1340 includes an inter-core task manager 1345, which acts as a thread dispatcher dispatching execution threads to one or more shader cores 1355A-1355N, and a tiling unit 1358 accelerating tiling operations for slice-based rendering, where scene rendering operations are subdivided in the image space, for example to utilize regional spatial consistency in the scene or optimize the use of internal caches. Ray tracing utilizing machine learning is also included.

[0152] As described above, ray tracing is a graphics processing technique in which light transmission is simulated through entity-based rendering. One of the key operations in ray tracing is handling visibility queries, which require traversing and intersection testing of nodes in the bounded volume hierarchy (BVH).

[0153] Ray and path tracing techniques calculate advanced effects such as shadows, glossiness, and indirect lighting by tracing rays and paths through each pixel and using random sampling. Using only a few samples is fast but produces noisy images, while using many samples produces high-quality images but is expensive.

[0154] Machine learning includes any circuit, code, or combination thereof that can progressively improve the performance of a given task or progressively produce more accurate predictions or decisions. Some machine learning engines can perform these tasks or produce these predictions / decisions without explicitly programming the performance of the task or the production of the predictions / decisions. Various machine learning techniques exist, including (but not limited to) supervised and semi-supervised learning, unsupervised learning, and reinforcement learning.

[0155] In recent years, breakthrough solutions for real-time ray / path tracing have emerged in the form of "denoising"—procedures using image processing techniques to produce high-quality, filtered / denoised images from noisy, low-sample-count inputs. The most effective denoising techniques rely on machine learning techniques, in which a machine learning engine learns what a noisy image might look like if more samples were used. In one particular implementation, machine learning is performed by a convolutional neural network (CNN); however, the basic principles of this invention are not limited to CNN implementations. In such implementations, low-sample-count inputs and realism are used to generate training data. Training a CNN can predict converging pixels from the vicinity of noisy pixel inputs surrounding the pixel in question.

[0156] Despite its imperfections, this AI-based denoising technique has proven remarkably effective. However, it's important to note that good training data is crucial because the network may predict incorrect results. For instance, if an animation film studio trains a denoising CNN on past films featuring land scenes and then attempts to denoise frames in a new film featuring water scenes using the trained CNN, the denoising performance will be subpar.

[0157] To address this issue, learning data can be dynamically collected during rendering, and machine learning engines such as CNNs can be continuously trained based on the data currently being processed, thereby continuously improving the machine learning engine for the task at hand. Therefore, the training phase may still be performed before runtime, but machine learning weights are continuously adjusted as needed during runtime. Thus, by limiting the generation of learning data to a sub-region of the image per frame or per N frames, the high cost of computing the reference data required for training is avoided. In particular, noisy input to the frames is generated to denoise the entire frame using the current network. Furthermore, as described below, reference pixels in small regions are generated and used for continuous training.

[0158] Although this paper describes a CNN implementation, any form of machine learning engine can be used, including but not limited to systems that perform supervised learning (e.g., building a mathematical model of a dataset containing both inputs and desired outputs), unsupervised learning (e.g., evaluating input data of some type of structure) and / or a combination of supervised and unsupervised learning.

[0159] Existing noise reduction implementations operate in both the training and runtime phases. During the training phase, a network topology is defined that receives an NxN pixel region with various per-pixel data channels (e.g., pixel color, depth, normal, normal bias, primitive ID, and albedo) and produces the final pixel color. A set of "representative" training data is generated using a low number of input values ​​from a single frame, referenced to a "desired" pixel color calculated with a very high number of samples. The network is trained against these inputs, resulting in a set of "ideal" weights for the network. In these implementations, the reference data is used to train the network weights so that the network output is closest to the desired result.

[0160] During runtime, the given, pre-computed ideal network weights are loaded and the network is initialized. For each frame, a low-sample-number denoised input image (i.e., the same one used for training) is generated. For each pixel, a given neighborhood of the pixel input is run through the network to predict the color of the "denoised" pixel, thereby generating a denoised frame.

[0161] Figure 15 shows the initial training implementation. The machine learning engine 1500 (e.g., CNN) receives an N×N pixel region as high-sample-count image data 1702, which has various per-pixel data channels, such as pixel color, depth, normal, normal bias, raw ID, and albedo, and produces the final pixel color. Low-sample-count input values ​​1501 are used to generate representative training data. The network is trained on these inputs to produce a set of "ideal" weights 1505, which the machine learning engine 1500 then uses at runtime to denoise the low-sample-count image.

[0162] To improve the above technique, a denoising stage is added to generate new training data for each frame or subset of frames (e.g., every N frames, where N=2, 3, 4, 10, 25, etc.). In particular, as shown in FIG16, one or more regions in each frame, referred to herein as "new reference regions" 1602, are selected and presented in a separate high sample number buffer 1604 with a high sample number. The low sample number buffer 1603 stores the low sample number input frames 1601 (including the low sample region 1604 corresponding to the new reference region 1602).

[0163] The position of the new reference area 1602 can be randomly selected. Alternatively, the position of the new reference area 1602 can be adjusted in a predetermined manner for each new frame (e.g., using a predefined movement of the area between frames, a specified area limited to the center of the frame, etc.).

[0164] Regardless of the new reference region selected, the machine learning engine 1600 uses it to continuously refine and update the trained weights 1605 for noise reduction. Specifically, reference pixel colors from each new reference region 1602 and noisy reference pixel inputs from the corresponding low-sample-count region 1607 are presented. Supplementary training is then performed on the machine learning engine 1600 using the high-sample-count reference region 1602 and the corresponding low-sample-count region 1607. This training, unlike the initial training, is performed continuously for each new reference region 1602 during runtime, ensuring accurate training of the machine learning engine 1600. For example, the data channels of each pixel (e.g., pixel color, depth, normal, normal deviation, etc.) can be evaluated and used by the machine learning engine 1600 to adjust the trained weights 1605. As in the training case (Figure 15), the machine learning engine 1600 is trained towards a set of ideal weights 1605 to remove noise from the low-sample-count input frame 1601 to produce a denoised frame 1620. However, based on the new image characteristics of the novel low-sample-number input frame 1601, the training weights 1605 are continuously updated.

[0165] The retraining operation performed by the machine learning engine 1600 can be executed simultaneously in a background program on a graphics processing unit (GPU) or a host processor. A drawing loop, which can be implemented as a driver element and / or GPU hardware element, can continuously generate new training data (e.g., in the form of a new reference region 1602) and place it in a queue. The background training program executed on the GPU or host processor can continuously read new training data from this queue, retrain the machine learning engine 1600, and update it with new weights 1605 at appropriate time intervals.

[0166] Figure 17 shows an example of such an implementation of the background training program 1700 implemented by the host CPU 1710. In particular, the background training program 1700 uses a new reference region 1602 with a high number of samples and a corresponding low-sample region 1604 to continuously update the trained weights 1605, thereby updating the machine learning engine 1600.

[0167] As shown in Figure 18A, a non-limiting example of a multiplayer online game, different host machines 1820-1822 individually generate background training programs 1700A-C and send them to the server 1800 (e.g., a game server) as reference regions. The server 1800 then uses the new reference regions received from each of the hosts 1821-1822 to perform training on the machine learning engine 1810, updating the weights 1805 as described above. These weights 1805 are sent to the host machine 1820, which stores the weights 1605A-C, thereby updating each individual machine learning engine (not shown). Because a large number of reference regions can be provided to the server 1800 in a short period of time, the weights can be updated efficiently and accurately for any given application being performed by a user (e.g., an online game).

[0168] As shown in Figure 18B, different host machines may generate newly trained weights (e.g., based on training / reference region 1602 as described above) and share these newly trained weights with server 1800 (e.g., such as a game server), or use a peer-to-peer sharing protocol. The machine learning management component 1810 on the server uses the new weights received from each host machine to generate a set of combined weights 1805. For example, the combined weights 1805 may be an average value generated from the new weights and continuously updated as described herein. Once generated, copies of the combined weights 1605A-C can be sent and stored on each host machine 1820-1821, which can then use the combined weights as described herein to perform a noise reduction operation.

[0169] Hardware manufacturers may also use a semi-closed-loop update mechanism. For example, a reference network may be included as part of a driver distributed by the hardware manufacturer. As the driver uses the techniques described herein to generate new training data and continuously submits it back to the hardware manufacturer, the hardware manufacturer will use this information to continue improving its machine learning implementation for the next driver update.

[0170] In an exemplary implementation (e.g., in batch movie rendering on a rendering field), the renderer transfers newly generated training regions to a dedicated server or database (in the rendering field of the studio), which aggregates data from multiple rendering nodes over time. Separate processing on individual machines continuously improves the studio's dedicated noise reduction network, and new rendering jobs always use the latest trained network.

[0171] The machine learning method is shown in Figure 19. The method can be implemented on the architecture described herein, but is not limited to any particular system or graphics processing architecture.

[0172] At 1901, as part of the initial training phase, low-sample-count and high-sample-count image data are generated for a plurality of image frames. At 1902, the high / low-sample-count image data are used to train a machine learning denoising engine. For example, a set of convolutional neural network weights associated with pixel features can be updated based on the training. However, any machine learning architecture can be used.

[0173] At 1903, during runtime, a low-sample-count image frame and at least one reference region with a high sample-count are generated. At 1904, the machine learning engine and / or separate training logic (e.g., background training module 1700) use the high-sample-count reference region to continuously improve the training of the machine learning engine. For example, the high-sample-count reference region can be used in conjunction with a corresponding portion of the low-sample-count image to continue teaching the machine learning engine 1904 how to perform denoising most effectively. For example, in a CNN implementation, this might involve updating the weights associated with the CNN.

[0174] Several variations described above can be implemented, such as the manner in which the feedback loop to the machine learning engine is configured, the entity that generates the training data, the manner in which the training data is fed back to the training engine, and how the improved network is provided to the rendering engine. Furthermore, although the above example uses a single reference region to perform continuous training, any number of reference regions can be used. Moreover, as mentioned above, the reference regions can have different sizes, can be used on different numbers of image frames, and can be positioned at different locations within the image frames using different techniques (e.g., random, according to a predetermined pattern, etc.).

[0175] Furthermore, although convolutional neural networks (CNNs) are described as an example of machine learning engine 1600, the basic principles of this invention can be implemented using any form of machine learning engine capable of continuously improving its functionality with new training data. Other machine learning implementations, by way of example and not limitation, include group method of data handling (GMDH), long short-term memory, deep reservoir computing, deep belief networks, tensor deep stacked networks, and deep predictive coding networks, to name just a few. Efficient distributed denoising devices and methods.

[0176] As mentioned above, denoising has become a key feature for real-time ray tracing with smooth, noise-free images. Rendering can be performed on distributed systems across multiple devices, but so far, existing denoising frameworks have all run on a single instance on a single machine. If rendering is being performed across multiple devices, they may not have access to all the rendered pixels to compute the denoised portion of the image.

[0177] A distributed denoising algorithm is proposed that works with both artificial intelligence (AI) and non-AI-based denoising techniques. Image regions are either distributed across nodes through distributed rendering operations or split and distributed from a single frame buffer. When needed, ghosting regions of neighboring regions sufficient for denoising are collected from adjacent nodes, and the resulting slices are synthesized into the final image. Distributed processing.

[0178] Figure 20 shows multiple nodes 2021-2023 that are being drawn. Although only three nodes are shown for simplicity, the basic principles of the invention are not limited to any particular number of nodes. In fact, a single node can be used to implement certain embodiments of the invention.

[0179] Each of nodes 2021-2023 draws a portion of the image, producing regions 2011-2013 in this example. Although rectangular regions 2011-2013 are shown in Figure 20, regions of any shape can be used, and any device can process any number of regions. The regions required for a sufficiently smooth noise reduction operation by the nodes are called ghost regions 2011-2013. In other words, ghost regions 2001-2003 represent all the data required to perform noise reduction at a specified quality level. Decreasing the quality level reduces the size of the ghost region, thus increasing the amount of data required, while increasing the quality level increases the ghost region and the corresponding amount of data required.

[0180] If a node such as node 2021 does indeed have a copy of a portion of the ghosted region 2001 required for denoising its region 2011 at a specified quality level, then the node will retrieve the required data from one or more "neighboring" nodes, such as node 2022, which possesses a portion of the ghosted region 2001, as shown in the figure. Similarly, if node 2022 does indeed have a copy of a portion of the ghosted region 2002 required for denoising its region 2012 at a specified quality level, then node 2022 will retrieve the required ghosted region data 2032 from node 2021. The retrieval can be performed via buses, interconnects, high-speed memory structures, networks (e.g., high-speed Ethernet), or even on-chip interconnects in multi-core chips capable of distributing rendering work among multiple cores (e.g., for rendering large images at extremely high resolution or time-varying resolution). Each node 2021-2023 may contain a separate execution unit or a specified set of execution units within the graphics processor.

[0181] The specific amount of data to be transmitted depends on the noise reduction technique used. Furthermore, the data from the ghosting regions can contain any data necessary to improve noise reduction for each corresponding region. For example, the ghosting region data can contain image color / wavelength, intensity / alpha data, and / or normals. However, the basic principles of the invention are not limited to any particular set of ghosting region data. Additional Details

[0182] For slower networks or interconnects, existing general-purpose lossless or lossy compression can be used to utilize the compression of the data. Examples include, but are not limited to, zlib, gzip, and the Lempel-Ziv-Markov Chain Algorithm (LZMA). Since the increments in the information hit by rays between attention frames may be very sparse, further content-specific compression can be used, and when a node already has increments collected from previous frames, only samples that contribute to those increments need to be sent. These samples can be selectively pushed to the node that collected them, or the node can request samples from other nodes. Lossless compression is used for certain types of data and code, while lossy compression is used for other types of data.

[0183] Figure 21 shows additional details of the interaction between nodes 2021-2022. Each node 2021-2022 contains ray-tracing rendering circuitry 2081-2082 for rendering the respective image regions 2011-2012 and ghost regions 2001-2002. Denoising units 2100-2111 perform denoising operations on regions 2011-2012 respectively, with each node 2021-2022 responsible for rendering and denoising. Denoising units 2021-2022 may, for example, contain circuitry, software, or any combination thereof to generate denoised regions 2121-2122 respectively. As mentioned, when generating denoised regions, denoising units 2021-2022 may need to depend on data in ghost regions owned by different nodes (e.g., denoising unit 2100 may need data from ghost region 2002 owned by node 2022).

[0184] Therefore, noise reduction units 2100-2111 can generate noise reduction regions 2121-2122 using data from regions 2011-2012 and ghosted regions 2001-2002, respectively, at least a portion of which can be received from another node. As described herein, region data managers 2101-2102 can manage data transmission from ghosted regions 2001-2002. Compressor / decompressor units 2131-2132 can respectively perform compression and decompression of ghosted region data exchanged between nodes 2021-2022.

[0185] For example, the area data manager 2101 of node 2021 can send data from ghost region 2001 to compressor / decompressor 2131 according to a request from node 2022. Compressor / decompressor 2131 compresses the data to produce compressed data 2106 and transmits it to node 2022, thereby reducing bandwidth on interconnects, networks, buses, or other data communication links. Then, the compressor / decompressor 2132 of node 2022 decompresses the compressed data 2106, and noise reduction unit 2111 uses the decompressed ghost data to generate a noise-reduced region 2012 that may have higher quality than data from region 2012 alone. The area data manager 2102 can store the decompressed data from ghost region 2001 in a cache, memory, temporary file, or other storage to provide to noise reduction unit 2111 when generating noise-reduced region 2122. A similar set of operations can be performed to provide data from ghosted region 2002 to denoiser 2100 on node 2021. Denoiser 2100 combines this data with data from region 2011 to produce a higher-quality denoised region 2121. (Data capture or plotting)

[0186] If connections between devices such as nodes 2021-2022 are slow (i.e., below critical wait times and / or critical bandwidth), rendering ghosted regions may be regionally faster than requesting results from other devices. This can be determined at runtime by tracking network transaction speeds and linearly extrapolating rendering time for the size of the ghosted region. In cases where rendering the entire ghosted region is faster, multiple devices may end up rendering the same portion of the image. The resolution of the rendered portion of the ghosted region can be adjusted based on variations in the base region and a determined degree of blur. Load balancing

[0187] Static and / or dynamic load balancing schemes can be used to distribute processing load among various nodes in 2021-2023. For dynamic load balancing, the variance determined by the noise reduction filter may require more time for noise reduction, but will drive the number of samples used to render specific areas of the scene, while areas with low variance and blurred images require fewer samples. Specific areas allocated to specific nodes can be dynamically adjusted based on data from previous frames, or dynamic communication can be made between devices as they are rendered to ensure that all devices have the same workload.

[0188] Figure 22 illustrates how monitors 2201-2202 running on each node 2021-2022 collect performance metrics, including but not limited to the time consumed for transmitting data through network interfaces 2211-2212, the time consumed for denoising a region (with and without ghosting region data), and the time consumed for plotting each region / ghosting region. Monitors 2201-2202 report these performance metrics to manager or load balancer node 2201, which analyzes the data to identify the current workload on each node 2021-2022 and may determine more efficient patterns for processing various denoised regions 2121-2122. Then, manager node 2201 allocates new workloads for new regions to nodes 2021-2022 based on the detected load. For example, manager node 2201 may send more work to nodes with lower loads and / or reallocate work from overloaded nodes. In addition, load balancer node 2201 can send reconfiguration commands to adjust the specific methods by which each node performs rendering and / or noise reduction (some examples of which are described above). Determining ghosting areas.

[0189] The size and shape of the ghost regions 2001-2002 can be determined based on the denoising algorithm implemented by denoisers 2100-2111. Their respective sizes can then be dynamically modified based on the detected number of denoised sample variations. The learning algorithm used for AI denoising itself can be used to determine the appropriate region size, or in other cases such as bilateral blurring, a predetermined filter width will determine the size of the ghost regions 2001-2002. In an exemplary implementation using the learning algorithm, the machine learning engine can be executed on manager node 2201 and / or the machine learning portion can be executed on each of the single nodes 2021-2023 (e.g., see Figures 18A-B and the related text above). Collect the final image.

[0190] The final image can be generated by collecting the drawn and denoised regions from each of nodes 2021-2023, without the need for ghosting regions or normals. In Figure 22, for example, denoised regions 2121-2122 are sent to the region processor 2280 of the manager node 2201, which combines the regions to produce the final denoised image 2290, which is then displayed on the display 2290. The region processor 2280 can combine regions using various 2D compositing techniques. Although displayed as separate elements, the region processor 2280 and the denoised image 2290 can be integrated with the display 2290. The various nodes 2021-2022 can use direct transmission techniques to send the denoised regions 2121-2122, and may use various lossy or lossless compression of the region data.

[0191] As games move to the cloud, AI noise reduction remains an expensive operation. Therefore, it may be necessary to distribute noise reduction processing across multiple nodes in 2021-2022 to achieve real-time frame rates for traditional games or virtual reality (VR) that requires higher frame rates. Film studios also frequently perform rendering in large rendering fields, which can be used for faster noise reduction.

[0192] Figure 23 shows an exemplary method for performing distributed rendering and denoising. This method can be implemented within the context of the system architecture described above, but is not limited to any particular system architecture.

[0193] At 2301, the graphics work is dispatched to a plurality of nodes, which perform ray tracing operations to draw regions of the image frame. Each node may already have the data required to perform operations in memory. For example, two or more nodes may share a common memory, or the node's region memory may already store data from previous ray tracing operations. Alternatively, or additionally, some data may be sent to each node.

[0194] At 2302, determine the "ghosting region" required for the specified noise reduction level (that is, within an acceptable performance level). The ghosting region contains any data required to perform the specified noise reduction level, including data owned by one or more other nodes.

[0195] At 2303, data relating to the ghosting region (or a portion thereof) is exchanged between nodes. At 2304, each node performs denoising on its respective region (e.g., using the exchanged data), and at 2305, the results are combined to produce the final denoised image frame.

[0196] As shown in Figure 22, a manager node or master node can dispatch work to nodes, and then combine the work performed by the nodes to produce the final image frame. A peer-based architecture can be used, where nodes are peers and they exchange data to draw and denoise the final image frame.

[0197] The nodes described herein (e.g., nodes 2021-2023) can be graphics processing computing systems interconnected via high-speed networks. Alternatively, nodes can be individual processing elements coupled to high-speed memory structures. All nodes can share a common virtual memory space and / or a common physical memory. Alternatively, nodes can be a combination of CPUs and GPUs. For example, the aforementioned manager node 2201 can be a CPU and / or software executing on a CPU, and nodes 2021-2022 can be GPUs and / or software executing on GPUs. Various types of nodes can be used while still adhering to the basic principles of the invention. Exemplary Neural Network Implementation

[0198] There are many types of neural networks; a simple type is the feedforward network. A feedforward network can be implemented as a non-cyclic graph in which nodes are arranged in layers. Typically, a feedforward network topology includes an input layer and an output layer separated by at least one hidden layer. The hidden layer transforms the input received by the input layer into a representation that can be used to produce an output in the output layer. Network nodes are fully connected to nodes in adjacent layers via edges, but there are no edges between nodes in each layer. Data received at the nodes in the input layer of the feedforward network is propagated (i.e., "feedforward") to the nodes in the output layer via an enable function that calculates the state of nodes in each consecutive layer of the network based on coefficients ("weights") associated with each edge of the connecting layer, respectively. The output of the neural network algorithm can take various forms depending on the specific model represented by the algorithm being executed.

[0199] Before a machine learning algorithm can be used to model a specific problem, a training dataset is used to train the algorithm. Training a neural network involves selecting a network topology, using a set of training data representing one of the problems modeled by the network, and adjusting the weights until the network model performs with the minimum error across all instances of the training dataset. For example, during supervised learning training for a neural network, in response to inputs representing instances in the training dataset, the outputs produced by the network are compared with the outputs labeled "correct" for the instances, an error signal representing the difference between the outputs and the labeled outputs is calculated, and the weights associated with the connections are adjusted to minimize the error as the error signal propagates back through the network layers. The network is considered "trained" when the error of each output produced from instances of the training dataset is minimized.

[0200] The accuracy of machine learning algorithms is significantly affected by the quality of the dataset used to train them. Training processing can be computationally intensive and may require a significant amount of time on conventional general-purpose processors. Therefore, parallel processing hardware is used to train many types of machine learning algorithms. This is particularly useful for optimizing the training of neural networks, as the computations performed in tuning the coefficients in neural networks are inherently suited for parallel implementation. Specifically, many machine learning algorithms and software applications have been adapted to utilize parallel processing hardware within general-purpose graphics processing units.

[0201] Figure 24 is an overview diagram of the machine learning software stack 2400. The machine learning application 2402 can be configured to train a neural network using a training dataset or configured to implement machine intelligence using a trained deep neural network. The machine learning application 2402 may include training and inference capabilities for neural networks and / or dedicated software that can be used to train the neural network prior to deployment. The machine learning application 2402 can implement any type of machine intelligence, including but not limited to image recognition, mapping and localization, autonomous navigation, speech synthesis, medical image or language translation.

[0202] Hardware acceleration of the machine learning application 2402 can be enabled via the machine learning framework 2404. The machine learning framework 2404 can be implemented on the hardware described herein, such as the processing system 100 containing the processors and components described herein. Elements described in FIG. 24 with the same or similar names as elements in any other figures herein describe the same elements as in other figures, can operate or function in a similar manner, may contain the same components, and may be linked to other entities (such as entities elsewhere in this document), but are not limited thereto. The machine learning framework 2404 can provide a library of machine learning primitives. Machine learning primitives are typically basic operations performed by machine learning algorithms. Without the machine learning framework 2404, developers of machine learning algorithms would need to create and optimize the main computational logic associated with the machine learning algorithms, and then re-optimize the computational logic when developing new parallel processors. Instead, machine learning applications can be configured to perform the necessary computations using primitives provided by the machine learning framework 2404. Exemplary primitives include tensor convolution, activation functions, and pooling, which are computational operations performed when training convolutional neural networks (CNNs). The machine learning framework 2404 can also provide primitives to implement basic linear algebra subroutines that are executed by many machine learning algorithms, such as matrix and vector operations.

[0203] The machine learning framework 2404 can process the input data received from the machine learning application 2402 and generate appropriate input for the computing framework 2406. The computing framework 2406 can extract the basic instructions provided to the GPGPU driver 2408, so that the machine learning framework 2404 can utilize hardware acceleration via the GPGPU hardware 2410 without requiring the machine learning framework 2404 to have an in-depth understanding of the architecture of the GPGPU hardware 2410. In addition, the computing framework 2406 can enable hardware acceleration of the machine learning framework 2404 across various types and generations of GPGPU hardware 2410.

[0204] GPGPU Machine Learning Acceleration

[0205] Figure 25 shows a multi-GPU computing system 2500, which may be a variant of the processing system 100. Therefore, disclosure of any feature in conjunction with the processing system 100 herein also discloses a corresponding conjunction with the multi-GPU computing system 2500, but is not limited thereto. Elements in Figure 25 having the same or similar names as elements in any other figures herein describe the same elements as in the other figures, can operate or function in a similar manner, may contain the same components, and may link to entities as described elsewhere herein, but are not limited thereto. The multi-GPU computing system 2500 may include a processor 2502 coupled to a plurality of GPGPUs 2506A-D via a host interface switch 2504. The host interface switch 2504 may be, for example, a PCI express switch device that couples the processor 2502 to a PCI express bus, through which the processor 2502 can communicate with the aggregate of GPGPUs 2506A-D. Each of the plurality of GPGPUs 2506A-D may be an instance of the aforementioned GPGPU. The GPGPU 2506A-D can be interconnected via a set of high-speed point-to-point GPU-to-GPU links 2516. High-speed GPU-to-GPU links can be connected to each of the GPGPU 2506A-D via dedicated GPU links. The P2P GPU links 2516 enable direct communication between the GPGPU 2506A-D without requiring communication through the main interface bus (connected to the processor 2502). Using the GPU-to-GPU traffic directed to the P2P GPU links, the main interface bus remains available for system memory access or communication with other instances of the multi-GPU computing system 2500, for example, via one or more network devices. Instead of connecting the GPGPU 2506A-D to the processor 2502 via the main interface switch 2504, the processor 2502 may include direct support for the P2P GPU links 2516 and can be directly connected to the GPGPU 2506A-D.

[0206] Implementation of Machine Learning Neural Networks

[0207] The computational architecture described herein can be configured to perform multiple types of parallel processing, making it particularly suitable for training and deploying neural networks for machine learning. Neural networks can be generalized as networks with graph-like relational capabilities. As is well known in the art, there are various types of neural network implementations for machine learning. As previously mentioned, a paradigmatic type of neural network is the feedforward network.

[0208] The second exemplary type of neural network is the convolutional neural network (CNN). A CNN is a specialized feedforward neural network used to process data with known, raster-like topology (such as image data). Therefore, CNNs are commonly used in computational vision and image recognition applications, but they can also be used in other types of pattern recognition, such as speech and language processing. Nodes in the input layer of a CNN are organized into a set of "filters" (feature detectors driven by individual columns found in the retina), and the outputs of each set of filters are propagated to nodes in successive layers of the network. The computation of a CNN involves applying convolutional mathematical calculations to each filter to produce the output of that filter. Convolution is a special kind of mathematical calculation performed by two functions to produce a third function that is a modified version of one of the two original functions. In convolutional network terminology, the first function to the convolution can be called the input, and the second function can be called the convolution kernel. The output can be called a feature map. For example, the input to a convolutional layer can be a multidimensional array of data defining various color components of an input image. A convolutional kernel can be a multidimensional array of parameters, which are tuned by a neural network training procedure.

[0209] Recursive Neural Networks (RNNs) are a family of feedforward neural networks that include feedback connections between layers. RNNs model continuous data by sharing parameter data from different parts of the neural network. The architecture of an RNN includes periods. These periods represent the influence of the current value of a variable on its value at a future time, because at least a portion of the output data from the RNN is used as feedback to process subsequent inputs in the sequence. This feature makes RNNs particularly useful for language processing, as language data can be composed of variable units.

[0210] The following graphical representations illustrate feedforward, CNN, and RNN networks, and describe general procedures for training and deploying each of those types of networks individually. It will be understood that these descriptions are exemplary and not limiting, and in general, the concepts shown can be commonly applied to deep neural networks and machine learning techniques.

[0211] The above-described exemplary neural network can be used to perform deep learning. Deep learning is machine learning that uses deep neural networks. Deep neural networks used for deep learning are artificial neural networks composed of multiple hidden layers, unlike shallow neural networks that only include a single hidden layer. Deeper neural networks are typically more computationally intensive for training. However, the additional hidden layers of the network enable multi-step pattern recognition, which leads to a reduction in output error compared to shallow machine learning techniques.

[0212] Deep neural networks used in deep learning typically include a front-end network for performing feature recognition, coupled to a back-end network that represents a mathematical model that performs operations (e.g., object classification, speech recognition, etc.) based on feature representations provided to the model. Deep learning enables machine learning to be performed without requiring manual feature engineering to be applied to the model. Instead, deep neural networks learn features based on statistical structures or correlations within the input data. The learned features are then fed to a mathematical model that maps the measured features to the output. The mathematical model used by the network is generally specialized for the specific task to be performed, and different models will be used to perform different tasks.

[0213] Once a neural network is constructed, a learning model can be applied to it to train it to perform a specific task. This learning model describes how to adjust the weights within the model to reduce the network's output errors. Error backpropagation is a common method used to train neural networks. Input vectors are submitted to the network for processing. The network's output is compared to the desired output using a loss function, and error values ​​are calculated for each neuron in the output layer. These error values ​​are then backpropagated until each neuron has a relevant error value that roughly represents its contribution to the original output. The network can then learn from those errors using algorithms (such as stochastic gradient descent) to update the weights of the neural network.

[0214] Figures 26 and 27 show exemplary convolutional neural networks. Figure 26 shows the various layers within a CNN. As shown in Figure 26, an exemplary CNN used to model image processing receives input 2602 describing the red, green, and blue (RGB) components of an input image. Input 2602 can be processed by multiple convolutional layers (e.g., convolutional layer 2604, convolutional layer 2606). The outputs from the multiple convolutional layers can be selectively processed by a set of fully connected layers 2608. As previously described for feedforward networks, neurons in fully connected layers have full connections to all activations in previous layers. The outputs from fully connected layers 2608 can be used to produce outputs from the network. Activations within fully connected layers 2608 can be computed using matrix multiplication instead of convolutions. Not all CNN implementations utilize fully connected layers. For example, in some implementations, convolutional layer 2606 can produce outputs for the CNN.

[0215] The convolutional layers are sparsely connected, unlike the traditional neural network configuration seen in the fully connected layers 2608. Traditional neural network layers are fully connected, such that each output unit interacts with each input unit. However, as shown, the convolutional layers are sparsely connected because the output of the convolution of a column is input (replacing the individual state values ​​of each node in that column) to nodes in subsequent layers. The kernels associated with these convolutional layers perform convolution operations, and their outputs are passed to the next layer. This dimensionality reduction within the convolutional layers is a form that allows the CNN to be scaled for processing large images.

[0216] Figure 27 shows an exemplary computational stage within a convolutional layer of a CNN. The input to the convolutional layer 2712 of the CNN can be processed in three stages of the convolutional layer 2714. These three stages may include a convolutional stage 2716, a detector stage 2718, and a pooling stage 2720. The convolutional layer 2714 can then output data to successive convolutional layers. The final convolutional layer of the network can produce output feature map data or provide input to a fully connected layer, for example, to generate sorted values ​​for the input to the CNN.

[0217] In convolution stage 2716, several convolutions are performed in parallel to produce a set of linear activations. Convolution stage 2716 may include affine transformations, which are any transformations that can be specified as linear transformations plus transformations. Affine transformations include rotations, transformations, scaling, and combinations of these transformations. The convolution stage computes the output of a function (e.g., a neuron), which is connected to a specific region in the input, which can be determined as a region associated with the neuron. The neuron computes the inner product between the weights of the neurons and the region in the input to which the neurons are connected. The output from convolution stage 2716 defines a set of linear activations, which are processed by successive stages of convolutional layer 2714.

[0218] These linear primings can be handled by the detector stage 2718. In the detector stage 2718, each linear priming system is handled by a nonlinear priming function. The nonlinear priming function adds nonlinearity to the overall network without affecting individual cells of the convolutional layer. Several types of nonlinear priming functions can be used. One particular type is the Rectified Linear Unit (ReLU), which uses a priming function defined as f(x) = max(0,x) such that the priming system is thresholded at zero.

[0219] Pooling stage 2720 uses a pooling function that replaces the output of convolutional layer 2706 with a summary statistic of the nearby output. Pooling functions can be used to introduce transformation invariance into neural networks, such that small transformations to the input do not change the pooling output. Invariance to region transformations may be useful in situations where the presence of a feature in the input data is more important than the precise location of that feature. Various types of pooling functions can be used during pooling stage 2720, including max pooling, average pooling, and 12-norm pooling. Furthermore, some CNN implementations do not include a pooling stage. Instead, such implementations use an additional convolutional stage with an increased stride relative to the previous convolutional stage.

[0220] The output from convolutional layer 2714 can then be processed by the next layer 2722. The next layer 2722 can be either an additional convolutional layer or a fully connected layer 2708. For example, the first convolutional layer 2704 of Figure 27 can output to the second convolutional layer 2706, and the second convolutional layer can output to the first layer of the fully connected layer 2708.

[0221] Figure 28 shows an exemplary recursive neural network 2800. In a recursive neural network (RNN), the previous state of the network influences the output of the current state. RNNs can be built in various ways using a variety of functions. The use of RNNs typically revolves around using mathematical models to predict the future based on previous sequences of inputs. For example, RNNs can be used to perform statistical language modeling to predict future words based on previous sequences of words. The RNN 2800 shown can be described as having an input layer 2802 that receives input vectors, a hidden layer 2804 that implements a recursive function, a feedback mechanism 2805 that causes a "memory" of previous states, and an output layer 2806 that outputs the results. The RNN 2800 operates according to time steps. The state of the RNN at a given time step is influenced by the feedback mechanism 2805 according to previous time steps. For a given time step, the state of the hidden layer 2804 is defined by the previous state and the input of the current time step. The initial input (x1) in the first time step can be processed by the hidden layer 2804. The second input (x2) can be processed by the hidden layer 2804 using state information determined during the processing of the initial input (x1). The given state can be computed as s_t=f(Ux_t+ Ws_(t-1)), where U and W are parameter matrices. The function f is typically nonlinear, such as a variation of the hyperbolic tangent function (Tanh) or the rectified function f(x)=max(0,x). However, the specific mathematical function used for the hidden layer 2804 may vary depending on the specific implementation details of the RNN 2800.

[0222] In addition to the basic CNN and RNN networks described above, variations of these networks can be made. One example RNN variation is the Long Short-Term Memory (LSTM) RNN. LSTM RNNs are able to learn long-term dependencies that may be necessary for processing longer language sequences. A variation of CNN is a convolutional deep neural network, which has a similar structure to CNNs and is trained in a similar manner to deep neural networks. Deep neural networks (DBNs) are generative neural networks consisting of multiple layers with random (arbitrary) variables. DBNs can be trained layer by layer using greedy unsupervised learning. The learned weights of the DBN can then be used to provide pre-trained neural networks by determining the optimal initial set of weights for this type of neural network.

[0223] Figure 29 illustrates the training and deployment of deep neural networks. Once the given network has been constructed for a task, the neural network is trained using training dataset 2902. Various training frameworks 2904 have been developed to accelerate the hardware of the training process. For example, the machine learning framework described above can be configured as a training framework. Training framework 2904 can be linked to an untrained neural network 2906 and cause the untrained neural network to be trained using the parallel processing resources described herein to produce a trained neural network 2908.

[0224] To begin the training process, the initial weights can be randomly selected or pre-trained using a deep-belief network. The training cycle is then performed in either a supervised or unsupervised manner.

[0225] Supervised learning is a learning method in which training is performed as an intermediary operation, such as when the training dataset 2902 includes inputs paired with desired outputs for inputs, or when the training dataset includes inputs with known outputs and the outputs of the neural network are manually graded. The network processes these inputs and compares the resulting outputs with a set of expected or desired outputs. Errors are then fed back through the system. The training framework 2904 can adjust the weights of the untrained neural network 2906. The training framework 2904 provides tools to monitor how the untrained neural network 2906 appropriately converges toward a model suitable for producing correct answers based on the known input data. The training process is repeated as the weights of the network are adjusted to improve the outputs produced by the neural network. The training process can continue until the neural network reaches a statistically desirable accuracy associated with the trained neural network 2908. The trained neural network 2908 can then be deployed to implement any number of machine learning operations.

[0226] Unsupervised learning is a learning method in which a network attempts to train itself using unlabeled data. Therefore, for unsupervised learning, the training dataset 2902 will include input data without any associated output data. The untrained neural network 2906 can learn clusters within the unlabeled inputs and determine how individual inputs relate to the overall dataset. Unsupervised training can be used to generate self-organizing maps, which are a type of trained neural network 2907 capable of performing operations that can reduce the dimensionality of the data. Unsupervised training can also be used to perform anomaly detection, allowing the identification of data points in the input dataset that deviate from the normal pattern of the data.

[0227] Variations in supervised and unsupervised training can also be utilized. Semi-supervised learning is a technique where the training data set 2902 includes a mixture of labeled and unlabeled data with the same distribution. Incremental learning is a variation of supervised learning, in which input data is continuously used to further train the model. Incremental learning causes the trained neural network 2908 to adapt to new data 2912 without forgetting the knowledge instilled within the network during the initial training period.

[0228] Whether supervised or unsupervised, training procedures, especially for deep neural networks, can be too computationally intensive for a single computing node. Instead of using a single computing node, a distributed network of computing nodes can be used to accelerate the training process.

[0229] Figure 30A is a block diagram illustrating distributed learning. Distributed learning is a training model that uses multiple distributed computing nodes (such as the nodes described above) to perform supervised or unsupervised training of neural networks. Each distributed computing node may include one or more host processors and one or more general-purpose processing nodes, such as highly parallel general-purpose graphics processing units. As shown, distributed learning can perform model parallelism 3002, data parallelism 3004, or a combination of model and data parallelism.

[0230] In model parallelism 3002, different computing nodes in a distributed system can perform training computations for different parts of a single network. For example, the layers of a neural network can be trained by different processing nodes in a distributed system. Advantages of model parallelism include the ability to scale to particularly large models. Separating the computations associated with different layers of a neural network enables the training of very large neural networks (where the weights of all layers will not fit into the memory of a single computing node). In some instances, model parallelism can be particularly useful for performing unsupervised training of large neural networks.

[0231] In data parallelism 3004, different nodes in a distributed network have complete instances of the model, and each node receives different portions of the data. The results from the different nodes are then combined. While different methods of data parallelism are possible, data parallelism training requires a technique to combine the results and synchronize the model parameters between the nodes. Exemplary schemes for combining data include parameter averaging and update-based data parallelism. Parameter averaging trains the nodes on a subset of the training data and sets the global parameters (e.g., weights, biases) as the average of the parameters from each node. Parameter averaging uses a central parameter server that stores the parameter data. Update-based data parallelism is similar to parameter averaging, except that updates to the model are transferred instead of parameters being transferred from the nodes to the parameter server. Furthermore, update-based data parallelism can be performed in a distributed manner, where the updates are compressed and transferred between nodes.

[0232] The combined model and data parallelism 3006 can be implemented, for example, in a distributed system where each computing node includes multiple GPUs. Each node can have a complete instance of the model, wherein separate GPUs within each node are used to train different parts of the model.

[0233] Distributed training carries an increased workload compared to training on a single machine. However, the parallel processors and GPGPUs described herein can each implement various techniques to reduce the workload of distributed training, including techniques for enabling high-bandwidth GPU-to-GPU data transfer and accelerating remote data synchronization. Exemplary Machine Learning Applications

[0234] Machine learning can be applied to solve a variety of technical problems, including but not limited to computer vision, autonomous driving and navigation, speech recognition, and language processing. Computer vision has traditionally been one of the most active research areas for machine learning applications. Applications of computer vision range from reproducing human visual abilities (such as recognizing faces) to generating new types of visual abilities. For example, computer vision applications can be configured to identify sound waves from vibrations induced in objects visible in a video feed. Parallel processors accelerate machine learning systems, enabling computer vision applications to use significantly larger training datasets than previous approaches, and causing inference systems to be deployed using low-power parallel processors.

[0235] Parallel processors accelerate machine learning in autonomous driving applications, including lane and road sign recognition, obstacle avoidance, navigation, and driving control. Accelerated machine learning techniques can be used to train driving models based on datasets that define appropriate responses to specific training inputs. The parallel processors described herein enable rapid training of increasingly complex neural network-like systems used in autonomous driving solutions and facilitate the deployment of low-power inference processors suitable for integration into mobile platforms within autonomous vehicles.

[0236] Parallel processors have accelerated deep neural networks, enabling machine learning solutions for Automatic Speech Recognition (ASR). ASR comprises functions that generate the most probable language sequence given a sequence of input sound waves. The use of deep neural networks to accelerate machine learning has led to the replacement of previously used Hidden Markov Models (HMMs) and Gaussian Mixture Models (GMMs) for ASR.

[0237] Parallel processors can also be used to accelerate machine learning and natural language processing. Automated learning programs can utilize statistical inference algorithms to generate models that are resistant to errors or unfamiliar inputs. Exemplary natural language processor applications include automated machine translation between human languages.

[0238] Parallel processing platforms for machine learning can be divided into training platforms and deployment platforms. Training platforms are typically highly parallel and include optimizations to accelerate multi-GPU single-node training and multi-node multi-GPU training. Exemplary parallel processors suitable for training include the highly parallel general-purpose graphics processing units and / or multi-GPU computing systems described herein. Conversely, deployed machine learning platforms typically include lower-power parallel processors suitable for use in products such as cameras, autonomous robots, and autonomous vehicles.

[0239] Figure 30B shows a paradigmatic inference system single-chip (SOC) 3100 suitable for performing inference using a trained model. The elements of Figure 30B, having the same or similar names as those in any other figures herein, describe the same elements as in the other figures, can operate or function in a similar manner, may contain the same elements, and may be linked to other entities as elsewhere herein, but are not limited thereto. The SOC 3100 may integrate processing components including a media processor 3102, a vision processor 3104, a GPGPU 3106, and a multi-core processor 3108. The SOC 3100 may additionally include on-chip memory 3105, which enables a shared pool of on-chip data accessible by the various processing components. The processing components may be optimized for low-power operation, enabling deployment on various machine learning platforms, including autonomous vehicles and autonomous robots. For example, one implementation of the SOC 3100 may be used as part of the main control system of an autonomous vehicle. When the SOC 3100 is configured for use in autonomous vehicles, the SOC is designed and configured to comply with the relevant functional safety standards of the deployment jurisdiction.

[0240] During operation, the media processor 3102 and the vision processor 3104 can work together to accelerate computer vision operations. The media processor 3102 can cause low-latency decoding of multiple high-resolution (e.g., 4K, 8K) video streams. The decoded video streams can be written to a buffer in on-chip memory 3105. The vision processor 3104 can then parse the decoded video and perform preliminary processing operations on the frames of the decoded video to prepare them for processing using a trained image recognition model. For example, the vision processor 3104 can accelerate the convolution operations of a CNN used to perform image recognition on high-resolution video data, while the back-end model computation is performed by the GPGPU 3106.

[0241] The multi-core processor 3108 may include control logic for ordering and synchronizing data transfer and shared memory operations performed by the media processor 3102 and the vision processor 3104. The multi-core processor 3108 may also function as an application processor for executing software applications that can utilize the inference computing capabilities of the GPGPU 3106. For example, at least a portion of navigation and driving logic may be implemented in software executed on the multi-core processor 3108. This software may directly send computational workloads to the GPGPU 3106, or the computational workloads may be sent to the multi-core processor 3108, which may offload at least a portion of those operations to the GPGPU 3106.

[0242] The GPGPU 3106 may include compute clusters, such as low-power configurations of compute clusters DPLAB06A-DPLAB06H within the highly parallel general-purpose graphics processing unit DPLAB00. The compute clusters within the GPGPU 3106 can support instructions that are explicitly optimized to perform inference computations on trained neural networks. For example, the GPGPU 3106 can support instructions for performing low-precision computations such as 8-bit and 4-bit integer vector operations. Ray Tracing Architecture

[0243] In one implementation, the graphics processor includes circuitry and / or code for performing real-time ray tracing. A set of dedicated ray tracing cores may be included in the graphics processor to perform the various ray tracing operations described herein, including ray traversal and / or ray intersection operations. In addition to the ray tracing cores, multiple sets of graphics processing cores may be included for performing programmable shading operations and multiple sets of tensor cores for performing matrix operations on tensor data.

[0244] Figure 31 shows an exemplary portion of such a graphics processing unit (GPU) 3105, which includes a dedicated set of graphics processing resources arranged in multi-core groups 3100A-N. The GPU 3105 may be a variant of the graphics processor 300, the GPGPU 1340, and / or any other graphics processor described herein. Thus, the disclosure of any features for the graphics processor also discloses corresponding combinations with the GPU 3105, but is not limited thereto. Furthermore, the elements of Figure 31 having the same or similar names as elements in any other figures herein are described as the same elements as those in the other figures, may operate or function in a similar manner, may contain the same components, and may be linked to other entities as described elsewhere herein, but are not limited thereto. Although only detailed information about a single multi-core group 3100A is provided, it should be understood that other multi-core groups 3100B-N may be equipped with the same or similar sets of graphics processing resources.

[0245] As shown in the figure, the multi-core group 3100A may include a set of graphics cores 3130, a set of tensor cores 3140, and a set of ray tracing cores 3150. A scheduler / dispatcher 3110 schedules and dispatches graphics threads to execute on the various cores 3130, 3140, and 3150. A set of register files 3120 stores the operand values ​​used by the cores 3130, 3140, and 3150 when executing graphics threads. These may include, for example, integer registers for storing integer values, floating-point registers for storing floating-point values, vector registers for storing compressed data elements (integer and / or floating-point data elements), and slice registers for storing tensor / matrix values. Slice registers may be implemented as a combined collection of vector registers.

[0246] One or more Level 1 (L1) caches and texture units 3160 store graphics data, such as texture data, vertex data, pixel data, ray data, bounding volume data, etc., in regions within each multi-core group 3100A. A Level 2 (L2) cache 3180, shared by all or a subset of the multi-core groups 3100A-N, stores graphics data and / or instructions for multiple concurrent graphics threads. As shown, the L2 cache 3180 can be shared among multiple multi-core groups 3100A-N. One or more memory controllers 3170 couple the GPU 3105 to memory 3198, which may be system memory (e.g., DRAM) and / or dedicated graphics memory (e.g., GDDR6 memory).

[0247] Input / output (I / O) circuitry 3195 couples GPU 3105 to one or more I / O devices 3195, such as digital signal processors (DSPs), network controllers, or user input devices. On-chip interconnects can be used to couple I / O devices 3190 to GPU 3105 and memory 3198. One or more I / O memory management units (IOMMUs) 3170 of I / O circuitry 3195 directly couple I / O devices 3190 to system memory 3198. IOMMU 3170 can manage multiple sets of page tables to map virtual addresses to physical addresses in system memory 3198. Furthermore, I / O devices 3190, CPU 3199, and GPU 3105 can share the same virtual address space.

[0248] The IOMMU 3170 can also support virtualization. In this case, it can manage a first set of page tables for mapping guest / graphics virtual addresses to guest / graphics entity addresses, and a second set of page tables for mapping guest / graphics entity addresses to system / host entity addresses (e.g., in system memory 3198). The base address of each of the first and second sets of page tables can be stored in a control register and swapped out on a context switcher (e.g., to provide access to the relevant set of page tables to a new context). Although not shown in Figure 31, each core 3130, 3140, 3150 and / or multi-core group 3100A-N can include a translation back buffer (TLB) to cache guest virtual to guest entity translation, guest entity to host entity translation, and guest virtual to host entity translation.

[0249] The CPU 3199, GPU 3105, and I / O device 3190 can be integrated on a single semiconductor chip and / or chip package. The memory 3198 shown can be integrated on the same chip or coupled to the memory controller 3170 via an off-chip interface. In one implementation, memory 3198 includes GDDR6 memory that shares the same virtual address space with other physical system-level memory, although the basic principles of the invention are not limited to this particular implementation.

[0250] The Tensor Core 3140 may contain a plurality of execution units specifically designed to perform matrix operations, which are fundamental computational operations used for performing deep learning operations. For example, simultaneous matrix multiplication can be used for neural network training and inference. The Tensor Core 3140 can perform matrix processing using various operand precisions, including single-precision floating-point numbers (e.g., 32-bit), half-precision floating-point numbers (e.g., 16-bit), integer arrays (16-bit), byte arrays (8-bit), and half-byte arrays (4-bit). The implementation of the neural network can also extract features from each rendering scheme, potentially combining details from multiple frames to construct a high-quality final image.

[0251] In deep learning implementations, parallel matrix multiplication operations can be scheduled to be performed on the tensor core 3140. Training neural networks, in particular, requires a large number of matrix dot product operations. To handle the inner product formulas of N×N×N matrix multiplications, the tensor core 3140 can contain at least N dot product processing elements. Before matrix multiplication begins, a complete matrix is ​​loaded into a slice register, and at least one row of a second matrix is ​​loaded in each of the N cycles. N dot products are processed in each cycle.

[0252] Depending on the specific implementation, matrix elements can be stored with different precisions, including 16-bit words, 8-bit bytes (e.g., INT8), and 4-bit half-bytes (e.g., INT4). Different precision modes can be specified for the tensor core 3140 to ensure that the most efficient precision is used for different workloads (e.g., inference workloads that can tolerate quantization to bits and half-bytes).

[0253] For both real-time and non-real-time ray tracing implementations, the ray tracing core 3150 can be used to accelerate ray tracing operations. Specifically, the ray tracing core 3150 may include ray traversal / intersection circuitry for performing ray traversal using a bounded volume hierarchy (BVH) and identifying intersections between rays and primitives enclosed within the BVH volume. The ray tracing core 3150 may also include circuitry for performing depth testing and culling (e.g., using a Z-buffer or similar arrangement). In one implementation, the ray tracing core 3150 performs traversal and intersection operations consistent with the image denoising techniques described herein, at least a portion of which can be performed on the tensor core 3140. For example, the tensor core 3140 may implement a deep learning-like neural network to perform denoising on frames generated by the ray tracing core 3150. However, the CPU 3199, the graphics core 3130, and / or the ray tracing core 3150 may also implement all or part of the denoising and / or deep learning algorithms.

[0254] Furthermore, as described above, a distributed denoising method can be employed, wherein the GPU 3105 is coupled to other computing devices via a network or high-speed interconnect. The interconnected computing devices can additionally share neural network learning / training data to improve the speed at which the entire system learns to perform denoising for different types of image frames and / or different graphics applications.

[0255] The ray tracing core 3150 can handle all BVH traversals and ray-primitive intersections, thus avoiding overloading the graphics core 3130 with thousands of instructions per ray. Each ray tracing core 3150 may include a first set of dedicated circuitry for performing bounding box tests (e.g., for traversal operations) and a second set of dedicated circuitry for performing ray-triangle intersection tests (e.g., for traversed intersecting rays). Therefore, the multi-core group 3100A can simply start the ray detector, and the ray tracing core 3150 can independently perform ray traversal and intersection and return hit data (e.g., hit, miss, multiple hits, etc.) to the execution thread context. While the ray tracing core 3150 is performing traversal and intersection operations, other cores 3130, 3140 can be released to perform other graphics or computational tasks.

[0256] Each ray tracing core 3150 may include traversal units for performing BVH test operations and intersection units for performing ray-primitive intersection tests. The intersection units may then generate "hit," "miss," or "multiple hit" responses and provide them to the appropriate execution thread. During traversal and intersection operations, execution resources of other cores (e.g., graphics core 3130 and tensor core 3140) may be freed up to perform other forms of graphics work.

[0257] A hybrid rasterization / ray tracing method that operates between the graphics core 3130 and the ray tracing core 3150 can also be used.

[0258] The ray tracing core 3150 (and / or other cores 3130, 3140) may include hardware support for ray tracing instruction sets, such as Microsoft's DirectX Raytracing (DXR), which includes DispatchRays commands and shaders for ray generation, nearest hit, any hit, and miss, which can assign a unique set of shaders and textures to each object. Another ray tracing platform that may be supported by the ray tracing core 3150, graphics core 3130, and tensor core 3140 is Vulkan 1.1.85. However, it should be noted that the basic principles of the present invention are not limited to any particular ray tracing ISA.

[0259] Typically, various cores 3150, 3140, and 3130 can support ray tracing instruction sets containing instructions / functions for ray generation, nearest hit, any hit, ray-primitive intersection, primitive and hierarchy bounding box construction, miss, access, and exceptions. More specifically, ray tracing instructions can be included to perform the following functions:

[0260] Ray Generation - Ray generation instructions can be executed for each pixel, sample, or other user-defined job assignment.

[0261] Closest Hit - Can execute the closest hit command to locate the closest intersection point between a ray and a primitive in the scene.

[0262] Any Hit - The Any Hit command can identify multiple intersections between a ray and primitives in the scene, thus potentially identifying new nearest intersections.

[0263] Intersection - The intersection command performs a ray-primary intersection test and outputs the result.

[0264] Constructing a Boundary Box by Primitive - This instruction constructs a boundary box around a given primitive or group of primitives (e.g., when constructing a new BVH or other acceleration data structure).

[0265] Missing - Indicates that the ray missed all geometry in the scene or a specified area of ​​the scene.

[0266] Access - Indicates the sub-volume that the ray will traverse.

[0267] Exceptions - Contains various types of exception handlers (e.g., invoked for various error conditions).

[0268] The media processing circuit 3197 includes fixed-function circuitry for encoding, decoding, and transcoding media into one or more media encoding formats, or encoding, decoding, and transcoding media between one or more media encoding formats, including but not limited to Moving Picture Experts Group (MPEG) formats such as MPEG-2, Advanced Video Coding (AVC) formats such as H.264 / MPEG-4 AVC, and Society of Motion Picture and Television Engineers (SMPTE) 421M / VC-1, and Joint Group of Photographers (JPEG) formats such as JPEG and Motion JPEG (MJPEG). To process these formats, one embodiment of the media processing circuitry includes logic for performing scaling / quantization, intra-frame prediction, inter-frame prediction, motion compensation, and motion estimation.

[0269] The display processor 3193 may include a 2D display engine and a display controller. The display processor 3193 may also include specialized logic capable of operating independently of other circuitry of the GPU 3105. The display controller includes an interface for coupling to a display device (not shown), which may be a system-integrated display device, such as in a laptop computer, or an external display device connected via a display device connector. Layered ray tracing

[0270] Bounding volume hierarchies (BVHs) are typically used to improve the efficiency of operations performed on graphical primitives and other graphical objects. A BVH is a hierarchical tree structure built upon a set of geometric objects. At the top of the tree is a root node, which encloses all geometric objects in a given scene. Each geometric object is wrapped in a bounding volume that forms the leaf nodes of the tree. These nodes are then grouped into smaller sets and contained within larger bounding volumes. These are then recursively grouped and contained within other larger bounding volumes, ultimately resulting in a tree structure with a single bounding volume represented by a root node at the top. Bounding volume hierarchies are used to efficiently support various operations on a set of geometric objects, such as collision detection, primitive culling, and ray traversal / intersection operations used in ray tracing.

[0271] In the ray tracing architecture, a ray traverses the BVH to determine ray-primitive intersections. For example, if a ray does not pass through the root node of the BVH, then the ray will not intersect any primitives enclosed by the BVH, and no further processing of the ray is required for this set of primitives. If a ray passes through the first child node of the BVH but not the second child node, then there is no need to test the ray for any primitives contained in the second child node. Thus, the BVH provides an efficient mechanism for testing ray-primitive intersections.

[0272] Groups of continuous rays (referred to as "beams") can be tested for BVH rather than individual rays. Figure 32 shows an exemplary beam 3201 outlined by four distinct rays. Any ray intersecting the patch 3200 defined by the four rays is considered to be within the same beam. Although beam 3201 in Figure 32 is defined by a rectangular arrangement of rays, beams can be defined in various other ways while still conforming to the basic principles of the invention (e.g., circular, elliptical, etc.).

[0273] Figure 33 illustrates how the ray tracing engine 3310 of GPU 3320 implements the ray tracing technique described herein. Specifically, ray generation circuitry 3304 generates a plurality of rays to which traversal and intersection operations are to be performed. However, instead of performing traversal and intersection operations on a single ray, traversal and intersection operations are performed using a hierarchy of beams 3307 generated by beam hierarchy construction circuitry 3305. The beam hierarchy is analogous to a bounded volume hierarchy (BVH). For example, Figure 34 provides an example of a master beam 3400 that can be subdivided into a plurality of distinct components. Specifically, the master beam 3400 can be divided into quadrants 3401-3404, and each quadrant itself can be divided into sub-quadrants, such as sub-quadrants AD within quadrant 3404. The master beam can be subdivided in various ways. For example, the master beam can be divided in half (instead of quadrants), and each half can be divided in half, and so on. Regardless of the subdivision, a hierarchical structure is generated in a similar manner to BVH. For example, there is a root node representing the main beam 3400, the first level of the sub-nodes is represented by quadrants 3401-3404, the second level of the sub-nodes is represented by sub-quadrants AD, and so on.

[0274] Once the beam hierarchy 3307 is constructed, the traversal / intersection circuit 3306 can perform traversal / intersection operations using the beam hierarchy 3307 and the BVH 3308. Specifically, beams can be tested against the BVH and the culling portions of beams that do not intersect with any part of the BVH. Using the data shown in FIG34, for example, if sub-beams associated with sub-regions 3402 and 3403 do not intersect with the BVH or a specific branch of the BVH, they can be culled relative to the BVH or said branch. The remaining portions 3401, 3404 can be tested against the BVH by performing a depth-first search or other search algorithms.

[0275] Figure 35 shows a method for ray tracing. The method can be implemented in the context of the graphics processing architecture described above, but is not limited to any particular architecture.

[0276] At 3500, a master beam containing a plurality of rays is constructed, and at 3501, the beam is subdivided, generating a hierarchical data structure to establish a beam hierarchy. Operations 3500-3501 can be performed as a single integration operation that constructs the beam hierarchy from the plurality of rays. At 3502, the beam hierarchy is used with BVH to cull rays (from the beam hierarchy) and / or nodes / primitives from BVH. At 3503, ray-primitive intersections are determined for the remaining rays and primitives. Lossy and lossless packet compression in a distributed ray tracing system.

[0277] Ray tracing operations can be distributed across multiple compute nodes coupled together via a network. For example, Figure 36 shows a ray tracing cluster 3600 containing multiple ray tracing nodes 3610-3613, which execute ray tracing operations in parallel, making it possible to combine results on one of the nodes. In the architecture shown, ray tracing nodes 3610-3613 are communicatively coupled to a client-side ray tracing application 3630 via a gateway.

[0278] One of the difficulties of a distributed architecture is the large amount of compressed data that must be transmitted between each ray tracing node 3610-3613. Both lossless and lossy compression techniques can be used to reduce the amount of data transmitted between ray tracing nodes 3610-3613.

[0279] To achieve lossless compression, instead of sending packets filled with the results of certain types of operations, data or commands that allow the receiving node to reconstruct the results are sent. For example, random sampling of area light and ambient occlusion (AO) operations do not necessarily require direction. Therefore, the sending node can simply send a random seed, which the receiving node then uses to perform random sampling. For example, if the scene is distributed across nodes 3610-3612 (to sample light 1 at points p1-p3), only the light ID and origin need to be sent to nodes 3610-3612. Each node can then independently random sample the light. The random seed can be generated by the receiving node. Similarly, for the main ray hit point, ambient occlusion (AO) and soft shadow sampling can be calculated on nodes 3610-3612 without waiting for the origin of the consecutive frames. Furthermore, if it is known that a group of rays will reach the same point light source, a command instructing the receiving node to apply the light source to the group of rays can be sent. As another example, if N ambient light-shielding rays pass through a single point, a command can be sent to generate N samples from that point.

[0280] Various additional techniques can be applied to lossy compression. For example, a quantization factor can be used to quantize all coordinate values ​​associated with BVH, primitives, and rays. Furthermore, 32-bit floating-point values ​​used for data (such as BVH nodes and primitives) can be converted to 8-bit integer values. In a paradigmatic implementation, the boundaries of the ray packet are stored with full precision, but individual ray points P1-P3 are projected onto the boundaries as index offsets. Similarly, multiple regional coordinate systems can be generated using 8-bit integer values ​​as regional coordinates. The positions of the origins of each of these regional coordinate systems can be encoded using full-precision (e.g., 32-bit floating-point) values, thereby effectively connecting the global coordinate system and the regional coordinate system.

[0281] The following is an example of lossless compression. An example of the ray data format used internally by a ray tracing program is as follows:

[0282] Instead of sending the original data of each node, the data can be compressed by grouping the values ​​and, where possible, creating implicit rays using applicable metadata.

[0283] Bundled and grouped ray data flags can be used for general data or masks with modifiers. For example: origins are shared.

[0284] All ray data is compressed, except that a single origin is stored in all rays. RayPacket.flags are set for RAYPACKET_COMMON_ORIGIN. When RayPacket is decompressed upon receipt, the origin is filled from the single origin value. The origin is shared only in certain rays.

[0285] All ray data, except for rays sharing a common origin, is compressed. For each unique set of shared origins, there is an operator for compression that identifies the operation (shared origin), stores the origin, and masks the ray sharing information. This can be done on any shared value between nodes (such as material ID, primitive ID, origin, direction, normal, etc.). Send implicit rays.

[0286] Typically, ray data can be obtained at the receiving end, while generating ray data using minimal metadata. A very common example is generating multiple auxiliary rays to randomly sample a region. Instead of the sender generating the auxiliary rays, the receiver sends and manipulates them; the sender can send a command stating that any relevant information is needed to generate the ray, and the ray is generated at the receiving end. In cases where it is necessary for the sender to first generate the ray to determine which receiver it is sent to, the ray is generated and a random seed can be sent to regenerate the exact same ray.

[0287] For example, to sample hit points, 64 shaded rays are used to sample the area light source, with all 64 rays intersecting regions from the same computation N4. A RayPacket with a common source and normal is established. More data could be sent if the pixel contribution from the receiver's shading is desired, but for this example, we assume we only want to return data on whether a ray hits another node. A RayOperation is established to generate the shaded ray operation, and it is assigned the lightID value to be sampled and a random number seed. When N4 receives the ray packet, it generates fully filled ray data by filling all rays with the shared original data and setting the direction based on the lightID randomly sampled using the random number seed, to produce the same ray as the original sender. When returning the results, only the binary result of each ray needs to be returned, which can be handled by masking the rays.

[0288] In this example, sending the original 64 rays would require 104 bytes * 64 rays = 6656 bytes. If the returned rays are also sent in their original format, this will double to 13312 bytes. Using lossless compression, only 29 bytes are sent for common ray sources, normals, and ray generation operations with seed and ID, while 8 bytes are returned for intersecting masks. This results in a data compression ratio of approximately 360:1 over the network. This does not include the overhead of processing the message itself, which needs to be identified in some way, but this depends on the implementation. Other operations can be performed to recalculate the ray origin and direction of the pixel ID from the primitive ray, recalculate the pixel ID based on the range in the ray packet, and many other possible implementations for recalculating values. Similar operations can be used on any single ray or group of rays sent, including shading, reflection, refraction, ambient occlusion, intersection, volumetric intersection, shading, bounce reflection in path tracing, etc.

[0289] Figure 37 shows additional details of the two ray tracing nodes 3710-3711, which perform compression and decompression of ray tracing packets. Specifically, when the first ray tracing engine 3730 is ready to transmit data to the second ray tracing engine 3731, the ray compression circuit 3720 performs lossy and / or lossless compression of the ray tracing data as described herein (e.g., converting 32-bit values ​​to 8-bit values, replacing the original data with instructions for reconstructed data, etc.). The compressed ray packet 3701 is transmitted from network interface 3725 to network interface 3726 via a local area network (e.g., a 10Gb / s, 100Gb / s Ethernet). The ray decompression circuit then decompresses the ray packet as appropriate. For example, it can execute commands to reconstruct the ray tracing data (e.g., using a random seed to perform random sampling for illumination operations). The ray tracing engine 3731 then uses the received data to perform ray tracing operations.

[0290] In the opposite direction, the ray compression circuit 3741 compresses the ray data, the network interface 3726 transmits the compressed ray data through the network (e.g., using the techniques described herein), the ray decompression circuit 3740 decompresses the ray data as necessary, and the ray tracing engine 3730 uses the data in ray tracing operations. Although shown as separate units in FIG. 37, the ray decompression circuits 3740-3741 can be integrated into the ray tracing engines 3730-3731 respectively. For example, in the case of commands to compress ray data including reconstructing ray data, these commands can be executed by each of the respective ray tracing engines 3730-3731.

[0291] As shown in FIG38, the X-ray compression circuit 3720 may include a lossy compression circuit 3801 to perform the lossy compression techniques described herein (e.g., converting 32-bit floating-point coordinates to 8-bit integer coordinates) and a lossless compression circuit 3803 to perform lossless compression techniques (e.g., sending commands and data to allow the X-ray recompression circuit 3821 to reconstruct the data). The X-ray decompression circuit 3721 includes a lossy decompression circuit 3802 and a lossless decompression circuit 3804 for performing lossless decompression.

[0292] Figure 39 shows another exemplary method. The method can be implemented on a ray tracing architecture or other architectures described herein, but is not limited to any particular architecture.

[0293] At 3900, ray data is received, which will be transmitted from the first ray tracing node to the second ray tracing node. At 3901, a lossy compression circuit performs lossy compression on the first ray tracing data, and at 3902, a lossless compression circuit performs lossless compression on the second ray tracing data. At 3903, the compressed ray tracing data is sent to the second ray tracing node. At 3904, a lossy / lossless decompression circuit performs lossy / lossless decompression of the ray tracing data, and at 3905, the second ray tracing node performs a ray tracing operation on the decompressed data. A graphics processor with hardware-accelerated hybrid ray tracing.

[0294] Next, a hybrid rendering pipeline is proposed, which performs rasterization on graphics core 3130 and ray tracing operations on ray tracing core 3150, graphics core 3130, and / or CPU core 3199. For example, instead of a main raycasting stage, rasterization and depth testing can be performed on graphics core 3130. Ray tracing core 3150 can then generate auxiliary rays for ray reflection, refraction, and shadowing. Furthermore, certain areas of the scene where ray tracing core 3150 will perform ray tracing operations will be selected (e.g., based on material property thresholds such as the degree of high reflectivity), while other areas of the scene will be rasterized and rendered on graphics core 3130. This hybrid implementation can be used in real-time ray tracing applications, where latency is a critical issue.

[0295] The ray traversal architecture described below can, for example, use existing Single Instruction Multiple Data (SIMD) and / or Single Instruction Multiple Thread (SIMT) graphics processors to perform programmable shading and ray traversal control, while using dedicated hardware acceleration for critical functions such as BVH traversal and / or intersection. SIMD occupancy can be improved by recombining the generated shaders at specific points during traversal and before shading. This is achieved by dynamically sorting shaders on-chip using dedicated hardware. SIMD occupancy is further improved by managing recursion by splitting functions into consecutive parts that are executed before being returned to and recombined.

[0296] Programmable control of ray traversal / intersection is achieved by decomposing the traversal function into an internal traversal that can be implemented as fixed-function hardware and an external traversal that is executed on a GPU processor and is programmable through a user-defined traversal shader. By conservatively truncating the internal traversal state during the transition between internal and external traversals, the cost of passing the traversal context between hardware and software can be reduced.

[0297] Programmable control of ray tracing can be represented by the different shader types listed in Table A below. Each type can have multiple shaders. For example, each material can have a different hit shader. Shader type Functionality host Emit main rays Hit Bidirectional reflection distribution function (BRDF) sampling, emission of auxiliary rays Any hit Calculate the transmittance of alpha texture geometry miss Calculate the radiation from the light source intersect Intersecting custom shapes Traversal Instance selection and transformation Callable General functions Table A

[0298] Recursive ray tracing can be initiated via API functions that instruct the graphics processor to launch a set of primary shaders or intersection circuits, which can generate ray scene intersections for the primary ray. Other shaders, such as traversal, hit, or miss shaders, are generated sequentially. The shader that generates the child shaders can also receive return values ​​from the child shaders. Callable shaders are general-purpose and can be generated directly by another shader or return values ​​to the calling shader.

[0299] Figure 40 shows a graphics processing architecture including shader execution circuitry 4000 and fixed-function circuitry 4010. The general-purpose execution hardware subsystem includes multiple Single Instruction Multiple Data (SIMD) and / or Single Instruction Multiple Thread (SIMT) cores / executive units (EUs) 4001 (that is, each core may contain multiple execution units), one or more samplers 4002, and a first-order (L1) cache 4003 or other forms of region memory. The fixed-function hardware subsystem 4010 includes a message unit 4004, a scheduler 4007, a ray BVH traversal / intersection circuit 4005, a sorting circuit 4008, and a region L1 cache 4006.

[0300] In operation, the master dispatcher 4009 dispatches a set of master rays to the scheduler 4007, which schedules the work to shaders that execute on the SIMD / SIMT core / EU 4001. The SIMD core / EU 4001 may be the ray-tracing core 3150 and / or the aforementioned graphics core 3130. The execution of the master shader generates additional work to be performed (e.g., to be performed by one or more sub-shaders and / or fixed-function hardware). The message unit 4004 distributes the work generated by the SIMD core / EU 4001 to the scheduler 4007, accesses the required idle stack pool, the sorting circuit 4008, or the ray BVH intersection circuit 4005. If additional work is sent to the scheduler 4007, it is scheduled for processing on the SIMD / SIMT core / EU 4001. Prior to scheduling, sorting circuit 4008 may sort rays into groups or binaries as described herein (e.g., grouping rays with similar properties). Ray BVH intersection circuit 4005 uses BVH volumes to perform ray intersection tests. For example, ray BVH intersection circuit 4005 may compare ray coordinates with each level of BVH to identify volumes intersected by the rays.

[0301] A shader record, a user-assigned structure, can be used to reference the shader. This user-assigned structure contains an index pointing to the incident function, vendor-specific metadata, and global parameters of the shader executed by the SIMD core / EU 4001. Each execution instance of the shader is associated with a call stack, which can be used to store parameters passed between parent and child shaders. The call stack can also store a reference to the continued functionality executed upon call return.

[0302] Figure 41 shows a set of example-allocated stacks 4101, which includes a main shader stack, a hit shader stack, a traversal shader stack, a continuation function stack, and a ray BVH intersection stack (as described, which can be executed by fixed-function hardware 4010). New shader calls can implement new stacks from the idle stack pool 4102. The call stacks (e.g., stacks consisting of a set of specified stacks) can be cached in region L1 caches 4003, 4006 to reduce access latency.

[0303] There may be a finite number of call stacks, each allocated a fixed maximum size "Sstack" in a contiguous region of memory. Therefore, the base address of the stack can be calculated directly from the stack index (SID), since base address = SID * Sstack. When a job is scheduled to SIMD core / EU 4001, the stack ID can be allocated and released by the scheduler 4007.

[0304] The master dispatcher 4009 may include a graphics processor command processor that dispatches the master shader in response to dispatch commands from the host (e.g., CPU). If the scheduler 4007 can assign a stack ID to each SIMD channel, the scheduler 4007 can receive these dispatch requests and start the master shader on the SIMD processor thread. Stack IDs can be assigned from an idle stack pool 4102 initialized at the beginning of the dispatch command.

[0305] The execution shader can generate a child shader by sending a generation message to the messaging unit 4004. This command contains the stack ID associated with the shader and also contains an index pointing to the child shader record for each active SIMD channel. The parent shader can only send this message once for an active channel. After sending generation messages for all relevant channels, the parent shader may terminate.

[0306] The shader executing on the SIMD core / EU 4001 can also generate fixed-function tasks such as ray BVH intersections using the generated messages and shader record indicators reserved for fixed-function hardware. As mentioned, the message passing unit 4004 sends the generated ray BVH intersection task to the fixed-function ray BVH intersection circuit 4005 and sends the callable shader directly to the sorting circuit 4008. The sorting circuit can group the shaders by shader record indicators to obtain SIMD batches with similar characteristics. Therefore, stack IDs from different parent shaders can be grouped in the same batch by the sorting circuit 4008. The sorting circuit 4008 sends the grouped batches to the scheduler 4007, which accesses the shader record from the graphics memory 2511 or the last-level cache (LLC) 4020 and starts the shader on the processor thread.

[0307] A continuation can be considered a callable shader and can also be referenced through a shader record. When a child shader is created and its value is returned to the parent shader, a pointer to the continuation shader record can be pushed onto the call stack 4101. When the child shader returns, the continuation shader record can be popped from the call stack 4101, and a continuation shader may be created. Optionally, the created continuation can be created through a sorting unit similar to that of a callable shader and started on a processor thread.

[0308] As shown in Figure 42, the sorting circuit 4008 groups the generated tasks to establish SIMD batches for coloring using shader recording indices 4201A, 4201B, and 4201n. Stack IDs or context IDs in the sorted batches can be grouped from different assignments and different input SIMD channels. The grouping circuit 4210 can perform sorting using a Content Addressable Memory (CAM) structure 4201 containing a plurality of entries, each entry being identified by a tag 4201. As described above, the tag 4201 can be the corresponding shader recording indices 4201A, 4201B, and 4201n. The CAM structure 4201 can store a limited number of tags (e.g., 32, 64, 128, etc.), each tag associated with an incomplete SIMD batch corresponding to a shader recording index.

[0309] For each incoming generation command, each SIMD channel has a corresponding stack ID (displayed as 16 context IDs 0-15 in each CAM entry) and shader recording indicators 4201A-B, ...n (used as tag values). The grouping circuit 4210 can compare the shader recording indicator of each channel with the tag 4201 in the CAM structure 4201 to find a matching batch. If a matching batch is found, the stack ID / context ID can be added to said batch. Otherwise, a new entry with a new shader recording indicator tag may be created, potentially discarding older entries with incomplete batches.

[0310] The execution shader can cancel the call stack when it is empty by sending a cancellation message to a message unit. The cancellation message is relayed to the scheduler, which returns the stack ID / context ID of the active SIMD channel to the idle pool.

[0311] A hybrid approach for ray traversal operations, combining fixed-function ray traversal and soft ray traversal, is proposed. This provides the flexibility of soft ray traversal while maintaining the efficiency of fixed-function traversal. Figure 43 shows an accelerated structure that can be used for hybrid traversal, which is a two-level tree with a single top-level BVH 4300 and several bottom-level BVHs 4301 and 4302. Graphical elements are shown on the right to indicate the inner traversal path 4303, the outer traversal path 4304, the traversal node 4305, the leaf node 4306 with a triangle, and the leaf node 4307 with a custom primitive.

[0312] Leaf nodes 4306 with triangles in the top-level BVH 4300 can reference triangles, intersection shader records of custom primitives, or traversal shader records. Leaf nodes 4306 with triangles in the bottom-level BVHs 4301-4302 can only reference triangles and intersection shader records of custom primitives. The type of reference is encoded in leaf nodes 4306. Internal traversal 4303 refers to the traversal within each BVH 4300-4302. Internal traversal operations involve the calculation of ray BVH intersections, and traversal across BVH structures 4300-4302 is called external traversal. Internal traversal operations can be implemented efficiently in fixed-function hardware, while external traversal operations can be performed with acceptable performance using programmable shaders. Therefore, the fixed-function circuit 4010 can be used to perform the internal traversal operation, and the external traversal operation can be performed using the shader execution circuit 4000, which includes the SIMD / SIMT core / EU 4001 for performing programmable shaders.

[0313] Note that, for simplicity, the SIMD / SIMT core / EU 4001 is sometimes referred to herein as "core," "SIMD core," "EU," or "SIMD processor." Similarly, the ray BVH traversal / intersection circuit 4005 is sometimes referred to simply as "traversal unit," "traversal / intersection unit," or "traversal / intersection circuit." When alternative terms are used, as described herein, the specific name used to specify the corresponding circuit / logic does not change the basic function performed by the circuit / logic.

[0314] Furthermore, although shown as a single component in FIG40 for illustrative purposes, the traversal / intersection unit 4005 may comprise different traversal units and separate intersection units, each of which may be implemented in circuits and / or logic as described herein.

[0315] A traversal shader can be generated when a ray intersects a traversal node during its internal traversal. The sorting circuit 4008 can group these shaders via shader record indices 4201A-B,n to create a SIMD batch initiated by the scheduler 4007 to execute SIMD on the graphics SIMD core / EU 4001. The traversal shader can be modified in several ways to support a wide range of applications. For example, the traversal shader can select a coarser level of detail (LOD) BVH or transform the ray to achieve a rigid body transformation. The traversal shader can then generate an internal traversal of the selected BVH.

[0316] The inner traversal calculates the ray BVH intersection by traversing BVH and calculating the ray-box and ray-triangle intersections. The inner traversal is generated in the same way as the shader by sending a message to the message transmission circuit 4004, and the message transmission circuit 4004 relays the corresponding generation message to the ray BVH intersection circuit 4005 that calculates the ray BVH intersection.

[0317] The stack used for internal traversal can be regionally stored in fixed-function circuitry 4010 (e.g., within L1 cache 4006). Internal traversal may terminate when a ray intersects a leaf node corresponding to a traversal shader or intersection shader, and the internal stack is truncated. The truncated stack, along with pointers to the ray and BVH, can be written to memory at the location specified by the calling shader, and the corresponding traversal or intersection shader can then be generated. If the ray intersects any triangle during internal traversal, the corresponding hit information can be provided as input parameters to these shaders, as shown in the following code. These generated shaders can be grouped by sorting circuitry 4008 to create SIMD batches for execution.

[0318] Truncating the internal stack traversal reduces the cost of overflowing it into memory. The method described on pages 107–111 of "Restart Trail for Stackless BVH Traversal, High Performance Graphics (2010)" truncates the stack to a small number of entries at the top of the stack, where a 42-bit restart trace and a 6-bit depth value can be applied. The restart trace indicates the branch that has been performed internally in the BVH, and the depth value indicates the traversal depth corresponding to the last stack entry. This is sufficient information to resume the internal traversal later.

[0319] The inner traversal is complete when the inner stack is empty and there are no more BVH nodes to test. In this case, an outer stack handler is generated, which pops the top of the outer stack and resumes the traversal if the outer stack is not empty.

[0320] The external traversal can execute the main traversal state machine and can be implemented in the code executed by the shader execution circuit 4000. The internal traversal query can be generated under the following conditions: (1) when a new ray is generated by the hit shader or the main shader; (2) when the traversal shader selects the BVH for traversal; and (3) when the external stack handler resumes the internal traversal of the BVH.

[0321] As shown in Figure 44, prior to the internal traversal, space is allocated to the call stack 4405 for the fixed-function circuitry 4010 to store the truncated internal stack 4410. Offsets 4403-4404 of the top of the call stack and the internal stack are maintained in traversal state 4400, which is also stored in memory 2511. Traversal state 4400 also contains hit information for rays in world space 4401 and object space 4402, as well as the nearest intersecting primitives.

[0322] The traversal shader, intersection shader, and outer stack handler are all generated by the ray BVH intersection circuit 4005. The traversal shader is allocated on the call stack 4405 before initiating a new internal traversal for the second-level BVH. The outer stack handler is the shader responsible for updating hit information and resuming all unfinished internal traversal tasks. After traversal is complete, the outer stack handler is also responsible for generating hit or miss shaders. Traversal is complete when no pending internal traversal queries are generated. After traversal is complete and an intersection is found, a hit shader is generated; otherwise, a miss shader is generated.

[0323] Although the above hybrid traversal scheme uses two levels of BVH hierarchy, it is also possible to implement any number of BVH hierarchy and make corresponding changes in the external traversal implementation.

[0324] Furthermore, although the fixed-function circuit 4010 for performing ray BVH intersection has been described above, other system components can also be implemented in the fixed-function circuit. For example, the aforementioned external stacking process can be an internal (user-invisible) shader, which can potentially be implemented in the fixed-function BVH traversal / intersection circuit 4005. This implementation can be used to reduce the number of shader stages dispatched and the round trips between the fixed-function intersection hardware 4005 and the processor.

[0325] The examples described herein use user-defined features to implement programmable coloring and ray traversal control, which can be executed with higher SIMD efficiency on current and future GPU processors. Programmable control of ray traversal implements several important features, such as program instantiation, random level of detail selection, custom primitive intersection, and lazy BVH updates.

[0326] A programmable Multiple Instruction Multiple Data (MIMD) ray tracing architecture is also provided, which supports speculative execution of hit and intersection shaders. In particular, the architecture focuses on reducing the scheduling and communication load between the programmable SIMD / SIMT core / execution unit 4001 described above with respect to Figure 40 and the fixed-function MIMD traversal / intersection unit 4005 in the hybrid ray tracing architecture. Several speculative execution schemes for hit and intersection shaders are described below, which can be dispatched from the traversal hardware in a single batch, thereby avoiding multiple traversal and shading round trips. Dedicated circuitry can be used to implement these techniques.

[0327] Embodiments of the present invention are particularly advantageous in use cases where multiple hit or intersection shaders need to be executed from ray traversal queries, which would impose a considerable load when there is no dedicated hardware support to implement them. These include, but are not limited to, the nearest k-hit query (which starts a hit shader for the k nearest intersections) and multiple programmable intersection shaders.

[0328] The techniques described herein can be implemented as an extension of the architecture shown in Figure 40 (and described with respect to Figures 40-44). In particular, the current embodiments of the invention are based on the architecture with enhancements to improve the performance of the above-described use cases.

[0329] The performance limitation of the hybrid ray traversal architecture is the load of initiating traversal queries from the execution unit and the load of calling programmable shaders from the ray traversal hardware. This load creates an "execution round trip" between the programmable core 4001 and the traversal / intersection unit 4005 when multiple hit or intersection shaders are called during the traversal of the same ray. This also puts additional stress on the sorting unit 4008, which needs to extract SIMD / SIMT consistency from the individual shader calls.

[0330] Several states of ray tracing require programmable control, which can be represented through the different shader types listed in Table A above (i.e., major, hit, any hit, miss, intersection, traversal, and callable). Each type can have multiple shaders. For example, each material can have a different hit shader. Some shader types are defined in the current Microsoft® Ray Tracing API.

[0331] As a brief recap, recursive ray tracing is initiated by an API function that instructs the GPU to launch a set of primary shaders that can generate ray intersections with the primary ray (implemented in hardware and / or software). This can then generate other shaders, such as traversal, hit, or miss shaders. The shader that generates the child shaders can also receive return values ​​from the shaders. Callable shaders are general-purpose functions that can be generated directly by another shader or return values ​​to the calling shader.

[0332] Ray traversal calculates ray scene intersections by traversing nodes in the intersection bounded volume hierarchy (BVH). Recent research indicates that using techniques better suited to fixed-function hardware (such as reduced-precision arithmetic, BVH compression, per-ray state machines, dedicated intersection pipelines, and custom caches) can improve the efficiency of calculating ray scene intersections by more than an order of magnitude.

[0333] The architecture shown in Figure 40 includes such a system, in which an array of SIMD / SIMT core / execution units 4001 interacts with fixed-function ray tracing / intersection units 4005 to perform programmable ray tracing. Programmable shaders are mapped to SIMD / SIMT threads on the execution units / cores 4001, where SIMD / SIMT utilization, execution, and data consistency are critical for optimal performance. Ray queries often compromise consistency for various reasons, such as: ․ Traversal discrepancies: In rays that favor asynchronous ray processing, the duration of BVH traversals varies greatly. ․ Execution discrepancies: Rays generated from different channels of the same SIMD / SIMT thread may result in different shader calls. ․ Data access discrepancies: For example, rays hitting different surfaces sample different BVH nodes and primitives, and shaders access different textures. Many other scenarios can also cause data access discrepancies.

[0334] The SIMD / SIMT core / execution unit 4001 may be a variant of the core / execution unit described herein, including graphics cores 415A-415B, shader cores 1355A-N, graphics core 3130, graphics execution unit 608, execution unit 852A-B, or any other core / execution unit described herein. The SIMD / SIMT core / execution unit 4001 may also replace graphics cores 415A-415B, shader cores 1355A-N, graphics core 3130, graphics execution unit 608, execution unit 852A-B, or any other core / execution unit described herein. Therefore, the disclosure of any feature in combination with graphics cores 415A-415B, shader cores 1355A-N, graphics core 3130, graphics execution unit 608, execution unit 852A-B, or any other core / execution unit described herein also discloses a corresponding combination with the SIMD / SIMT core / execution unit 4001 of FIG40, but is not limited thereto.

[0335] The fixed-function ray tracing / intersection unit 4005 can overcome the first two challenges by processing each ray individually and out of order. However, this would disrupt the SIMD / SIMT grouping. Therefore, the sorting unit 4008 is responsible for forming new, consistent SIMD / SIMT groups of shader calls to reassign them to the execution unit.

[0336] The advantages of this architecture are readily apparent compared to a purely software-based ray tracing implementation directly on a SIMD / SIMT processor. However, there is a message-passing load between the SIMD / SIMT core / execution unit 4001 (sometimes referred to herein simply as the SIMD / SIMT processor or core / EU) and the MIMD traversal / intersection unit 4005. Furthermore, the sorting unit 4008 may not be able to extract perfect SIMD / SIMT utilization from disjointed shader calls.

[0337] Use cases where shaders may be called particularly frequently during traversal can be identified. Enhancements for the hybrid MIMD ray tracing processor are described to significantly reduce the communication load between the core / EU 4001 and the traversal / intersection unit 4005. This can be particularly beneficial when finding the k closest intersection points and implementing a programmable intersection shader. However, please note that the techniques described herein are not limited to any particular processing scheme.

[0338] The following is a summary of the high-order cost of ray tracing context switching between the core / EU 4001 and the fixed-function traversal / intersection unit 4005. Most of the performance overhead is caused by these two context switches whenever a shader call is required during a single ray traversal.

[0339] Each SIMD / SIMT channel emitting a ray generates a production message to the traversal / intersection unit 4005 associated with the traversed BVH. Data (ray traversal context) is relayed to the traversal / intersection unit 4005 via the production message and (cache) memory. When the traversal / intersection unit 4005 is ready to allocate a new hardware thread to the production message, it loads the traversal state and performs the traversal on the BVH. A setup fee is also required before the first traversal step on the BVH.

[0340] Figure 45 shows the operation flow of the programmable ray tracing pipeline. The shading element, which includes traversal 4502 and intersection point 4503, can be implemented in a fixed-function circuit, while the remaining elements can be implemented by a programmable core / execution unit.

[0341] The main ray shader 4501 sends work to the traversal circuit at 4502, which traverses the current ray through the BVH (or other acceleration structure). When a leaf node is reached, the traversal circuit calls the intersection circuit at 4503, which, upon recognizing a ray-triangle intersection, calls any hit shader at 4504 (which can provide the result back to the indicated traversal circuit).

[0342] Alternatively, the traversal can terminate before reaching a leaf node, and the nearest hit shader is called at 4507 (if a hit is recorded) or the miss shader is called at 4506 (in the event of a miss).

[0343] As shown in 4505, if the traversal circuit reaches a custom primitive leaf node, the intersection shader can be invoked. The custom primitive can be any non-triangular primitive, such as a polygon or polyhedron (e.g., tetrahedron, solid pixel, hexahedron, wedge, pyramid, or other "unstructured" volume). The intersection shader 4505 identifies any intersection between the ray and the custom primitive and assigns it to any hit shader 4504 that implements any hit processing.

[0344] When the hardware traversal 4502 reaches the programmable stage, the traversal / intersection unit 4005 can generate shader dispatch messages to the associated shaders 4505-4507, which correspond to a single SIMD channel of the execution unit used to execute the shader. Since dispatches occur in an arbitrary order of rays and are divergent in the called program, the sorting unit 4008 can accumulate multiple dispatch calls to extract coherent SIMD batches. The traversal / intersection unit 4005 can write the updated traversal state and optional shader parameters to memory 2511.

[0345] In the k closest intersection problem, the nearest hit shader is executed for the first k intersections 4507. Traditionally, this would mean ending ray traversal, calling the hit shader, and generating a new ray from the hit shader to find the next closest intersection (with an offset from the ray origin so the same intersection doesn't happen again). Clearly, this implementation requires generating k rays for a single ray. Another implementation uses insertion sort, operating on any-hit shaders 4504, calling them for all intersections and maintaining a global list of closest intersections. The main problem with this approach is that there is no upper bound on the number of any-hit shader calls.

[0346] As mentioned, the intersection shader 4505 can be invoked on non-triangle (custom) primitives. Depending on the result of the intersection test and the traversal state (the node to be processed intersects with the primitive), traversal of the same ray may continue after the intersection shader 4505 is executed. Therefore, finding the closest hit point may require multiple round trips to the execution unit.

[0347] By changing the traversal hardware and shader scheduling model, the focus can also be placed on reducing SIMD-MIMD context switching between intersecting shaders 4505 and hitting shaders 4504, 4507. First, the ray traversal circuit 4005 defers shader calls by accumulating multiple potential calls and dispatching them in larger batches. Furthermore, some calls that become unnecessary may be discarded at this stage. Moreover, the shader scheduler 4007 can aggregate multiple shader calls from the same traversal context into a single SIMD batch, causing a single ray to generate a message. In a paradigmatic implementation, the traversal hardware 4005 suspends the traversal thread and waits for the results of multiple shader calls. This mode of operation is referred to herein as "speculatory" shader execution because it allows dispatching multiple shaders, some of which might not be called when using sequential calls.

[0348] Figure 46A shows an example of a traversal operation encountering multiple custom primitives 4650 in a subtree, while Figure 46B shows how to solve this problem using three intersecting dispatch cycles C1-C3. Specifically, the scheduler 4007 may require three cycles to submit the work to the SIMD processor 4001, and the traversal circuitry 4005 requires three cycles to provide the results to the sorting unit 4008. The traversal state 4601 required by the traversal circuitry 4005 can be stored in memory such as a region cache (e.g., L1 cache and / or L2 cache). A. Deferring ray tracing shader calls

[0349] The hardware traversal state 4601 can also be modified to allow the accumulation of multiple potential intersection or hit calls in the list. At a given time during traversal, each entry in the list can be used to generate a shader call. For example, k closest intersections can be accumulated on traversing hardware 4005 and / or in traversal state 4601 in memory, and a hit shader can be called for each element once the traversal is complete. For hit shaders, multiple potential intersections may accumulate in subtrees within the BVH.

[0350] For the use case closest to k, the advantage of the method is that, instead of making k-1 round trips on the SIMD core / EU 4001 and making round trips on k-1 new ray generation messages, all hit shaders are called from the same traversal thread during a single traversal operation on the traversal circuit 4005. A potential implementation challenge is that guaranteeing the execution order of hit shaders is not easy (the standard "round trip" method guarantees that the hit shader closest to the intersection executes first, etc.). This can be addressed by relaxing the synchronization or ordering of hit shaders.

[0351] For the intersection shader use case, the traversal circuit system 4005 does not know in advance whether a given shader will return a positive intersection test. However, multiple intersection shaders can be executed speculatively, and if at least one returns a positive hit, it is merged into the global nearest hit. Specific implementations need to find the optimal number of deferred intersection tests to reduce the number of dispatch calls, but avoid calling too many redundant intersection shaders. B. Summary shader calls for traversal circuits

[0352] When multiple shaders are dispatched from the same ray generated from the traversal circuit 4005, branches can be established in the flow of the ray traversal algorithm. This can be problematic for intersecting shaders, as the remainder of the BVH traversal depends on the results of intersection tests for all dispatches. This means that synchronous operations must wait for the results of shader calls, which can be a challenge on asynchronous hardware.

[0353] Two points regarding the merging of shader call results can be: the SIMD processor 4001 and the traversal circuit 4005. Regarding the SIMD processor 4001, multiple shaders can use a standard stylized model to synchronize and aggregate their results. A relatively simple approach is to use global atoms and aggregate the results in a shared data structure in memory, where the intersection results of multiple shaders can be stored. Then, the last shader can parse the data structure and call back to the traversal circuit 4005 to continue the traversal.

[0354] A more efficient method can also be implemented, which restricts the execution of multiple shader calls to channels of the same SIMD thread on the SIMD processor 4001. Then, a SIMD / SIMT reduction operation is used to reduce intersection tests in the region (instead of relying on global atoms). This implementation may depend on new circuitry within the sorting unit 4008 to keep a small batch of shader calls within the same SIMD batch.

[0355] The execution of the traversal thread can be further paused on the traversal circuit 4005. Using the conventional execution model, when a shader is dispatched during traversal, the traversal thread is terminated, and the ray traversal state is saved to memory to allow other ray generation commands to be executed while the execution unit 4001 processes the shader. If only the traversal thread is paused, there is no need to store the traversal state, and the results of each shader can be waited for separately. The implementation may include circuitry to avoid deadlocks and provide sufficient hardware utilization.

[0356] Figures 47-48 show an example of a deferred model that invokes a single shader call on a SIMD core / execution unit 4001 with three shaders 4701. When reserved, all intersection tests are evaluated within the same SIMD / SIMT group. Therefore, the nearest intersection can also be computed on the programmable core / execution unit 4001.

[0357] As mentioned, shader aggregation and / or deferral, in whole or in part, can be performed by the traversal / intersection circuit 4005 and / or the core / EU scheduler 4007. Figure 47 illustrates how the shader deferral / aggregator circuit 4706 within the scheduler 4007 can defer the scheduling of shaders associated with a particular SIMD / SIMT thread / channel until a specified trigger event occurs. Upon detection of the trigger event, the scheduler 4007 dispatches multiple aggregated shaders in a single SIMD / SIMT batch to the core / EU 4001.

[0358] Figure 48 shows how the shader deferral / aggregator circuit 4805 within the traversal / intersection circuit 4005 can defer the scheduling of shaders associated with a particular SIMD thread / channel until a specified trigger event occurs. Once the trigger event is detected, the traversal / intersection circuit 4005 submits the aggregated shaders to the sorting unit 4008 in a single SIMD / SIMT batch.

[0359] However, note that shader deferral and aggregation techniques can be implemented within various other components, such as the sorting unit 4008, or can be distributed across multiple components. For example, the traversal / intersection circuit 4005 can perform a first set of shader aggregation operations, and the scheduler 4007 can perform a second set of shader aggregation operations to ensure that shaders for SIMD threads are efficiently scheduled on the core / EU 4001.

[0360] The "triggering event" that causes the aggregated shaders to be assigned to the core / EU can be a processing event such as a specific number of accumulated shaders or a minimum latency associated with a specific thread. Alternatively or additionally, the triggering event can be a time event, such as a certain duration from the postponement of the first shader or a specific number of processor cycles. The scheduler 4007 can also evaluate other variables such as the current workload on the core / EU 4001 and the traversal / intersection unit 4005 to determine when to assign SIMD / SIMT batches of shaders.

[0361] Depending on the specific system architecture and application requirements, different combinations of the above methods can be used to implement different embodiments of the present invention. Ray tracing instructions

[0362] The following ray tracing instructions are included in the instruction set architecture (ISA) supporting the CPU 3199 and / or GPU 3105. If executed by the CPU, the Single Instruction Multiple Data (SIMD) instructions can utilize vector / compressed source and destination registers to perform the described operations and can be decoded and executed by the CPU core. If executed by the GPU 3105, the instructions can be executed by the graphics core 3130. For example, any of the aforementioned execution units (EUs) 4001 can execute the instructions. Alternatively or additionally, the instructions can be executed by execution circuitry on the ray tracing core 3150 and / or the tensor core 3140.

[0363] Figure 49 shows the architecture used to execute the ray tracing instructions described below. The architecture shown can be integrated on one or more of the cores 3130, 3140, 3150 described above, which may be included in different processor architectures (see, for example, Figure 31 and related text).

[0364] In operation, the instruction fetching unit 4903 fetches the ray tracing instruction 4900 from memory 3198, and the decoder 4995 decodes the instruction. In one implementation, the decoder 4995 decodes the instruction to produce an executable operation (e.g., a micro-operation or micro-instruction in a microcode core). Alternatively, some or all of the ray tracing instruction 4900 may be executed without decoding, thus eliminating the need for the decoder 4904.

[0365] In any implementation, the scheduler / dispatcher 4905 schedules and dispatches instructions (or operations) across a set of functional units (FUs) 4910-4912. The illustrated implementation includes a vector FU 4910 for executing Single Instruction Multiple Data (SIMD) instructions that operate simultaneously on multiple compressed data elements stored in a vector register 4915, and a scalar FU 4911 for operating on scalar values ​​stored in one or more scalar registers 4916. An optional ray-tracing FU 4912 may operate on compressed data values ​​stored in the vector register 4915 and / or scalar values ​​stored in the scalar register 4916. In implementations without a dedicated FU 4912, the vector FU 4910 and possibly the scalar FU 4911 may execute the ray-tracing instructions described below.

[0366] Various FUs 4910-4912 access ray tracing data 4902 (e.g., traversal / intersection data) required to execute ray tracing instructions 4900 from vector registers 4915, scalar registers 4916, and / or regional cache subsystems 4908 (e.g., L1 cache). FUs 4910-4912 can also perform access to memory 3198 via load and store operations, and the cache subsystem 4908 can operate independently to regionally cache data.

[0367] Although ray tracing instructions can be used to increase the performance of ray traversal / intersection and BVH construction, they can also be applied to other areas, such as high-performance computing (HPC) and general-purpose GPU (GPGPU) implementations.

[0368] In the following description, double byte terms are sometimes abbreviated as dw, and unsigned byte terms are abbreviated as ub. Furthermore, the source and destination registers (e.g., src0, src1, dest, etc.) referred to below may refer to vector register 4915, or in some cases, a combination of vector register 4915 and scalar register 4916. Typically, vector register 4915 is used if the source or destination value used by the instruction contains compressed data elements (e.g., a location where the source or destination stores N data elements). Other values ​​may use scalar register 4916 or vector register 4915. Dequantization

[0369] An example of a dequantization instruction is to "dequantize" a previously quantized value. For example, in a ray tracing implementation, certain BVH subtrees can be quantized to reduce storage and bandwidth requirements. A dequantization instruction can take the form dequantize dest src0 src1 src2, where the source register src0 stores N unsigned bytes, the source register src1 stores 1 unsigned byte, the source register src2 stores 1 floating-point value, and the destination register dest stores N floating-point values. All of these registers can be vector registers 4915. Alternatively, src0 and dest can be vector registers 4915, while src1 and src2 can be scalar registers 4916.

[0370] The following code sequence defines a specific implementation of the dequantization instruction: In this example, ldexp multiplies a double-precision floating-point value by a specified power of 2 (i.e., ldexp(x, exp) = x * 2exp). In the code above, if the execution mask value associated with the current SIMD data element (execMask [i]) is set to 1, the SIMD data element at position i in src0 is converted to a floating-point value and multiplied by a power of the value in src1 (2src1value), and this value is added to the corresponding SIMD data element in src2. Selective minimum or maximum

[0371] As indicated by the bits in the bitmask, a selective minimum or maximum instruction can perform a minimum or maximum operation on each channel (that is, return the minimum or maximum value from a set of values). The bitmask can utilize a vector register 4915, a scalar register 4916, or a separate set of mask registers (not shown). The following code sequence defines a specific implementation of the minimum / maximum instruction: sel_min_max dest src0 src1 src2, where src0 stores N double words, src1 stores N double words, src2 stores one double word, and the destination register stores N double words.

[0372] The following code sequence defines a specific implementation of the selective minimum / maximum instruction: In this example, the value of (1 << i) & src2 (i shifted left by 1 and the union of src2) is used to select the minimum or maximum value of the i-th data element in src0 and src1. The operation is performed on the i-th data element only if the execution mask value associated with the current SIMD data element (execMask[i]) is set to 1. Shuffle Index Instruction

[0373] The shuffle index instruction can copy any set of input channels to the output channel. For a SIMD width of 32, the instruction can be executed with a lower output. The instruction takes the form: shuffle_index dest src0 src1 <optional flag>, where src0 stores N double words, src1 stores N unsigned bytes (i.e., index values), and dest stores N double words.

[0374] The following code sequence defines a specific implementation of the shuffle index instruction:

[0375] In the code above, the index in src1 identifies the current channel. If the i-th value in the execution mask is set to 1, a check is performed to ensure that the source channel is within the range of 0 to the SIMD width. If so, the flag (srcLaneMod) is set, and the destination data element i is set to be equal to the data element i of src0. If the channel is in range (i.e., valid), the index value of src1 (srcLane0) is used as the index of src0 (dst[i] = src0[srcLane]). Immediate shuffling Up / Dn / XOR instructions.

[0376] The immediate shuffle instruction can shuffle input data elements / channels based on the immediate value of the instruction. The immediate value can specify that the input channel should be shifted by 1, 2, 4, 8, or 16 positions. Optionally, an additional scalar source register can be specified as a fill value. When the source channel index is invalid, the fill value (if provided) will be stored in the data element location at the destination. If no fill value is provided, the data element location is set to all zeros.

[0377] The flag register can be used as a source mask. If the flag bit of the source channel is set to 1, the source channel can be marked as invalid, and then the instruction can continue to be executed.

[0378] The following are examples of different implementations of the immediate shuffle instruction: In this implementation, src0 stores N double words, src1 stores a double word containing the fill value (if it exists), and dest stores N double words containing the result.

[0379] The following code sequence defines a specific implementation of the immediate shuffle instruction:

[0380] Here, based on the value of the immediate value, the input data element / channel is shifted by 1, 2, 4, 8, or 16 positions. The register src1 is an additional scalar source register that serves as a fill value, which is stored in the data element position at the destination when the source channel index is invalid. If no fill value is provided and the source channel index is invalid, the data element position at the destination is set to 0. The flag register (FLAG) serves as a source mask. If the flag bit of the source channel is set to 1, the source channel is marked as invalid, and the instructions are as described above. Indirect shuffle Up / Dn / XOR instructions

[0381] The indirect shuffle instruction has a source operand (src1) that controls the mapping from the source channel to the destination channel. The indirect shuffle instruction can take the following form: where src0 stores N double words, src1 stores 1 double word, and dest stores N double words.

[0382] The following code sequence defines a specific implementation of the immediate shuffle instruction:

[0383] Therefore, the indirect shuffle instruction operates in a similar manner to the immediate shuffle instruction described above, but the mapping from the source channel to the destination channel is controlled by the source register src1 instead of an immediate value. (Connecting channel minimum / maximum instructions)

[0384] For floating-point and integer data types, the minimum / maximum instruction for the corresponding channel can be supported. The minimum instruction for the corresponding channel can be in the form of lane_min dest src0, and the maximum instruction for the corresponding channel can be in the form of lane_max dest src0, where src0 stores N double words and dest stores 1 double word.

[0385] As an example, the following code sequence defines a specific implementation of the minimum value of a connected channel: In this example, the double-word value at data element location i in the source register is compared with the data element in the destination register, and the minimum of the two values ​​is copied to the destination register. The maximum value of a connected channel operates in essentially the same way, the only difference being that the maximum value of the data element at location i and the destination value is selected. Connected Channel Minimum / Max Index Instructions

[0386] The minimum index instruction for a lane can take the form of lane_min_index dest src0, while the maximum index instruction for a lane can take the form of lane_max_index dest src0, where src0 stores N double words and dest stores 1 double word.

[0387] As an example, the following code sequence defines a specific implementation of the minimum index instruction for the corresponding channel: In this example, the destination index increments from 0 to the SIMD width, spanning the destination register. If the execution mask bit is set, the data element at position i in the source register is copied to the temporary storage location (tmp), and the destination index is set to the data element position i. Corresponding Channel Sorting Network Instruction

[0388] A cross-channel sorting network instruction can use an N-wide (stable) sorting network to sort all N input elements in ascending (sortnet_min) or descending (sortnet_max) order. The minimum / maximum versions of the instruction can take the form sortnet_min dest src0 and sortnet_max dest src0, respectively. In one implementation, src0 and dest store N double words. The minimum / maximum sort is performed on the N double words of src0, with ascending elements (for min) or descending elements (for max) stored in dest according to their respective sorting order. An example of a code sequence defining the instruction is: Cross-channel sorting network index instruction.

[0389] The cross-channel sorting network indexing instruction can sort all N input elements using an N-wide (stable) sorting network, but returns the permutation index in ascending (sortnet_min) or descending (sortnet_max) order. The minimum / maximum versions of the instruction can take the form sortnet_min_index dest src0 and sortnet_max_index dest src0, where src0 and dest each store N double words. An example code sequence defining the instruction is dst = apply_N_wide_sorting_network_min / max_index(src0).

[0390] Figure 50 shows a method for executing any of the above instructions. The method can be implemented on the specific processor architecture described above, but is not limited to any specific processor or system architecture.

[0391] At 5001, instructions for the main graphics thread are executed on the processor core. This may include any of the cores described above (e.g., graphics core 3130). When it is determined at 5002 that ray tracing work has been achieved within the main graphics thread, the ray tracing instructions are offloaded to ray tracing execution circuitry, which may be in the form of a functional unit (FU) as described above with respect to FIG. 49 or may be in a dedicated ray tracing core 3150 as described with respect to FIG. 31.

[0392] At 5003, the decoded ray tracing instructions are retrieved from memory, and at 5005, the instructions are decoded into executable operations (e.g., in embodiments requiring a decoder). At 5004, the ray tracing instructions are scheduled and dispatched for execution by the ray tracing circuitry. At 5005, the ray tracing circuitry executes the ray tracing instructions. For example, the instructions can be dispatched and executed on the aforementioned FU (e.g., vector FU 4910, ray tracing FU 4912, etc.) and / or graphics core 3130 or ray tracing core 3150.

[0393] When the execution of the ray tracing instruction is completed, the result is stored at 5006 (e.g., stored back into memory 3198), and the main graphics thread is notified at 5007. At 5008, the ray tracing result is processed in the context of the main thread (e.g., read from memory and integrated into the graphics rendering result).

[0394] In embodiments, the terms "engine," "module," or "logic" may refer to (or include, in part or in part) an application-specific integrated circuit (ASIC), electronic circuitry, a processor (shared, dedicated, or grouped), and / or memory (shared, dedicated, or grouped) that executes one or more software or firmware programs, combinational logic circuitry, and / or other suitable components that provide the described functionality. In embodiments, the engine, module, or logic may be implemented using firmware, hardware, software, or any combination of firmware, hardware, and software. [Boundary and Ray-frame intersection test]

[0395] Figure 51 is an illustration of a bounding body 5102 according to an embodiment. The shown bounding body 5102 is an axis aligned with a three-dimensional axis 5100. However, the embodiment is applicable to different bounding representations (e.g., oriented bounding boxes, discrete oriented polyhedra, spheres, etc.) and any number of dimensions. The bounding body 5102 defines the minimum and maximum extent of a three-dimensional object 5104 along each dimension of the axis 5100. To generate the BVH of the scene, a bounding box is constructed for each object in the set of objects in the scene. A set of parent bounding boxes can then be constructed around the group of bounding boxes constructed for each object.

[0396] Figures 52A-B show the representation of the hierarchy of boundary bodies for two-dimensional objects. Figure 52A shows a set of boundary bodies 5200 surrounding a group of geometric objects. Figure 52B shows an ordered tree 5202 of the boundary bodies 5200 of Figure 52A.

[0397] As shown in Figure 52A, the set of bounding bodies 5200 includes the root bounding body N1, which is the parent bounding body of all other bounding bodies N2-N7. Bounding bodies N2 and N3 are the internal bounding bodies between the root body N1 and the leaf bodies N4-N7. Leaf bodies N4-N7 contain the geometric objects O1-O8 used in the scene.

[0398] Figure 52B shows an ordered tree 5202 of bounding bodies N1-N7 and geometric objects O1-O8. The ordered tree 5202 shown is a binary tree, wherein each node of the tree has two child nodes. A data structure configured to contain information about each node may contain bounding information of the bounding bodies (e.g., bounding boxes) of the node, as well as references to at least each child node of the node.

[0399] The ordered tree 5202 of the bounding body represents a hierarchical version defining various operations, including but not limited to collision detection and raybox intersection. In the raybox intersection instance, nodes can be tested hierarchically, starting from the root node N1, which is the parent node of all other bounding body nodes in the hierarchy. If the raybox intersection test for the root node N1 fails, all other nodes in the tree may be bypassed. If the raybox intersection test for the root node N1 passes, the subtrees of the tree can be tested and traversed or bypassed in an ordered manner until at least the set of intersecting leaf nodes N4-N7 is determined. The exact testing and traversal algorithms used may vary depending on the embodiment.

[0400] Figure 53 is an illustration of a ray-frame intersection test according to an embodiment. During the ray-frame intersection test, ray 5302 is projected, and the equation defining the ray can be used to determine whether the ray intersects the plane defining the bounding box 5300 under test. Ray 5302 can be represented as O + D·t, where O corresponds to the origin of the ray, D is the direction of the ray, and t is a real value. Changing t can be used to define any point along the ray. Ray 5302 is said to intersect the bounding box 5300 when the maximum incident plane intersection distance is less than or equal to the minimum exit plane distance. For ray 5302 in Figure 53, the y-plane incident intersection distance is shown as tmin - y5304. The y-plane exit intersection distance is shown as tmax - y5308. The x-plane incident intersection distance can be calculated at tmin - x5306, and the x-plane exit intersection distance is shown as tmax - x5310. Therefore, a given ray 5302 can be mathematically represented as intersecting the bounding box at least in the x and y planes, since tmin-x5306 is less than tmax-y5308. To perform the ray-box intersection test using a graphics processor, the graphics processor is configured to store at least the acceleration data structure defining each bounding box to be tested. To use the bounding volume hierarchy for acceleration, references to the child nodes of the bounding boxes must be stored at least. Bounding volume node compression.

[0401] For axis-aligned bounding boxes in 3D space, the acceleration data structure can store the upper and lower bounds of the bounding boxes in three dimensions. The software implementation can use 32-bit floating-point numbers to store these boundaries, with each bounding box requiring a maximum of 2×3×4=24 bytes. For an N-width BVH node, N boxes and N sub-references must be stored. In total, the storage requirement for a 4-width BVH node is N*24 bytes, plus N*4 bytes for sub-references (assuming each reference is 4 bytes), resulting in a total of (24+4)*N bytes. This totals 112 bytes for a 4-width BVH node and 224 bytes for an 8-width BVH node.

[0402] In one embodiment, the size of the BVH node is reduced by storing a single, higher-precision parent bounding box that surrounds all child bounding boxes, and storing each child bounding box with lower precision relative to the parent box. Depending on the use case, different numerical representations can be used to store the high-precision parent bounding box and the lower-precision relative child bounding boxes.

[0403] Figure 54 is a block diagram showing an exemplary quantized BVH node 5410 according to an embodiment. The quantized BVH node 5410 may contain higher precision values ​​to define parent bounding boxes for the BVH node. For example, single-precision or double-precision floating-point values ​​may be used to store parent_lower_x 5412, parent_lower_y 5414, parent_lower_z 5416, parent upper_x 5422, parent_upper_y 5424, and parent_upper_z 5426. The value of the child bounding box of each child bounding box stored in the node may be quantized and stored as a lower precision value, such as a fixed-point representation of the bounding box value defined relative to the parent bounding box. For example, child_lower_x 5432, child_lower_y 5434, child_lower_z 5436, and child_upper_x 5442, child_upper_y 5444, and child_upper_z 5446 can be stored as fixed-point values ​​with lower precision. Furthermore, a child reference 5452 can be stored for each child. The child reference 5452 can be an index to a table storing the location of each child node, or it can be a pointer to the child node.

[0404] As shown in Figure 54, single-precision or double-precision floating-point values ​​can be used to store the parent bounding box, while M-bit fixed-point values ​​can be used to encode the relative child bounding boxes. The data structure of the quantized BVH node 5410 in Figure 54 can be defined through the quantized N-width BVH nodes shown in Table 1 below.

[0405] The quantization nodes in Table 1 reduce the data structure size by quantizing sub-values, while maintaining the precision of the base thread by storing higher-precision values ​​for the range of the parent bounding box. In Table 1, real numbers represent higher-precision numeric representations (e.g., 32-bit or 64-bit floating-point values), while UintM represents a lower-precision unsigned integer using M bits to represent fixed-point numeric precision. References are used to indicate the type of reference to a child node (e.g., a 4-bit index for an 8-bit index).

[0406] A general example of the method can use a 32-bit sub-reference, a single-precision floating-point value for the parent boundary, and M=8 bits (1 byte) relative to the sub-boundary. The compressed node will then require 6*4 + 6*N + 4*N bytes. For a 4-width BVH, this totals 64 bytes (compared to 112 bytes in the uncompressed version); for an 8-width BVH, this totals 104 bytes (compared to 224 bytes in the uncompressed version).

[0407] To traverse such compressed BVH nodes, the graph processing logic can decompress the relative sub-bounding boxes and then intersect the decompressed nodes using standard methods. The uncompressed lower bound can then be obtained for each dimension x, y, and z. Equation 1 below shows the formula used to obtain the sub-lower_x value.

[0408] In Equation 1 above, M represents the number of bits representing the precision of the fixed-point representation of the sub-boundary. The logic for decompressing sub-data for each dimension of the BVH node can be implemented as shown in Table 2 below.

[0409] Table 2 shows the calculation method for the floating-point value of the lower bounding box of the child bounding box, which is based on the range of the parent bounding box and the fixed-point value of the child bounding box, stored as an offset from the range of the parent bounding box. The upper bounding box of the child bounding box can be calculated in a similar way.

[0410] In one embodiment, decompression performance can be improved by storing a scaled parent bounding box size, such as (parent_upper_x - parent_lower_x) / (2M-1), instead of the parent_upper_x / y / z values. In this embodiment, the child bounding box range can be calculated according to the example logic shown in Table 3.

[0411] Note that in the optimized version, decompression / dequantization can be expressed as MAD instructions (multiply-accumulate), for which hardware support exists. In one embodiment, SIMD / vector logic can be used to perform operations on each child node, thereby enabling the simultaneous evaluation of each child node within a node.

[0412] While the above methods work well for shader- or CPU-based implementations, one embodiment provides dedicated hardware configured to perform ray tracing operations that include ray-box intersection testing using bounding volume hierarchies. In this embodiment, the dedicated hardware can be configured to store a further quantized representation of BVH node data and automatically dequantize this data when performing ray-box intersection testing.

[0413] Figure 55 is a block diagram of a composite floating-point data block 5500 used by a quantized BVH node 5510 according to another embodiment. In one embodiment, the logic used to support the composite floating-point data block 5500, in contrast to a 32-bit single-precision floating-point representation or a 64-bit double-precision floating-point representation of the parent bounding box range, can be defined by dedicated logic in the graphics processor. The composite floating-point (CFP) data block 5500 may include a 1-bit sign bit 5502, a variable-size (E-bit) sign integer exponent 5504, and a variable-size (K-bit) mantissa 5506. Multiple values ​​for E and K can be configured by adjusting values ​​stored in a configuration register of the graphics processor. In one embodiment, the values ​​of E and K can be configured independently within a range of values. In one embodiment, a fixed set of related values ​​for E and K can be selected via the configuration register. In one embodiment, each single value for E and K is hard-coded into the BVH logic of the graphics processor. The values ​​E and K enable CFP data block 5500 to be used as a custom (e.g., dedicated) floating-point data type that can be tailored to a dataset.

[0414] Using the CFP data block 5500, the graphics processor can be configured to store bounding box data in quantized BVH nodes 5510. In one embodiment, the lower bounds of the parent bounding boxes (parent_lower_x 5512, parent lower_y 5514, parent_lower_z 5516) are stored with a precision determined by the values ​​of E and K selected for the CFP data block 5500. The precision of the stored values ​​of the parent bounding box lower bounds will typically be set higher than the values ​​of the child bounding boxes (child_lower_x 5524, child_upper_x 5526, child_lower_y 5534, child_upper_y 5536, child_lower_z 5544, child_upper_z 5546), which will be stored as fixed-point values. The scaled parent bounding box size is stored as a power of 2 (e.g., exp_x 5522, exp_y 5532, exp_z 5542). Additionally, a reference for each child (e.g., child reference 5552) can be stored. The size of the quantized BVH node 5510 can be scaled based on the width stored in each node (e.g., the number of children), the amount of storage used to store child references, and the bounding box value of each child node that increases with each additional node.

[0415] Table 4 below shows the logic used to implement the quantization BVH node of Figure 55.

[0416] As shown in Table 4, a composite floating-point data block (e.g., struct Float) can be defined to represent the value of the parent bounding box. The Float structure contains a 1-bit sign (int1 sign), an E-bit sign integer (intE exponent) for storing powers of 2, and a K-bit unsigned integer (uintK mantissa) for representing the mantissa used to store the high-precision range. For child bounding box data, M-bit unsigned integers (uintM child_lower_x / y / z; uintM child_upper_x / y / z) can be used to store fixed-point numbers to encode relative child boundaries.

[0417] For the example of E=8, K=16, M=8 and using 32 bits for the subreference, the QuantizedNodeHW structure in Table 4 has a size of 52 bytes for a 4-width BVH and 92 bytes for an 8-width BVH. This is a reduction in structure size compared to the quantized node in Table 1, and a significant reduction in structure size compared to existing implementations. It should be noted that for the mantissa (K=16), one bit of the mantissa can be implied, thus reducing the storage requirement to 15 bits.

[0418] The layout of the BVH node structure in Table 4 enables streamlined hardware to perform ray-box intersection tests for sub-boundaries. Hardware complexity is reduced for several factors. A lower number of bits can be chosen for K because the relative sub-boundaries add an additional M bits of precision. The scaled parent bounding box size is stored as a power of 2 (exp_x / y / z column), which simplifies calculations. Furthermore, the calculations are refactored to reduce the size of the multipliers.

[0419] In one embodiment, the ray intersection logic of the graphics processor calculates the hit distance of a ray to an axis-aligned plane to perform a ray-frame test. The ray intersection logic may use BVH node logic, which includes support for the quantization node structures in Table 4. The logic may use a higher-precision parent lower bound and the quantization relative range of the child frame to calculate the distance to the parent bounding box lower bound. Exemplary logic for x-plane calculations is shown in Table 5 below.

[0420] Regarding the logic in Table 5, if we assume that single-precision floating-point precision represents a ray, then a 23-bit multiplier by a 15-bit multiplier can be used because the parent_lower_x value is stored as a 15-bit mantissa. The distance between the lower bounds of the parent bounding box on the y and z planes can be calculated in a manner similar to calculating dist_parent_lower_x.

[0421] Using the parent lower bound, the intersection distance to the relative child bounding box can be calculated for each child bounding box, as illustrated in Table 5 for the calculation of dist_child_lower_x and dist_child_upper_x. The calculation of dist_child_lower / upper_x / y / z values ​​can be performed using a 23-bit multiplier by an 8-bit multiplier.

[0422] According to an embodiment, FIG56 shows the use of quantized values ​​to define the intersection of ray boxes of child bounding box 5610 relative to parent bounding box 5600. By applying the ray box intersection distance determination equation for the x-plane shown in Table 5, the distance along ray 5602 at which the ray intersects the boundary of parent bounding box 5600 along the x-plane can be determined. The position dist_parent_lower_x 5603 can be determined, in which ray 5602 intersects the lower bounding plane 5604 of parent bounding box 5600. Based on dist_parent_lower_x 5603, dist_child_lower_x 5605 can be determined, in which the ray intersects the minimum bounding plane 5606 of child bounding box 5610. Furthermore, based on dist_parent_lower_x 5603, the position of dist_child_upper_x 5607 can be determined, where the ray intersects the maximum bounding plane 5608 of the child bounding box 5610. A similar determination can be performed for each dimension (e.g., along the y and z axes) defining the parent bounding box 5600 and the child bounding box 5610. The intersecting plane distance can then be used to determine whether the ray intersects the child bounding box. In one embodiment, the graphics processing logic can use SIMD and / or vector logic to determine the intersection distances of multiple dimensions and multiple bounding boxes in parallel. Furthermore, at least a first portion of the computation described herein can be executed on a graphics processor, while a second portion of the computation can be executed on one or more application processors coupled to the graphics processor.

[0423] Figure 57 is a flowchart of BVH decompression and traversal logic 5700 according to an embodiment. In one embodiment, the BVH decompression and traversal logic resides in dedicated hardware logic of the graphics processor, or may be executed by shader logic executed on the execution resources of the graphics processor. The BVH decompression and traversal logic 5700 may cause the graphics processor to perform operations to calculate the distance along the ray to the lower boundary plane of the parent boundary body, as shown in block 5702. In block 5704, the logic may calculate the distance to the lower boundary plane of the child boundary body in part based on the calculated distance to the lower boundary plane of the parent boundary body. In block 5706, the logic may calculate the distance to the upper boundary plane of the child boundary body in part based on the calculated distance to the lower boundary plane of the parent boundary body.

[0424] In block 5708, the BVH decompression and traversal logic 5700 can determine the ray intersection of a sub-boundary body in part based on the distances to the upper and lower bounding planes of the sub-boundary body, although the intersection distance for each dimension of the bounding box will be used to determine the intersection. In one embodiment, the BVH decompression and traversal logic 5700 determines the ray intersection of a sub-boundary body by determining whether the maximum incident plane intersection distance of the ray is less than or equal to the minimum exit plane distance. In other words, a ray intersects a sub-boundary body when it enters the boundary body along all defined planes before leaving the boundary body along any defined plane. If at 5710, the BVH decompression and traversal logic 5700 determines that a ray intersects a sub-boundary body, the logic can traverse the child nodes of the boundary body to test the sub-boundary body within the child nodes, as shown in block 5712. At block 5712, node traversal can be performed, where references to nodes associated with intersecting bounding boxes can be accessed. A child bounding body can become a parent bounding body, and children of intersecting bounding bodies can be evaluated. If, at 5710, the BVH decompression and traversal logic 5700 determines that a ray does not intersect with a child bounding body, then the branch of the bounding hierarchy associated with the child bounding body is skipped, as shown in block 5714, because the ray will not intersect with any bounding body under any subtree branch associated with the non-intersecting child bounding body. Further compression via shared plane bounding boxes.

[0425] For any N-width BVH using a bounding box, a bounding body hierarchy can be constructed such that each of the six faces of the 3D bounding box is shared by at least one child bounding box. In a 3D shared-plane bounding box, 6 × log2 N bits can be used to indicate whether a given face of the parent bounding box is shared with a child bounding box. For a 3D shared-plane bounding box, when N=4, 12 bits are used to indicate the shared plane, where each bit of the two bits is used to identify which of the four children reuses each potential shared parent plane. Each bit can be used to indicate whether a particular child reuses a parent plane. In the case of a 2-width BVH, 6 additional bits can be added to indicate whether a plane (e.g., a side) of the bounding box is shared by a child for each face of the parent bounding box. Although the SPBB concept can be applied to any number of sizes, in one embodiment, the benefits of SPBB are generally highest for a 2-width (e.g., binary) SPBB.

[0426] When using BVH node quantization as described herein, the use of shared plane bounding boxes can further reduce the amount of data stored. In the 3D, 2-width BVH example, the six shared plane bits can refer to the min_x, max_x, min_y, max_y, min_z, and max_z of the parent bounding box. If the min_x bit is zero, the first child will inherit the shared plane from the parent bounding box. For each child that shares a plane with the parent bounding box, it is not necessary to store the quantized value of the plane, which reduces the storage and decompression costs of the node. Furthermore, higher precision values ​​of the plane can be used for the child bounding boxes.

[0427] Figure 58 is an illustration of an exemplary two-dimensional shared plane bounding box 5800. The two-dimensional (2D) shared plane bounding box (SPBB) 5800 includes a left child 5802 and a right child 5804. For a 2D binary SPBB, 4 log2 2 additional bits can be used to indicate which of the four shared planes of the parent bounding box is shared, where each plane has an associated bit. In one embodiment, zero can be associated with the left child 5802 and one can be associated with the right child, such that the shared plane bits of the SPBB 5800 are min_x=0; max_x=1; min_y=0; max_y=0, because the left child 5802 shares the lower_x, upper_y, and lower_y planes with the parent SPBB 5800, while the right child 5804 shares the upper_x plane.

[0428] Figure 59 is a flowchart of a shared plane BVH logic 5900 according to an embodiment. The shared plane BVH logic 5900 can be used to reduce the number of quantized values ​​stored for the upper and lower ranges of one or more sub-bounding boxes, reduce the decompression / dequantization cost of BVH nodes, and improve the accuracy of ray-box intersection test values ​​for the sub-bounding boxes of BVH nodes. In one embodiment, the shared plane BVH logic 5900 includes defining a parent bounding box on a set of sub-bounding boxes such that the parent bounding box shares one or more planes with one or more sub-bounding boxes, as shown in block 5902. In one embodiment, the parent bounding box can be defined by selecting a set of existing axis-aligned bounding boxes for geometric objects in the scene, and defining the parent bounding box based on the minimum and maximum ranges of the set of bounding boxes in each plane. For example, the upper plane value of each plane of the parent bounding box is defined as the maximum value of each plane within the set of sub-bounding boxes. In block 5904, the shared plane BVH logic 5900 can encode shared subplanes for each plane of the parent bounding box. As shown in block 5906, during ray-box intersection testing, the shared plane BVH logic 5900 can inherit the parent plane value of the subplane with the shared plane. The shared plane value of the child can be inherited with higher precision, where the parent plane value is stored in the BVH node structure, and the generation and storage of lower precision quantization values ​​for the shared plane can be bypassed. Example of a ray / path tracing architecture.

[0429] Figure 60 shows an exemplary ray / path tracing architecture on which embodiments of the present invention can be implemented. In the embodiment, the traversal circuit 6002 can be configured to have or be programmed with box-to-box test logic 6003 for performing box-to-box tests as described below (that is, in addition to performing box-to-box tests when traversing a ray through a node of the BVH). The embodiment shown in

[0430] includes a shader execution circuit 4000 for executing shader code and processing related ray tracing data 4902 (e.g., BVH node data and ray data), a ray tracing acceleration circuit 6010 including a traversal circuit 6002 and an intersection circuit 6003 for performing traversal and intersection operations respectively, and a memory 3198 for storing the code and related data processed by the RT acceleration circuit 6010 and the shader execution circuit 4000.

[0431] In one embodiment, the shader execution circuitry 4000 includes a plurality of core / execution units 4001 that execute shader code to perform various forms of data-parallel operations. For example, in one embodiment, the core / execution unit 4001 may execute a single instruction across multiple channels, wherein each instance of the instruction operates on data stored in a different channel. For example, in a SIMT implementation, each instance of the instruction is associated with a different thread. During execution, the L1 cache stores certain ray-tracing data for efficient access (e.g., recently or frequently accessed data).

[0432] A group of primary rays can be assigned to scheduler 4007, which schedules work to shaders executed by core / EU 4001. Core / EU 4001 can be ray tracing core 3150, graphics core 3130, CPU core 3199, or other types of circuitry capable of executing shader code. One or more primary ray shaders 6001 process the primary rays and generate additional work (e.g., to be executed by one or more sub-shaders) that will be executed by ray tracing acceleration circuitry 6010 and / or core / EU 4001. New work generated by primary ray shaders 6001 or other shaders executed by core / EU 4001 can be assigned to sorting circuitry 4008, which sorts rays into groups or binary sub-shaders as described herein (e.g., grouping rays into groups with similar characteristics). Scheduler 4007 then schedules the new work on core / EU 4001.

[0433] Other executable shaders include any hit shader 4514 and the closest hit shader 4507, which are hits as described above (e.g., identifying any hit or closest hit for a given ray). Missing shaders 4506 handle ray misses (e.g., where the ray does not intersect with a node / primitive). As mentioned, various shaders can be referenced using shader records, which may contain one or more metrics, vendor-specific metadata, and global parameters. In one embodiment, shader records are identified by a Shader Record Identifier (SRI). In one embodiment, each execution instance of a shader is associated with a call stack 5203, which stores parameters passed between parent and child shaders. Call stack 6021 may also store references to continued functionality executed upon call return.

[0434] When processing rays, traversal circuit 6002 causes each ray to traverse through the nodes of the BVH, working down the hierarchy of the BVH (e.g., through parent nodes, child nodes, and leaf nodes) to identify the nodes / primitives traversed by the ray. When processing query boxes, traversal circuit 6002 (according to box-to-box test logic 6003) traverses each query box through the BVH nodes, comparing the query box coordinates with the BVH node coordinates to determine overlap.

[0435] The intersection circuit 6003 performs a ray / frame intersection test, determines the hit point on the primitive, and generates a result in response to the hit. The traversal circuit 6002 and the intersection circuit 6003 can retrieve work from one or more call stacks 6021. In the ray tracing acceleration circuit 6010, the call stack 6021 and the associated ray and frame data 4902 can be stored in the area ray tracing cache (RTC) 6007 or other area storage devices for efficient access by the traversal circuit 6002 and the intersection circuit 6003.

[0436] The ray tracing acceleration circuit 6010 may be a variation of the various traversal / intersection circuits described herein, including ray BVH traversal / intersection circuits 4005, traversal circuit 4502, and intersection circuit 4503, and ray tracing core 3150. The ray tracing acceleration circuit 6010 may also replace the ray BVH traversal / intersection circuits 4005, traversal circuit 4502, and intersection circuit 4503, and ray tracing core 3150, or any other circuit / logic used to process BVH stacks and / or perform traversal / intersection. Therefore, any combination of features disclosed herein also discloses corresponding combinations with, but is not limited to, the ray tracing acceleration circuit 6010.

[0437] Referring to FIG. 61, one embodiment of the traversal circuit 6002 includes a first memory bank 6101 and a second memory bank 6102, each containing a plurality of entries loaded from memory for storing a corresponding plurality of incident rays or frames 6106. Corresponding first and second stacks 6103 and 6104 each contain selected BVH node data 6190-6191 read from memory and regionally stored for processing. As described herein, in one embodiment, stacks 6103-6104 are "short" stacks containing a limited number of entries for storing BVH node data. While also shown separately from ray banks 6101-6102, stacks 6103-6104 may also be maintained within their respective ray banks 6101-6102. Alternatively, stacks 6103-6104 may be stored in separate regional memories or caches.

[0438] One embodiment of the traversal processing circuit 6110 alternates between two libraries 6101-6102 and stacks 6103-6104 (e.g., in a ping-pong manner) when selecting the next ray or box and the node to be processed. For example, the traversal processing circuit 6110 can select a new ray / box and BVH node from the alternating libraries and stacks each clock cycle, thereby ensuring efficient operation. However, it should be noted that this particular arrangement is not necessary to comply with the basic principles of the invention. As mentioned, one embodiment of the traversal processing circuit 6110 includes box-to-box test logic 6003 for traversing query boxes through the BVH as described herein.

[0439] In one embodiment, the allocator 6105 balances the entries of incident rays / boxes 6106 entering the first and second memory libraries 6101-6102 based on the current relative values ​​of a set of library allocation counters 6120. In one embodiment, the library allocation counters 6120 maintain a count of the number of untraversed rays / boxes in each of the first and second memory libraries 6101-6102. For example, the first library allocation counter may increment when the allocator 6105 adds a new ray or box to the first library 6101, and may decrement when a ray or box is processed from the first library 6101. Similarly, the second library allocation counter may increment when the allocator 6105 adds a new ray or box to the second library 6101, and may decrement when a ray or box is processed from the second library 6101.

[0440] In one embodiment, the allocator 6105 assigns the currently input ray or box to the library associated with the smaller counter value. If the two counters are equal, the allocator 6105 can select either library or a library different from the one selected when the counters were equal last time. In one embodiment, each ray / box is stored in one entry of one of libraries 6101-6102, and each library contains 32 entries for storing up to 32 rays and / or boxes. However, the basic principles of the invention are not limited to these details.

[0441] In various situations, such as when the shader needs to perform a series of operations, the traversal circuit 6002 must pause the traversal operation and save the current ray / box and associated BVH nodes. For example, if a non-opaque object is a hit or procedural texture, the traversal circuit 6002 saves the stacks 6103-6104 to memory and executes the required shader. Once the shader has finished processing the hit (or other data), the traversal circuit 6002 restores the state of the libraries 6101-6102 and the stacks 6103-6104 from memory.

[0442] In one embodiment, the traversal / stack tracker 6148 continuously monitors traversal and stacking operations and stores restart data in the tracking array 6149. For example, if the traversal circuitry 6002 has traversed nodes N, N0, N1, N2, and N00 and the resulting traversal, the traversal / stack tracker 6148 updates the tracking array to indicate that the traversal of these nodes has been completed and / or to indicate the next node to be processed from the stack. When the traversal circuitry system 6002 restarts, restart data is read from the tracking array 6149 so that traversal can restart at the correct stage without retracing any BVH nodes (and wasting cycles). The restart data stored in the tracking array 6149 is sometimes referred to as the "restart track" or "RST". Dynamic precision floating-point unit

[0443] The IEEE 754 double-precision binary floating-point format specifies a 64-bit binary representation with 1 sign bit, 11-bit exponent, and 53 significant bits, of which 52 bits are explicitly stored. The IEEE 754 single-precision binary floating-point format specifies a 32-bit binary representation with 1 sign bit, 8-bit exponent, and 24 significant bits, of which 23 bits are explicitly stored. The IEEE 754 half-precision binary floating-point format specifies a 16-bit binary representation with 1 sign bit, 5-bit exponent, and 11 significant bits, of which 10 bits are explicitly stored. For non-zero exponent values, the implicit significant bits are defined as 1, and when all exponent bits are 0, the implicit significant bits are defined as 0. Floating-point units capable of performing arithmetic operations in double-precision, single-precision, and half-precision are known in the art. For example, existing floating-point units can perform 64-bit floating-point operations, 32-bit single-precision floating-point operations, or 16-bit half-precision floating-point operations.

[0444] Embodiments of the present invention extend this capability by providing supporting instructions and related logic, thereby enabling variable precision operations with dynamic rounding. Floating-point instructions that allow variable precision operations can dynamically increase output by performing operations with lower precision where possible. In one embodiment, a set of instructions and related logic is provided, wherein output is increased by performing floating-point operations with the lowest possible precision without significant loss of information. In another embodiment, a set of instructions and related logic is provided, wherein the floating-point logic verifies a lower precision result relative to the result of execution with higher precision to determine whether any significant data loss has occurred.

[0445] FIG62 shows the elements of a dynamic precision floating-point unit 6200 according to one embodiment. In one embodiment, the dynamic precision floating-point unit 6200 includes a control unit 6202, a set of internal registers 6204, an exponent block 6206, and a sign block 6208. In addition to floating-point control logic known in the art, in one embodiment, the control unit 6202 additionally includes precision tracking logic 6212 and a number conversion unit 6222.

[0446] In one embodiment, precision tracking logic 6212 is hardware logic configured to track the number of available precision bits of computational data relative to a target precision. Precision tracking logic 6212 may track precision registers within exponent block 6206 and significant number block 6208 to track precision metrics, such as storing the minimum number of precision bits required to produce the computational value generated by exponent block 6206 and significant number block 6208. In one embodiment, the precision metric comprises a running average of the numerical precision required to computationally represent the data. In one embodiment, the precision metric comprises the maximum required precision within a given set of data. In one embodiment, dynamic precision floating-point unit 6200 supports instructions to read or reset register data used by precision tracking logic 6212 to produce the precision metric described herein. In one embodiment, a computational unit housing the dynamic precision floating-point unit supports instructions to set or reset register data used by precision tracking logic 6212. In one embodiment, precision tracking logic 6212 monitors an error accumulator 6234 in a set of internal temporary registers 6204. The error accumulator can be used to track accumulated errors (e.g., rounding errors) for a set of floating-point operations. In one embodiment,...

Claims

1. An apparatus for double-precision ray traversal, comprising: a bounded volume hierarchy (BVH) generator for constructing a BVH comprising a plurality of BVH nodes; a ray memory for storing rays to be traversed through one or more of the plurality of BVH nodes; and a ray traversal circuit comprising: a first plurality of 64-bit arithmetic logic units (ALUs) natively supporting double-precision floating-point arithmetic, the ray traversal circuit being configured to retrieve rays stored in the ray memory and the plurality of BVH nodes from the BVH generator to perform traversal, wherein the first ALU of the one or more ALUs traverses the first ray with double-precision floating-point precision through the first BVH nodes to generate double-precision ray traversal. The ray traversal circuitry includes: a floating-point traversal result; a dynamic rounding circuitry for selecting a rounding mode to round at least one of the double-precision floating-point traversal results to produce a lower-precision traversal result, when it is determined that the lower-precision traversal result sufficiently represents at least one of the double-precision floating-point traversal results; and a plurality of execution units (EUs) coupled to the ray traversal circuitry, comprising at least one of the plurality of EUs including a second plurality of 64-bit ALUs capable of natively performing double-precision floating-point operations, the at least one of the plurality of EUs being configured to execute one or more intersection shaders to perform ray primitive intersection tests with double-precision floating-point precision based on the lower-precision traversal result.

2. The device of claim 1, wherein the EU system executing the one or more intersection shaders is used to produce an intersection result, the intersection result comprising the intersection between the first ray and the first primitive associated with the first BVH node.

3. The device of claim 2, wherein the EU system executing the one or more intersecting shaders is used to generate additional work, the additional work including work to be performed by the one or more additional shaders and / or work to be performed by the ray traversal circuit.

4. The device as claimed in claim 3, wherein the one or more additional shaders comprise the closest hit shader, any hit shader, or a miss shader.

5. The device as claimed in claim 3, wherein the work to be performed by the ray traversal circuit comprises one or more secondary rays to be traversed through the BVH traversal.

6. The device as claimed in claim 1, wherein the first ALU comprises a 64-bit floating-point multiplier and a 64-bit floating-point adder.

7. The device as claimed in claim 6, wherein the first ALU includes a set of arithmetic registers for storing 64-bit floating-point input values ​​to be multiplied and / or added by the 64-bit floating-point multiplier and / or 64-bit floating-point adder.

8. The device of claim 7 further includes: dynamic precision control circuitry for causing the 64-bit floating-point multiplier and the 64-bit floating-point adder to perform multiplication and addition using 32-bit floating-point or 16-bit floating-point values, respectively.

9. A method for double-precision ray traversal, comprising: constructing a bounded volume hierarchy (BVH) from a BVH generator, the BVH comprising a plurality of BVH nodes; reading a first ray from a ray memory; traversing the first ray through the first BVH node among the plurality of BVH nodes with double-precision floating-point precision via a ray traversal circuit to produce a double-precision floating-point traversal result, the ray traversal circuit comprising a first 64-bit arithmetic logic unit (ALU) capable of natively performing double-precision floating-point operations, the ray traversal circuit being configured to retrieve rays stored in the ray memory and the plurality of BVH nodes from the BVH generator to perform traversal; and using a rounding mode to round the first ray. At least one of the double-precision floating-point traversal results is rounded to produce a lower-precision traversal result, and when it is determined that the lower-precision traversal result sufficiently represents the at least one of the double-precision floating-point traversal results; the lower-precision traversal result is provided to one or more of a plurality of execution units (EUs), at least one of the plurality of EUs including a second 64-bit ALU capable of natively performing double-precision floating-point operations; and an intersection test is performed on the lower-precision traversal result by one or more intersection shaders executed on one or more of the plurality of EUs, the intersection test including a double-precision floating-point ray primitive intersection test based on the lower-precision traversal result.

10. The method of claim 9, wherein the EU system of the one or more intersection shaders is used to produce an intersection result, the intersection result including the intersection between the first ray and the first primitive associated with the first BVH node.

11. The method of claim 10, wherein the EU system executing the one or more intersecting shaders is used to generate additional work, the additional work including work to be performed by the one or more additional shaders and / or work to be performed by the ray traversal circuit.

12. The method of claim 11, wherein the one or more additional shaders comprise the closest hit shader, any hit shader, or a miss shader.

13. The method of claim 11, wherein the work to be performed by the ray traversal circuit comprises one or more secondary rays to be traversed through the BVH traversal.

14. The method of claim 9, wherein the first ALU comprises a 64-bit floating-point multiplier and a 64-bit floating-point adder.

15. The method of claim 14, wherein the first ALU includes a set of operand registers for storing 64-bit floating-point input values ​​to be multiplied and / or added by the 64-bit floating-point multiplier and / or 64-bit floating-point adder.

16. The method of claim 15 further comprises: dynamically selecting the lower precision to be used by the 64-bit floating-point multiplier and the 64-bit floating-point adder to perform multiplication and addition using 32-bit floating-point or 16-bit floating-point values, respectively.

17. A machine-readable medium having stored code thereon, which, when executed by a machine, causes the machine to: construct a bounded volume hierarchy (BVH) comprising a plurality of BVH nodes by a BVH generator; read a first ray from a ray memory; and traverse the first ray through the first BVH node of the plurality of BVH nodes with double-precision floating-point precision via a ray traversal circuit to produce a double-precision floating-point traversal result, the ray traversal circuit comprising a first 64-bit arithmetic logic unit (ALU) capable of natively performing double-precision floating-point operations, the ray traversal circuit being configured to retrieve rays stored in the ray memory and the plurality of BVH nodes from the BVH generator to perform the traversal. The process involves: using a rounding mode to round at least one of the double-precision floating-point traversal results to produce a lower-precision traversal result, and, when it is determined that the lower-precision traversal result sufficiently represents at least one of the double-precision floating-point traversal results; providing the lower-precision traversal result to one or more of a plurality of execution units (EUs), at least one of the plurality of EUs including a second 64-bit ALU capable of natively performing double-precision floating-point operations; and performing an intersection test on the lower-precision traversal result through one or more intersection shaders executed on one or more of the plurality of EUs, the intersection test including a double-precision floating-point ray primitive intersection test based on the lower-precision traversal result.

18. The machine-readable medium of request item 17, wherein the EU system executing the one or more intersection shaders is used to produce an intersection result, the intersection result comprising the intersection between the first ray and the first primitive associated with the first BVH node.

19. The machine-readable medium of claim 18, wherein the EU system executing the one or more intersecting shaders is used to generate additional work, the additional work including work to be performed by the one or more additional shaders and / or work to be performed by the ray traversal circuit.

20. The machine-readable medium as requested in item 19, wherein the one or more additional shaders comprise the closest hit shader, any hit shader, or a miss shader.

21. A machine-readable medium as requested in claim 19, wherein the work to be performed by the ray traversal circuitry comprises one or more secondary rays to be traversed through the BVH traversal.

22. The machine-readable medium as requested in item 17, wherein the first ALU comprises a 64-bit floating-point multiplier and a 64-bit floating-point adder.

23. The machine-readable medium of claim 22, wherein the first ALU includes a set of arithmetic registers for storing 64-bit floating-point input values ​​to be multiplied and / or added by the 64-bit floating-point multiplier and / or 64-bit floating-point adder.

24. The machine-readable medium as requested in item 23 further includes: dynamically selecting the lower precision to be used by the 64-bit floating-point multiplier and the 64-bit floating-point adder to perform multiplication and addition using 32-bit floating-point or 16-bit floating-point values, respectively.

Images