Apparatus and method for processing and executing ray tracing instructions

By using the machine learning engine to dynamically train and adjust network weights in ray tracing, the problems of ray tracing technology in real-time performance and image quality are solved, and high-quality, resource-efficient image rendering is achieved.

CN112258620BActive Publication Date: 2024-11-08INTEL CORP
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202011270082.3
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Priority Date
2018-12-28
Filing Date
2019-11-28
Publication Date
2024-11-08
Estimated Expiration
2039-11-28

AI Technical Summary

Technical Problem

The existing ray tracing technology is resource-intensive in real-time performance, and the existing noise reduction framework is mainly aimed at single-machine operations. It is impossible to effectively access the noise reduction part of the calculated image when rendering across multiple devices, resulting in inconsistent image quality.

Method used

Using machine learning engines, such as convolutional neural networks (CNNs), dynamically collects learning data during rendering, and continuously trains and adjusts network weights at runtime, generating new training data for image subregions per frame or N frames. Improve noise reduction effect to ensure image quality improvement.

Benefits of technology

The generation of high-quality images in real-time ray tracing is realized, avoiding high-cost training data calculations, ensuring the smoothness and consistency of images, and improving the performance of ray tracing.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN112258620B_ABST
    Figure CN112258620B_ABST
Patent Text Reader

Abstract

The present application discloses an apparatus and method for processing and executing ray tracing instructions. An apparatus and method for executing ray tracing instructions. For example, one embodiment of the apparatus includes: an execution circuit for executing a de-discretization instruction to convert a plurality of discretized data values ​​into a plurality of de-discretized data values, the de-discretization instruction including a first source operand for identifying a plurality of packed discretized data values ​​in a source register and a destination operand for identifying a destination register in which the plurality of packed de-discretized data values ​​are to be stored, wherein the execution circuit is for converting each packed discretized data value in the source register into a floating point value, multiplying the floating point value with a first value to generate a first product and adding the first product to a second value to generate a de-discretized data value, and storing the de-discretized data value in a packed data element position in the destination register.
Need to check novelty before this filing date? Find Prior Art

Description

[0001] This invention patent application is a divisional application of the invention patent application with application number 201911193005.X filed on November 28, 2019 and named “Device and method for processing and executing ray tracing instructions”. background Technical Field

[0002] The present invention generally relates to the field of graphics processors. More specifically, the present invention relates to apparatus and methods for performing more efficient ray tracing operations. Background Art

[0004] Ray tracing is a technique in which light transport is simulated via physically based rendering. Although widely used in film rendering, until just a few years ago it was considered too resource intensive for real-time performance. One of the key operations in ray tracing is visibility queries for ray-scene intersections known as "ray traversals", which compute ray-scene intersections by traversing and intersecting nodes in a bounding volume hierarchy (BVH).

[0005] Denoising has become a critical feature for real-time ray tracing with smooth, noise-free images. Rendering can be done on multiple devices across a distributed system, but to date, existing denoising frameworks all operate on a single instance on a single machine. If rendering is being done across multiple devices, those devices may not have all rendered pixels accessible to compute the denoised portion of the image. BRIEF DESCRIPTION OF THE DRAWINGS

[0006] A better understanding of the present invention can be obtained from the following specific embodiments in conjunction with the following drawings, in which:

[0007] Figure 1 is a block diagram of an embodiment of a computer system having a processor with one or more processor cores and a graphics processor;

[0008] Figure 2 is a block diagram of one embodiment of a processor having one or more processor cores, an integrated memory controller, and an integrated graphics processor;

[0009] Figure 3 is a block diagram of one embodiment of a graphics processor, which may be a discrete graphics processing unit, or may be a graphics processor integrated with multiple processing cores;

[0010] Figure 4 is a block diagram of an embodiment of a graphics processing engine for a graphics processor;

[0011] Figure 5is a block diagram of another embodiment of a graphics processor;

[0012] Figure 6A-6B illustrates examples of implementation circuits and logic;

[0013] Figure 7 illustrates a graphics processor execution unit instruction format according to an embodiment;

[0014] Figure 8 is a block diagram of another embodiment of a graphics processor, the graphics processor including a graphics pipeline, a media pipeline, a display engine, thread execution logic, and a rendering output pipeline;

[0015] Fig.9A is a block diagram illustrating a graphics processor command format according to an embodiment;

[0016] Fig. 9B is a block diagram illustrating a graphics processor command sequence according to an embodiment;

[0017] Fig.10 illustrates an exemplary graphics software architecture for a data processing system according to an embodiment;

[0018] Figure 11A-11B illustrates an exemplary IP core development system that can be used to manufacture integrated circuits and exemplary package assemblies;

[0019] Fig.12 illustrates an exemplary system-on-chip integrated circuit that may be manufactured using one or more IP cores according to an embodiment;

[0020] Figure 13A-13B illustrates an exemplary graphics processor of a system-on-chip integrated circuit that may be manufactured using one or more IP cores;

[0021] Figure 14A-14B illustrates an exemplary graphics processor architecture;

[0022] Fig.15 illustrates one embodiment of an architecture for performing initial training of a machine learning architecture;

[0023] Fig.16 illustrates an embodiment in which a machine learning engine is continuously trained and updated during runtime;

[0024] Fig.17 illustrates another embodiment in which a machine learning engine is continuously trained and updated during runtime;

[0025] Figure 18A-18B illustrates an embodiment in which machine learning data is shared over a network; and

[0026] Fig.19 illustrates one embodiment of a method for training a machine learning engine;

[0027] Fig. 20 An embodiment is illustrated in which nodes exchange ghost region data to perform a distributed noise reduction operation;

[0028] Fig.21 illustrates one embodiment of an architecture in which image rendering and denoising operations are distributed across multiple nodes;

[0029] Fig. 22 Illustrate additional details of the architecture used for distributed rendering and denoising;

[0030] Fig.23 A method according to an embodiment of the present invention is illustrated;

[0031] Fig.24 An embodiment of a machine learning method is illustrated;

[0032] Fig.25 A plurality of general purpose graphics processors are shown interconnected;

[0033] Fig.26 Illustration of a collection of convolutional and fully connected layers used in machine learning implementations;

[0034] Fig. 27 An embodiment of a convolutional layer is shown.

[0035] Fig.28 illustrates an example of a collection of interconnected nodes in a machine learning implementation;

[0036] Fig.29 An embodiment of a training framework in which a neural network learns using a training data set is illustrated;

[0037] Fig. 30A Illustrate examples of model parallelism and data parallelism;

[0038] Fig. 30B An example of a system on a chip (SoC) is shown;

[0039] Fig.31 An example of a processing architecture is shown, the processing architecture including ray tracing cores and tensor cores;

[0040] Fig.32 An example of a beam of light is shown;

[0041] Fig.33 illustrates an embodiment of an apparatus for performing beam tracing;

[0042] Fig.34 An example of a beam hierarchy is shown;

[0043] Fig.35 illustrates a method for performing beam tracing;

[0044] Fig.36 illustrates an example of a distributed ray tracing engine;

[0045] Figure 37-Figure 38 illustrates an example of compression performed in a ray tracing system;

[0046] Fig.39 A method according to an embodiment of the present invention is illustrated;

[0047] Fig.40 An exemplary hybrid ray tracing device is shown;

[0048] Fig.41 illustrates an example of a stack for a ray tracing operation;

[0049] Fig.42 illustrates additional details of one embodiment of a hybrid ray tracing apparatus;

[0050] Fig.43 An example of a bounding volume hierarchy is shown;

[0051] Fig.44 Illustrate examples of call stack and traversal state storage;

[0052] Fig.45 illustrates one embodiment of an architecture for executing ray tracing instructions; and

[0053] Fig.46 An embodiment of a method for performing any of the above instructions shown is illustrated. DETAILED DESCRIPTION

[0054] In the following description, for purposes of explanation, numerous specific details are set forth in order to provide a thorough understanding of the embodiments of the invention described below. However, it will be apparent to those skilled in the art that embodiments of the invention may be practiced without some of these specific details. In other instances, well-known structures and devices are shown in block diagram form to avoid obscuring the basic principles of the embodiments of the invention.

[0055] Exemplary Graphics Processor Architecture and Data Types

[0056] System Overview

[0057] Figure 11 is a block diagram of a processing system 100 according to an embodiment. In various embodiments, system 100 includes one or more processors 102 and one or more graphics processors 108, and may be a single processor desktop system, a multi-processor workstation system, or a server system having a large number of processors 102 or processor cores 107. In one embodiment, system 100 is a processing platform incorporated within a system on a chip (SoC) integrated circuit for use in a mobile device, handheld device, or embedded device.

[0058] In one embodiment, system 100 may include or be incorporated into the following: a server-based game platform, a game console including a game and media console, a mobile game console, a handheld game console, or an online game console. In some embodiments, system 100 is a mobile phone, a smart phone, a tablet computing device, or a mobile Internet device. Processing system 100 may also include a wearable device that can be coupled to or integrated into a wearable device, such as a smart watch wearable device, a smart glasses device, an augmented reality device, or a virtual reality device. In some embodiments, processing system 100 is a television or set-top box device having one or more processors 102 and a graphical interface generated by one or more graphics processors 108.

[0059] In some embodiments, the one or more processors 102 each include one or more processor cores 107, which are used to process instructions that, when executed, perform operations for system and user software. In some embodiments, each of the one or more processor cores 107 is configured to process a specific instruction set 109. In some embodiments, the instruction set 109 can facilitate complex instruction set computing (CISC), reduced instruction set computing (RISC), or computing via very long instruction words (VLIW). Multiple processor cores 107 can each process different instruction sets 109, and different instruction sets 109 may include instructions for facilitating the simulation of other instruction sets. The processor core 107 may also include other processing devices, such as a digital signal processor (DSP).

[0060] In some embodiments, the processor 102 includes a cache memory 104. Depending on the architecture, the processor 102 may have a single internal cache or multiple levels of internal cache. In some embodiments, the cache memory is shared between various components of the processor 102. In some embodiments, the processor 102 also uses an external cache (e.g., a level 3 (L3) cache or a last level cache (LLC)) (not shown), which can be shared between the processor cores 107 using known cache coherence techniques. A register file 106 is additionally included in the processor 102, and the register file 106 may include different types of registers (e.g., integer registers, floating point registers, status registers, and instruction pointer registers) for storing different types of data. Some registers may be general purpose registers, while other registers may be specific to the design of the processor 102.

[0061] In some embodiments, one or more processors 102 are coupled to one or more interface buses 110 to transmit communication signals, such as address, data, or control signals, between the processor 102 and other components in the system 100. In one embodiment, the interface bus 110 may be a processor bus, such as a version of a direct media interface (DMI) bus. However, the processor bus is not limited to a DMI bus and may include one or more peripheral component interconnect buses (e.g., PCI, PCI Express), a memory bus, or other types of interface buses. In one embodiment, the processor(s) 102 include an integrated memory controller 116 and a platform controller hub 130. The memory controller 116 facilitates communication between memory devices and other components of the system 100, while the platform controller hub (PCH) 130 provides connections to I / O devices via a local I / O bus.

[0062] The memory device 120 may be a dynamic random access memory (DRAM) device, a static random access memory (SRAM) device, a flash memory device, a phase change memory device, or some other memory device having suitable performance to act as a process memory. In one embodiment, the memory device 120 may operate as a system memory for the system 100 to store data 122 and instructions 121 for use when one or more processors 102 execute applications or processes. The memory controller 116 is also coupled to an optional external graphics processor 112, which may communicate with one or more graphics processors 108 in the processor 102 to perform graphics operations and media operations. In some embodiments, a display device 111 may be connected to the processor(s) 102. The display device 111 may be one or more of the following: an internal display device, such as in a mobile electronic device or laptop device; or an external display device attached via a display interface (e.g., a display port, etc.). In one embodiment, the display device 111 may be a head mounted display (HMD), such as a stereoscopic display device for use in a virtual reality (VR) application or an augmented reality (AR) application.

[0063] In some embodiments, the platform controller hub 130 enables peripheral devices to be connected to the memory device 120 and the processor 102 via a high-speed I / O bus. The I / O peripherals include, but are not limited to, an audio controller 146, a network controller 134, a firmware interface 128, a wireless transceiver 126, a touch sensor 125, a data storage device 124 (e.g., a hard drive, flash memory, etc.). The data storage device 124 can be connected via a storage interface (e.g., SATA) or via a peripheral bus such as a peripheral component interconnect bus (e.g., PCI, PCI Express). The touch sensor 125 may include a touch screen sensor, a pressure sensor, or a fingerprint sensor. The wireless transceiver 126 may be a Wi-Fi transceiver, a Bluetooth transceiver, or a mobile network transceiver such as a 3G, 4G or long term evolution (LTE) transceiver. The firmware interface 128 enables communication with the system firmware and may be, for example, a unified extensible firmware interface (UEFI). The network controller 134 may enable a network connection to a wired network. In some embodiments, a high-performance network controller (not shown) is coupled to the interface bus 110. In one embodiment, the audio controller 146 is a multi-channel high definition audio controller. In one embodiment, the system 100 includes an optional legacy I / O controller 140 for coupling legacy (e.g., Personal System 2 (PS / 2)) devices to the system. The platform controller hub 130 may also be connected to one or more Universal Serial Bus (USB) controllers 142 to connect input devices, such as a keyboard and mouse 143 combination, a camera 144, or other USB input devices.

[0064] It will be appreciated that the illustrated system 100 is exemplary and not limiting, as other types of data processing systems configured in different manners may also be used. For example, instances of the memory controller 116 and the platform controller hub 130 may be integrated into a discrete external graphics processor, such as the external graphics processor 112. In one embodiment, the platform controller hub 130 and / or the memory controller 116 may be external to one or more processors 102. For example, the system 100 may include the external memory controller 116 and the platform controller hub 130, which may be configured as a memory controller hub and a peripheral controller hub within a system chipset that communicates with the processor(s) 102.

[0065] Figure 2 is a block diagram of an embodiment of a processor 200 having one or more processor cores 202A- 202N, an integrated memory controller 214 , and an integrated graphics processor 208 . Figure 2 Those elements with the same reference numerals (or names) as those of any other figures herein can operate or function in any manner similar to that described elsewhere herein, but are not limited thereto. Processor 200 may include additional cores up to and including additional core 202N represented by a dashed box. Each of processor cores 202A-202N includes one or more internal cache units 204A-204N. In some embodiments, each processor core also has access to one or more shared cache units 206.

[0066] Internal cache units 204A-204N and shared cache unit 206 represent a cache memory hierarchy within processor 200. The cache memory hierarchy may include at least one level of instruction and data cache within each processor core and one or more levels of shared mid-level cache, such as level 2 (L2), level 3 (L3), level 4 (L4), or other levels of cache, where the highest level of cache before external memory is classified as LLC. In some embodiments, cache coherence logic maintains coherence between each cache unit 206 and 204A-204N.

[0067] In some embodiments, the processor 200 may also include a set of one or more bus controller units 216 and a system agent core 210. The one or more bus controller units 216 manage a set of peripheral buses, such as one or more PCI buses or PCI Express buses. The system agent core 210 provides management functions for various processor components. In some embodiments, the system agent core 210 includes one or more integrated memory controllers 214 for managing access to various external memory devices (not shown).

[0068] In some embodiments, one or more of the processor cores 202A-202N include support for simultaneous multithreading. In such embodiments, the system agent core 210 includes components for coordinating and operating the cores 202A-202N during multithreaded processing. The system agent core 210 may additionally include a power control unit (PCU) that includes logic and components for regulating the power state of the processor cores 202A-202N and the graphics processor 208.

[0069] In some embodiments, the processor 200 additionally includes a graphics processor 208 for performing graphics processing operations. In some embodiments, the graphics processor 208 is coupled to a set of shared cache units 206 and a system agent core 210, which includes one or more integrated memory controllers 214. In some embodiments, the system agent core 210 also includes a display controller 211 for driving the graphics processor output to one or more coupled displays. In some embodiments, the display controller 211 can also be a separate module coupled to the graphics processor via at least one interconnect, or can be integrated within the graphics processor 208.

[0070] In some embodiments, a ring-based interconnect unit 212 is used to couple the internal components of the processor 200. However, alternative interconnect units may be used, such as point-to-point interconnects, switched interconnects, or other technologies, including those known in the art. In some embodiments, the graphics processor 208 is coupled to the ring interconnect 212 via an I / O link 213.

[0071] Exemplary I / O link 213 represents at least one of a plurality of various I / O interconnects, including an on-package I / O interconnect that facilitates communication between various processor components and a high-performance embedded memory module 218, such as an eDRAM module. In some embodiments, each of processor cores 202A-202N and graphics processor 208 use embedded memory module 218 as a shared last-level cache.

[0072] In some embodiments, the processor cores 202A to 202N are homogeneous cores that execute the same instruction set architecture. In another embodiment, the processor cores 202A-202N are heterogeneous in terms of instruction set architecture (ISA), wherein one or more of the processor cores 202A-202N execute a first instruction set, while at least one of the other cores executes a subset of the first instruction set or a different instruction set. In one embodiment, the processor cores 202A-202N are heterogeneous in terms of microarchitecture, wherein one or more cores with relatively high power consumption are coupled with one or more power cores with lower power consumption. In addition, the processor 200 may be implemented on one or more chips, or as a SoC integrated circuit having the components illustrated in addition to other components.

[0073] Figure 3 3 is a block diagram of a graphics processor 300, which may be a discrete graphics processing unit, or may be a graphics processor integrated with multiple processing cores. In some embodiments, the graphics processor communicates via a memory-mapped I / O interface to registers on the graphics processor and using commands placed into the processor memory. In some embodiments, the graphics processor 300 includes a memory interface 314 for accessing memory. The memory interface 314 may be an interface to local memory, one or more internal caches, one or more shared external caches, and / or to system memory.

[0074] In some embodiments, the graphics processor 300 also includes a display controller 302, which is used to drive the display output data to the display device 320. The display controller 302 includes hardware for compositing one or more overlay planes and multiple layers of video or user interface elements for the display. The display device 320 can be an internal or external display device. In one embodiment, the display device 320 is a head-mounted display device, such as a virtual reality (VR) display device or an augmented reality (AR) display device. In some embodiments, the graphics processor 300 includes a video codec engine 306 for encoding media to one or more media encoding formats, decoding media from one or more media encoding formats, or transcoding media between one or more media encoding formats, including but not limited to: Moving Picture Experts Group (MPEG) format (such as MPEG-2), Advanced Video Coding (AVC) format (such as H.264 / MPEG-4AVC), and Society of Motion Picture and Television Engineers (SMPTE) 421M / VC-1, and Joint Photographic Experts Group (JPEG) format (such as JPEG, and motion JPEG (MJPEG) format).

[0075] In some embodiments, graphics processor 300 includes a block image transfer (BLIT) engine 304 for performing two-dimensional (2D) rasterizer operations, including, for example, bit boundary block transfers. However, in one embodiment, 2D graphics operations are performed using one or more components of a graphics processing engine (GPE) 310. In some embodiments, GPE 310 is a compute engine for performing graphics operations, including three-dimensional (3D) graphics operations and media operations.

[0076] In some embodiments, GPE 310 includes a 3D pipeline 312 for performing 3D operations, such as rendering three-dimensional images and scenes using processing functions that act on 3D primitive shapes (e.g., rectangles, triangles, etc.). 3D pipeline 312 includes programmable and fixed-function elements that perform various tasks within elements and / or generated execution threads of 3D / media subsystem 315. Although 3D pipeline 312 can be used to perform media operations, embodiments of GPE 310 also include a media pipeline 316 that is specifically used to perform media operations, such as video post-processing and image enhancement.

[0077] In some embodiments, the media pipeline 316 includes fixed-function or programmable logic units for performing one or more specialized media operations, such as video decoding acceleration, video deinterlacing, and video encoding acceleration, instead of, or on behalf of the video codec engine 306. In some embodiments, the media pipeline 316 additionally includes a thread generation unit to generate threads for execution on the 3D / media subsystem 315. The generated threads perform calculations for media operations on one or more graphics execution units included in the 3D / media subsystem 315.

[0078] In some embodiments, the 3D / media subsystem 315 includes logic for executing threads generated by the 3D pipeline 312 and the media pipeline 316. In one embodiment, the pipeline sends thread execution requests to the 3D / media subsystem 315, which includes thread dispatch logic for arbitrating and dispatching various requests for available thread execution resources. The execution resources include an array of graphics execution units for processing 3D threads and media threads. In some embodiments, the 3D / media subsystem 315 includes one or more internal caches for thread instructions and data. In some embodiments, the subsystem also includes a shared memory for sharing data between threads and for storing output data, which includes registers and addressable memory.

[0079] Graphics processing engine

[0080] Figure 4is a block diagram of a graphics processing engine 410 of a graphics processor according to some embodiments. In one embodiment, the graphics processing engine (GPE) 410 is Figure 3 A version of GPE 310 is shown in FIG. Figure 4 Those elements having the same reference numerals (or names) as elements of any other figure herein can operate or function in any manner similar to that described elsewhere herein, but are not limited thereto. Figure 3 410. The 3D pipeline 312 and the media pipeline 316 of the GPE 410. The media pipeline 316 is optional in some embodiments of the GPE 410 and may not be explicitly included in the GPE 410. For example and in at least one embodiment, separate media and / or image processors are coupled to the GPE 410.

[0081] In some embodiments, GPE 410 is coupled to or includes a command stream converter 403, which provides a command stream to 3D pipeline 312 and / or media pipeline 316. In some embodiments, command stream converter 403 is coupled to a memory, which can be a system memory, or one or more of an internal cache memory and a shared cache memory. In some embodiments, command stream converter 403 receives commands from the memory and sends these commands to 3D pipeline 312 and / or media pipeline 316. These commands are instructions taken from a circular buffer that stores commands for 3D pipeline 312 and media pipeline 316. In one embodiment, the circular buffer may additionally include a batch command buffer that stores a batch of multiple commands. Commands for 3D pipeline 312 may also include references to data stored in memory, such as, but not limited to, vertex data and geometry data for 3D pipeline 312 and / or image data and memory objects for media pipeline 316. 3D pipeline 312 and media pipeline 316 process commands and data by executing operations via logic within the respective pipelines or by dispatching one or more execution threads to graphics core array 414. In one embodiment, graphics core array 414 includes blocks of one or more graphics cores (e.g., (multiple) graphics cores 415A, (multiple) graphics cores 415B), each block including one or more graphics cores. Each graphics core includes a collection of graphics execution resources, including: general execution logic and graphics-specific execution logic for performing graphics operations and computing operations; and fixed-function texture processing logic and / or machine learning and artificial intelligence acceleration logic.

[0082] In various embodiments, the 3D pipeline 312 includes fixed function logic and programmable logic for processing one or more shader programs, such as vertex shaders, geometry shaders, pixel shaders, fragment shaders, compute shaders, or other shader programs, by processing instructions and dispatching execution threads to the graphics core array 414. The graphics core array 414 provides a unified execution resource block for use in processing these shader programs. The multi-function execution logic (e.g., execution unit) within the (multiple) graphics cores 415A-415B of the graphics core array 414 includes support for various 3D API shader languages ​​and can execute multiple simultaneous execution threads associated with multiple shaders.

[0083] In some embodiments, graphics core array 414 also includes execution logic for performing media functions such as video and / or image processing. In one embodiment, the execution unit additionally includes general purpose logic that can be programmed to perform parallel general purpose computing operations in addition to performing graphics processing operations. The general purpose logic can be used with Figure 1 (multiple) processor cores 107 or Figure 2 The general logic within the cores 202A-202N in the process performs processing operations in parallel or in combination.

[0084] Output data generated by threads executing on graphics core array 414 may output data to memory in unified return buffer (URB) 418. URB 418 may store data for multiple threads. In some embodiments, URB 418 may be used to send data between different threads executing on graphics core array 414. In some embodiments, URB 418 may additionally be used for synchronization between threads on the graphics core array and fixed function logic within shared function logic 420.

[0085] In some embodiments, graphics core array 414 is scalable such that the array includes a variable number of graphics cores, each having a variable number of execution units based on the target power and performance level of GPE 410. In one embodiment, execution resources are dynamically scalable such that execution resources can be enabled or disabled as needed.

[0086] The graphics core array 414 is coupled to shared function logic 420, which includes a plurality of resources that are shared between the graphics cores in the graphics core array. The shared functions within the shared function logic 420 are hardware logic units that provide specialized, complementary functions to the graphics core array 414. In various embodiments, the shared function logic 420 includes, but is not limited to, sampler 421 logic, math 422 logic, and inter-thread communication (ITC) 423 logic. Additionally, some embodiments implement one or more caches 425 within the shared function logic 420.

[0087] Shared functionality is implemented when the demand for a given specialized function is insufficient to be included in the graphics core array 414. Instead, a single instantiation of that specialized function is implemented as an independent entity in the shared functionality logic 420 and is shared between execution resources within the graphics core array 414. The exact set of functions shared between the graphics core array 414 and included within the graphics core array 414 varies from embodiment to embodiment. In some embodiments, specific shared functions within the shared functionality logic 420 that are widely used by the graphics core array 414 may be included within the shared functionality logic 416 within the graphics core array 414. In various embodiments, the shared functionality logic 416 within the graphics core array 414 may include some or all of the logic within the shared functionality logic 420. In one embodiment, all logic elements within the shared functionality logic 420 may be replicated within the shared functionality logic 416 of the graphics core array 414. In one embodiment, the shared functionality logic 420 is excluded in favor of the shared functionality logic 416 within the graphics core array 414.

[0088] Figure 5 is a block diagram of the hardware logic of graphics processor core 500 according to some embodiments described herein. Figure 5 Those elements having the same reference numerals (or names) as elements of any other figure herein may operate or function in any manner similar to that described elsewhere herein, but are not limited thereto. In some embodiments, the illustrated graphics processor core 500 is included in Figure 4 4. A graphics processor core 500 (sometimes referred to as a core slice) may be one or more graphics cores within a modular graphics processor. An example of a graphics processor core 500 is a graphics core slice, and based on a target power envelope and performance envelope, a graphics processor as described herein may include multiple graphics core slices. Each graphics processor core 500 may include a fixed function block 530 coupled to multiple sub-cores 501A-501F (also referred to as sub-slices), which include blocks of modular general purpose and fixed function logic.

[0089] In some embodiments, fixed function block 530 includes a geometry / fixed function pipeline 536, which may be shared by all sub-cores in graphics processor core 500, for example, in a lower performance and / or lower power graphics processor implementation. In various embodiments, geometry / fixed function pipeline 536 includes a 3D fixed function pipeline (e.g., as in Figure 3 and Figure 4 3D pipeline 312 in ), video front end unit, thread generator and thread dispatcher, and unified return buffer manager, which manages such as Figure 4The unified return buffer 418 of the unified return buffer.

[0090] In one embodiment, fixed function block 530 also includes graphics SoC interface 537, graphics microcontroller 538, and media pipeline 539. Graphics SoC interface 537 provides an interface between graphics core 500 and other processor cores within the system on chip integrated circuit. Graphics microcontroller 538 is a programmable subprocessor that can be configured to manage various functions of graphics processor 500, including thread dispatching, scheduling, and preemption. Media pipeline 539 (e.g., Figure 3 and Figure 4 The media pipeline 316 of the embodiment includes logic for facilitating decoding, encoding, pre-processing and / or post-processing of multimedia data including image data and video data. The media pipeline 539 implements media operations via requests to computational or sampling logic within the sub-cores 501A-501F.

[0091] In one embodiment, the SoC interface 537 enables the graphics processor core 500 to communicate with a general application processor core (e.g., a CPU) and / or other components within the SoC, including memory hierarchy elements such as a shared last-level cache memory, system RAM, and / or embedded on-chip or on-package DRAM. The SoC interface 537 may also enable communication with fixed-function devices within the SoC, such as a camera imaging pipeline, and enable the use and / or implementation of global memory atomicity, which may be shared between the graphics processor core 500 and the CPU within the SoC. The SoC interface 537 may also implement power management control for the graphics processor core 500 and enable an interface between the clock domain of the graphics core 500 and other clock domains within the SoC. In one embodiment, the SoC interface 537 enables receiving a command buffer from a command stream converter and a global thread dispatcher, which are configured to provide commands and instructions to each of the one or more graphics cores within the graphics processor. These commands and instructions may be dispatched to media pipeline 539 when media operations are to be performed, or to geometry and fixed function pipelines (eg, geometry and fixed function pipeline 536, geometry and fixed function pipeline 514) when graphics processing operations are to be performed.

[0092] The graphics microcontroller 538 may be configured to perform various scheduling and management tasks for the graphics processor core 500. In one embodiment, the graphics microcontroller 538 may perform graphics and / or computational workload scheduling for each graphics parallel engine within the execution unit (EU) arrays 502A-502F, 504A-504F within the sub-cores 501A-501F. In this scheduling model, host software executing on a CPU core of a SoC including the graphics processor core 500 may submit a workload via one of a plurality of graphics processor doorbells, which invokes a scheduling operation to the appropriate graphics engine. The scheduling operation includes determining which workload to run next, submitting the workload to the command stream converter, preempting an existing workload running on the engine, monitoring the progress of the workload, and notifying the host software when the workload is completed. In one embodiment, graphics microcontroller 538 may also facilitate low power or idle states for graphics processor core 500, thereby providing graphics processor core 500 with the ability to save and restore registers within graphics processor core 500 across low power state transitions independent of the operating system and / or graphics driver software on the system.

[0093] The graphics processor core 500 may have more or fewer sub-cores 501A-501F than those shown, up to N modular sub-cores. For each group of N sub-cores, the graphics processor core 500 may also include shared function logic 510, shared and / or cache memory 512, geometry / fixed function pipelines 514, and additional fixed function logic 516 for accelerating various graphics and compute processing operations. The shared function logic 510 may include logic that is shared by each of the N sub-cores within the graphics processor core 500, and logic that is shared by each of the N sub-cores within the graphics processor core 500. Figure 4 4. The shared function logic 420 (e.g., sampler logic, math logic, and / or inter-thread communication logic) of the graphics processor core 500 may be associated with the shared function logic 420 (e.g., sampler logic, math logic, and / or inter-thread communication logic). The shared and / or cache memory 512 may be a last-level cache for the set 501A-501F of N sub-cores within the graphics processor core 500, and may also serve as a shared memory accessible by multiple sub-cores. The geometry / fixed function pipeline 514 may be included within the fixed function block 530 instead of the geometry / fixed function pipeline 536, and the geometry / fixed function pipeline 514 may include the same or similar logic units.

[0094] In one embodiment, the graphics processor core 500 includes additional fixed function logic 516, which may include various fixed function acceleration logic for use by the graphics processor core 500. In one embodiment, the additional fixed function logic 516 includes an additional geometry pipeline for use in position-only shading. In position-only shading, there are two geometry pipelines: a full geometry pipeline within the geometry / fixed function pipelines 516, 536; and a culling pipeline, which is an additional geometry pipeline that may be included in the additional fixed function logic 516. In one embodiment, the culling pipeline is a streamlined version of the full geometry pipeline. The full pipeline and the culling pipeline can execute different instances of the same application, each with a separate context. Position-only shading can hide long culling runs for discarded triangles, thereby enabling shading to be completed earlier in some instances. For example and in one embodiment, the culling pipeline logic within the additional fixed function logic 516 can execute the position shader in parallel with the main application and generally generate key results faster than the full pipeline because the culling pipeline only takes the position attributes of the vertices and shades them without performing rasterization and rendering of the pixels to the frame buffer. The culling pipeline can use the generated key results to calculate visibility information for all triangles, regardless of whether those triangles are culled. The full pipeline (which may be referred to as a replay pipeline in this example) can consume this visibility information to skip culled triangles, thereby only shading visible triangles that are ultimately passed to the rasterization stage.

[0095] In one embodiment, the additional fixed function logic 516 may also include machine learning acceleration logic, such as fixed function matrix multiplication logic, which is used to include an implementation for machine learning training or inference.

[0096] Included within each graphics sub-core 501A-501F is a collection of execution resources that can be used to perform graphics operations, media operations, and compute operations in response to requests made by a graphics pipeline, a media pipeline, or a shader program. Graphics sub-cores 501A-501F include: multiple EU arrays 502A-502F, 504A-504F; thread dispatch and inter-thread communication (TD / IC) logic 503A-503F; 3D (e.g., texture) samplers 505A-505F; media samplers 506A-506F; shader processors 507A-507F; and shared local memory (SLM) 508A-508F. EU arrays 502A-502F, 504A-504F each include multiple execution units, which are general-purpose graphics processing units capable of performing floating-point and integer / fixed-point logic operations to serve graphics operations, media operations, or computing operations (including graphics programs, media programs, or computing shader programs). TD / IC logic 503A-503F performs local thread dispatch and thread control operations for execution units within the sub-core, and facilitates communication between threads executed on the execution units of the sub-core. 3D samplers 505A-505F can read textures or other 3D graphics-related data into memory. 3D samplers can read texture data in different ways based on the configured sample state and the texture format associated with a given texture. Media samplers 506A-506F can perform similar read operations based on the type and format associated with the media data. In one embodiment, each graphics sub-core 501A-501F can alternately include unified 3D and media samplers. Threads executing on execution units within each of the sub-cores 501A-501F may utilize shared local memory 508A-508F within each sub-core, such that threads executing within a thread group can execute using a common pool of on-chip memory.

[0097] Execution Unit

[0098] Figure 6A-6B Thread execution logic 600 is illustrated including an array of processing elements employed in a graphics processor core in accordance with embodiments described herein. Figure 6A-6B Those elements having the same reference numerals (or names) as elements of any other figure herein may operate or function in any manner similar to that described elsewhere herein, but are not limited thereto. Fig. 6A An overview of thread execution logic 600 is shown, which may include Figure 5 Each sub-core 501A-501F illustrates a variation of the hardware logic. Figure 6B Illustrated are exemplary internal details of an execution unit.

[0099] As in Fig. 6AAs shown in FIG. 6 , in some embodiments, thread execution logic 600 includes a shader processor 602, a thread dispatcher 604, an instruction cache 606, a scalable execution unit array including a plurality of execution units 608A-608N, a sampler 610, a data cache 612, and a data port 614. In one embodiment, the scalable execution unit array can be dynamically scaled by enabling or disabling one or more execution units (e.g., execution units 608A, 608B, 608C, 608D, up to any one of 608N-1 and 608N) based on the computational requirements of the workload. In one embodiment, the included components are interconnected via an interconnect structure that links to each of the components. In some embodiments, thread execution logic 600 includes one or more connections to a memory (such as a system memory or a cache memory) through one or more of the instruction cache 606, the data port 614, the sampler 610, and the execution units 608A-608N. In some embodiments, each execution unit (e.g., 608A) is a self-contained programmable general-purpose computing unit capable of executing multiple simultaneous hardware threads while processing multiple data elements in parallel for each thread. In various embodiments, the array of execution units 608A-608N is scalable to include any number of separate execution units.

[0100] In some embodiments, execution units 608A-608N are primarily used to execute shader programs. Shader processor 602 can process various shader programs and can dispatch execution threads associated with shader programs via thread dispatcher 604. In one embodiment, thread dispatcher includes logic for arbitrating thread initiation requests from graphics pipelines and media pipelines and instantiating the requested threads on one or more execution units in execution units 608A-608N. For example, a geometry pipeline can dispatch vertex shaders, tessellation shaders, or geometry shaders to field execution logic for processing. In some embodiments, thread dispatcher 604 can also process runtime thread generation requests from executed shader programs.

[0101] In some embodiments, execution units 608A-608N support an instruction set including native support for many standard 3D graphics shader instructions, so that shader programs from graphics libraries (e.g., Direct 3D and OpenGL) are executed with minimal conversion. These execution units support vertex and geometry processing (e.g., vertex programs, geometry programs, vertex shaders), pixel processing (e.g., pixel shaders, fragment shaders), and general processing (e.g., compute and media shaders). Each execution unit in execution units 608A-608N is capable of multi-issue single instruction multiple data (SIMD) execution, and multi-threaded operations enable efficient execution environments in the face of high latency memory accesses. Each hardware thread within each execution unit has a dedicated high-bandwidth register file and associated independent thread states. For pipelines capable of integer operations, single-precision floating-point operations, and double-precision floating-point operations, capable of SIMD branching capabilities, capable of logical operations, capable of transcendental operations, and capable of other miscellaneous operations, execution is for multiple issues per clock. While waiting for data from memory or one of the shared functions, dependency logic within execution units 608A-608N causes the waiting thread to sleep until the requested data has returned. While the waiting thread is sleeping, hardware resources can be dedicated to processing other threads. For example, during a delay associated with a vertex shader operation, an execution unit can perform operations for a pixel shader, a fragment shader, or another type of shader program including a different vertex shader.

[0102] Each of the execution units 608A-608N operates on an array of data elements. The number of data elements is the "execution size," or number of lanes for an instruction. An execution lane is a logical unit used for the execution of data element access, masking, and flow control within an instruction. The number of lanes may be independent of the number of physical arithmetic logic units (ALUs) or floating point units (FPUs) used for a particular graphics processor. In some embodiments, the execution units 608A-608N support integer and floating point data types.

[0103] The execution unit instruction set includes SIMD instructions. Various data elements can be stored in registers as compact data types, and the execution unit will process each element based on the data size of the element. For example, when a 256-bit wide vector is operated, 256 bits of the vector are stored in registers, and the execution unit operates the vector as four separate 64-bit compact data elements (quad word (QW) size data elements), eight separate 32-bit compact data elements (double word (DW) size data elements), sixteen separate 16-bit compact data elements (word (W) size data elements) or thirty-two separate 8-bit data elements (byte (B) size data elements). However, different vector widths and register sizes are possible.

[0104] In one embodiment, one or more execution units may be combined into a fused execution unit 609A-609N, which has a common thread control logic (607A-607N) for the fused EU. Multiple EUs may be fused into a group of EUs. Each EU in the fused group of EUs may be configured to execute a separate SIMD hardware thread. The number of EUs in the fused group of EUs may vary depending on the embodiment. In addition, various SIMD widths may be executed EU by EU, including but not limited to SIMD8, SIMD16, and SIMD32. Each fused graphics execution unit 609A-609N includes at least two execution units. For example, the fused execution unit 609A includes a first EU 608A, a second EU 608B, and a common thread control logic 607A for the first EU 608A and the second EU 608B. Thread control logic 607A controls threads executing on fused graphics execution unit 609A, allowing each EU within fused execution units 609A-609N to execute using a common instruction pointer register.

[0105] One or more internal instruction caches (e.g., 606) are included in thread execution logic 600 to cache thread instructions for execution units. In some embodiments, one or more data caches (e.g., 612) are included to cache thread data during thread execution. In some embodiments, sampler 610 is included to provide texture sampling for 3D operations and media sampling for media operations. In some embodiments, sampler 610 includes specialized texture or media sampling functions to process texture data or media data during the sampling process before providing the sampled data to the execution unit.

[0106] During execution, the graphics pipeline and the media pipeline send thread initiation requests to the thread execution logic 600 via thread generation and dispatch logic. Once a set of geometric objects have been processed and rasterized into pixel data, the pixel processor logic (e.g., pixel shader logic, fragment shader logic, etc.) within the shader processor 602 is called to further calculate output information and cause the results to be written to the output surface (e.g., color buffer, depth buffer, stencil buffer, etc.). In some embodiments, the pixel shader or fragment shader calculates the value of each vertex attribute, and the value of each vertex attribute is interpolated across the rasterized object. In some embodiments, the pixel processor logic within the shader processor 602 then executes the pixel shader program or fragment shader program supplied by the application programming interface (API). To execute the shader program, the shader processor 602 dispatches the thread to the execution unit (e.g., 608A) via the thread dispatcher 604. In some embodiments, the shader processor 602 uses the texture sampling logic in the sampler 610 to access the texture data in the texture map stored in the memory. Arithmetic operations on texture data and input geometry data compute pixel color data for each geometry fragment, or discard one or more pixels without further processing.

[0107] In some embodiments, data port 614 provides a memory access mechanism for thread execution logic 600 to output processed data to memory for further processing in the graphics processor output pipeline. In some embodiments, data port 614 includes or is coupled to one or more cache memories (e.g., data cache 612) to cache data for memory access via the data port.

[0108] like Figure 6B As shown in FIG. 6 , the graphics execution unit 608 may include an instruction fetch unit 637, a general register file array (GRF) 624, an architectural register file array (ARF) 626, a thread arbiter 622, an issue unit 630, a branch unit 632, a set of SIMD floating point units (FPUs) 634, and a set of dedicated integer SIMD ALUs 635 in one embodiment. The GRF 624 and ARF 626 include a set of general register files and architectural register files associated with each simultaneous hardware thread that may be active in the graphics execution unit 608. In one embodiment, a per-thread architectural state is maintained in the ARF 626, while data used during thread execution is stored in the GRF 624. The execution state of each thread, including the instruction pointer for each thread, may be maintained in thread-specific registers in the ARF 626.

[0109] In one embodiment, graphics execution unit 608 has an architecture that is a combination of simultaneous multithreading (SMT) and fine-grained interleaved multithreading (IMT). The architecture has a modular configuration that can be fine-tuned at design time based on the target number of simultaneous threads and the number of registers per execution unit, where execution unit resources are divided across logic for executing multiple simultaneous threads.

[0110] In one embodiment, the graphics execution unit 608 can issue multiple instructions in coordination, and these instructions can be different instructions respectively. The thread arbiter 622 of the graphics execution unit thread 608 can dispatch the instruction to one of the following items for execution: the send unit 630, the branch unit 632 or (multiple) SIMD FPU 634. Each execution thread can access 128 general registers in the GRF 624, wherein each register can store 32 bytes that can be accessed as a SIMD 8 element vector with 32-bit data elements. In one embodiment, each execution unit thread has access to 4 kilobytes in the GRF 624, but the embodiment is not limited thereto, and more or less register resources can be provided in other embodiments. In one embodiment, up to seven threads can be executed synchronously, but the number of threads per execution unit can also vary according to the embodiment. In an embodiment in which seven threads can access 4 kilobytes, the GRF 624 can store a total of 28 kilobytes. Flexible addressing modes can allow multiple registers to be addressed together, thereby establishing a substantially wider register or representing a strided rectangular block data structure.

[0111] In one embodiment, memory operations, sampler operations, and other longer latency system communications are dispatched via "send" instructions executed by message passing send unit 630. In one embodiment, branch instructions are dispatched to a dedicated branch unit 632 to facilitate SIMD divergence and eventual convergence.

[0112] In one embodiment, the graphics execution unit 608 includes one or more SIMD floating point units (FPUs) 634 for performing floating point operations. In one embodiment, the FPUs 634 also support integer computations. In one embodiment, the FPUs 634 can SIMD perform up to a number M of 32-bit floating point (or integer) operations, or SIMD perform up to 2M 16-bit integer or 16-bit floating point operations. In one embodiment, at least one of the FPUs provides extended math capabilities that support high throughput transcendental math functions and double precision 64-bit floating points. In some embodiments, a set 635 of 8-bit integer SIMD ALUs also exists and can be specifically optimized to perform operations associated with machine learning computations.

[0113] In one embodiment, an array of multiple instances of graphics execution unit 608 may be instantiated in graphics sub-core groupings (e.g., sub-slices). For scalability, product architects may choose the exact number of execution units per sub-core grouping. In one embodiment, execution unit 608 may execute instructions across multiple execution lanes. In further embodiments, each thread executed on graphics execution unit 608 is executed on a different lane.

[0114] Figure 7 is a block diagram illustrating a graphics processor instruction format 700 according to some embodiments. In one or more embodiments, the graphics processor execution unit supports an instruction set having instructions in multiple formats. Solid line boxes illustrate components that are typically included in the execution unit instructions, while dashed lines include components that are optional or included only in a subset of the instructions. In some embodiments, the instruction format 700 described and illustrated are macroinstructions because they are instructions supplied to the execution unit, as opposed to microinstructions that result from instruction decoding that occurs once the instruction is processed.

[0115] In some embodiments, the graphics processor execution unit natively supports instructions in the 128-bit instruction format 710. Based on the selected instructions, instruction options, and number of operands, a 64-bit compact instruction format 730 may be used for some instructions. The native 128-bit instruction format 710 provides access to all instruction options, while some options and operations are limited in the 64-bit format 730. The native instructions available in the 64-bit format 730 vary from embodiment to embodiment. In some embodiments, the instructions are partially compressed using a set of index values ​​in the index field 713. The execution unit hardware references a set of compression tables based on the index values ​​and uses the compression table output to reconstruct the native instructions of the 128-bit instruction format 710.

[0116] For each format, the instruction opcode 712 defines the operation to be performed by the execution unit. The execution unit executes each instruction in parallel across multiple data elements of each operand. For example, in response to an addition instruction, the execution unit performs a synchronous addition operation across each color channel representing a texture element or a picture element. By default, the execution unit executes each instruction across all data channels of the operand. In some embodiments, the instruction control field 714 enables control of certain execution options, such as channel selection (e.g., assertion) and data channel order (e.g., mixing). For instructions of the 128-bit instruction format 710, the execution size field 716 limits the number of data channels that will be executed in parallel. In some embodiments, the execution size field 716 is not available for the 64-bit compact instruction format 730.

[0117] Some execution unit instructions have up to three operands, including two source operands src0 720, src1 722, and one destination operand 718. In some embodiments, the execution unit supports dual destination instructions, where one of the dual destinations is implicit. Data manipulation instructions may have a third source operand (e.g., SRC2 724), where the instruction opcode 712 determines the number of source operands. The last source operand of an instruction may be an immediate (e.g., hard-coded) value passed with the instruction.

[0118] In some embodiments, the 128-bit instruction format 710 includes an access / addressing mode field 726 that specifies, for example, whether direct register addressing mode or indirect register addressing mode is used. When direct register addressing mode is used, the register address of one or more operands is provided directly by bits in the instruction.

[0119] In some embodiments, the 128-bit instruction format 710 includes an access / addressing mode field 726 that specifies the addressing mode and / or access mode of the instruction. In one embodiment, the access mode is used to define the data access alignment for the instruction. Some embodiments support access modes including a 16-byte aligned access mode and a 1-byte aligned access mode, wherein the byte alignment of the access mode determines the access alignment of the instruction operand. For example, when in a first mode, the instruction may use byte-aligned addressing for source operands and destination operands, and when in a second mode, the instruction may use 16-byte aligned addressing for all source operands and destination operands.

[0120] In one embodiment, the addressing mode portion of the access / addressing mode field 726 determines whether the instruction uses direct addressing or indirect addressing. When direct register addressing mode is used, the bits in the instruction directly provide the register address of one or more operands. When indirect register addressing mode is used, the register address of one or more operands can be calculated based on the address register value and the address immediate field in the instruction.

[0121] In some embodiments, instructions are grouped based on the opcode 712 bit field to simplify opcode decoding 740. For 8-bit opcodes, bit 4, bit 5, and bit 6 allow the execution unit to determine the type of opcode. The exact opcode grouping shown is only an example. In some embodiments, the move and logic opcode group 742 includes data movement and logic instructions (e.g., move (mov), compare (cmp)). In some embodiments, the move and logic group 742 shares five most significant bits (MSB), wherein the move (mov) instruction adopts the form of 0000xxxxb, and the logic instruction adopts the form of 0001xxxxb. The flow control instruction group 744 (e.g., call (call), jump (jmp)) includes the instruction of 0010xxxxb (e.g., 0x20) form. The miscellaneous instruction group 746 includes a mixture of instructions, including the synchronization instruction (e.g., wait (wait), send (send)) of 0011xxxxb (e.g., 0x30) form. The parallel math instruction group 748 includes component-wise arithmetic instructions (e.g., add, multiply (mul)) of the form 0100xxxxb (e.g., 0x40). The parallel math group 748 performs arithmetic operations in parallel across data lanes. The vector math group 750 includes arithmetic instructions (e.g., dp4) of the form 0101xxxxb (e.g., 0x50). The vector math group performs arithmetic on vector operands, such as dot product calculations.

[0122] Graphics Pipeline

[0123] Figure 8 is a block diagram of another embodiment of a graphics processor 800 . Figure 8 Those elements having the same reference numerals (or names) as elements of any other figure herein may operate or function in any manner similar to that described elsewhere herein, but are not limited thereto.

[0124] In some embodiments, the graphics processor 800 includes a graphics pipeline 820, a media pipeline 830, a display engine 840, a thread execution logic 850, and a rendering output pipeline 870. In some embodiments, the graphics processor 800 is a graphics processor within a multi-core processing system that includes one or more general-purpose processing cores. The graphics processor is controlled by register writes to one or more control registers (not shown) or via commands issued to the graphics processor 800 through the ring interconnect 802. In some embodiments, the ring interconnect 802 couples the graphics processor 800 to other processing components, such as other graphics processors or general-purpose processors. Commands from the ring interconnect 802 are interpreted by a command stream converter 803, which supplies instructions to various components of the geometry pipeline 820 or the media pipeline 830.

[0125] In some embodiments, the command stream converter 803 directs the operation of the vertex fetcher 805, which reads vertex data from memory and executes vertex processing commands provided by the command stream converter 803. In some embodiments, the vertex fetcher 805 provides the vertex data to the vertex shader 807, which performs coordinate space transformation and lighting operations on each vertex. In some embodiments, the vertex fetcher 805 and the vertex shader 807 execute the vertex processing instructions by dispatching execution threads to the execution units 852A-852B via the thread dispatcher 831.

[0126] In some embodiments, execution units 852A-852B are arrays of vector processors with instruction sets for performing graphics operations and media operations. In some embodiments, execution units 852A-852B have an attached L1 cache 851 that is dedicated to each array or shared between arrays. The cache can be configured as a data cache, an instruction cache, or partitioned into a single cache containing data and instructions in different partitions.

[0127] In some embodiments, the geometry pipeline 820 includes a tessellation component for performing hardware accelerated tessellation of 3D objects. In some embodiments, the programmable shell shader 811 configures the tessellation operation. The programmable domain shader 817 provides back-end evaluation of the tessellation output. The tessellation 813 operates under the direction of the shell shader 811 and includes dedicated logic for generating a detailed set of geometric objects based on a coarse geometric model that is provided as input to the geometry pipeline 820. In some embodiments, if tessellation is not used, the tessellation components (e.g., the shell shader 811, the tessellation 813, and the domain shader 817) can be bypassed.

[0128] In some embodiments, the complete geometric object may be processed by the geometry shader 819 via one or more threads dispatched to the execution units 852A-852B, or may proceed directly to the clipper 829. In some embodiments, the geometry shader operates on the entire geometric object rather than on vertices or vertex patches as in previous stages of the graphics pipeline. If tessellation is disabled, the geometry shader 819 receives input from the vertex shader 807. In some embodiments, the geometry shader 819 is programmable by a geometry shader program to perform geometry tessellation when the tessellation unit is disabled.

[0129] Before rasterization, the clipper 829 processes the vertex data. The clipper 829 can be a fixed function clipper or a programmable clipper with clipping and geometry shader functions. In some embodiments, the rasterizer and depth test component 873 in the rendering output pipeline 870 dispatches a pixel shader to convert the geometric object into a pixel-by-pixel representation. In some embodiments, the pixel shader logic is included in the thread execution logic 850. In some embodiments, the application can bypass the rasterizer and depth test component 873 and access the unrasterized vertex data via the outflow unit 823.

[0130] The graphics processor 800 has an interconnect bus, interconnect structure, or some other interconnect mechanism that allows data and messages to be passed among the main components of the processor. In some embodiments, the execution units 852A-852B and associated logic units (e.g., L1 cache 851, sampler 854, texture cache 858, etc.) are interconnected via data ports 856 to perform memory accesses and communicate with the rendering output pipeline components of the processor. In some embodiments, the sampler 854, caches 851, 858, and execution units 852A-852B each have a separate memory access path. In one embodiment, the texture cache 858 can also be configured as a sampler cache.

[0131] In some embodiments, the rendering output pipeline 870 includes a rasterizer and depth test component 873, which converts vertex-based objects into associated pixel-based representations. In some embodiments, the rasterizer logic includes a windower / masker unit for performing fixed-function triangle and line rasterization. An associated rendering cache 878 and depth cache 879 are also available in some embodiments. The pixel operation component 877 performs pixel-based operations on data, but in some instances, pixel operations associated with 2D operations (e.g., using mixed bit block image transfer) are performed by the 2D engine 841, or replaced by the display controller 843 using an overlay display plane when displaying. In some embodiments, a shared L3 cache 875 can be used for all graphics components, allowing data to be shared without using main system memory.

[0132] In some embodiments, the graphics processor media pipeline 830 includes a media engine 837 and a video front end 834. In some embodiments, the video front end 834 receives pipeline commands from the command stream converter 803. In some embodiments, the media pipeline 830 includes a separate command stream converter. In some embodiments, the video front end 834 processes the media commands before sending them to the media engine 837. In some embodiments, the media engine 837 includes a thread generation function for generating threads for dispatching to the thread execution logic 850 via the thread dispatcher 831.

[0133] In some embodiments, the graphics processor 800 includes a display engine 840. In some embodiments, the display engine 840 is external to the processor 800 and is coupled to the graphics processor via the ring interconnect 802, or some other interconnect bus or structure. In some embodiments, the display engine 840 includes a 2D engine 841 and a display controller 843. In some embodiments, the display engine 840 includes dedicated logic that can operate independently of the 3D pipeline. In some embodiments, the display controller 843 is coupled to a display device (not shown), which can be a system integrated display device (such as in a laptop computer) or an external display device attached via a display device connector.

[0134] In some embodiments, geometry pipeline 820 and media pipeline 830 can be configured to perform operations based on multiple graphics and media programming interfaces, and are not dedicated to any application programming interface (API). In some embodiments, the driver software of the graphics processor converts the API call dedicated to a specific graphics or media library into a command that can be processed by the graphics processor. In some embodiments, all open graphics libraries (OpenGL), open computing language (OpenCL) and / or Vulkan graphics and computing APIs from Khronos Group are provided with support. In some embodiments, support can also be provided for the Direct3D library from Microsoft. In some embodiments, the combination of these libraries can be supported. Support can also be provided for the open source computer vision library (OpenCV). If the mapping from the pipeline of the future API to the pipeline of the graphics processor can be carried out, the future API with compatible 3D pipelines will also be supported.

[0135] Graphics Pipeline Programming

[0136] Fig.9A is a block diagram illustrating a graphics processor command format 900 according to some embodiments.

[0137] Fig. 9B is a block diagram illustrating a graphics processor command sequence 910 according to an embodiment. Fig.9AThe solid-line boxes in illustrate components that are generally included in the graphics commands, while the dashed lines include components that are optional or included only in a subset of the graphics commands. Fig.9A An exemplary graphics processor command format 900 includes data fields for identifying the client 902 of the command, a command operation code (opcode) 904, and data 906. A subopcode 905 and a command size 908 are also included in some commands.

[0138] In some embodiments, client 902 specifies the client unit of processing command data of graphics device. In some embodiments, graphics processor command parser checks the client field of each command to adjust the further processing of command and route command data to appropriate client unit. In some embodiments, graphics processor client unit includes memory interface unit, rendering unit, 2D unit, 3D unit and media unit. Each client unit has corresponding processing pipeline for processing command. Once command is received by client unit, client unit reads opcode 904 and sub-opcode 905 (if exists) to determine the operation to be performed. Client unit uses information in data field 906 to execute command. For some commands, explicit command size 908 is expected to specify the size of command. In some embodiments, command parser automatically determines the size of at least some commands in command based on command opcode. In some embodiments, command is aligned via multiples of double word.

[0139] Fig. 9B 910. In some embodiments, software or firmware of a data processing system featuring an embodiment of a graphics processor uses a version of the command sequence shown to establish, execute, and terminate a set of graphics operations. The sample command sequence is shown and described for exemplary purposes only, as the embodiments are not limited to these particular commands or command sequences. Moreover, the commands may be issued as batches of commands in a command sequence so that the graphics processor will process the command sequence in an at least partially simultaneous manner.

[0140] In some embodiments, the graphics processor command sequence 910 may begin with a pipeline dump clear command 912 to cause any active graphics pipeline to complete the currently pending commands of the pipeline. In some embodiments, the 3D pipeline 922 and the media pipeline 924 do not operate concurrently. The pipeline dump clear is performed to cause the active graphics pipeline to complete any pending commands. In response to the pipeline dump clear, the command parser for the graphics processor will suspend command processing until the active painting engine completes the pending operation and the associated read cache is invalidated. Optionally, any data marked as "dirty" in the rendering cache can be dumped to memory. In some embodiments, the pipeline dump clear command 912 can be used for pipeline synchronization, or before placing the graphics processor in a low power state.

[0141] In some embodiments, when a command sequence requires the graphics processor to explicitly switch between pipelines, a pipeline select command 913 is used. In some embodiments, a pipeline select command 913 is only required once in an execution context before issuing pipeline commands unless the context will issue commands for both pipelines. In some embodiments, a pipeline flush command 912 is required immediately before a pipeline switch via a pipeline select command 913.

[0142] In some embodiments, pipeline control commands 914 configure the graphics pipeline for operation and are used to program the 3D pipeline 922 and the media pipeline 924. In some embodiments, pipeline control commands 914 configure the pipeline state of the active pipeline. In one embodiment, pipeline control commands 914 are used for pipeline synchronization and for clearing data from one or more cache memories within the active pipeline before processing a batch of commands.

[0143] In some embodiments, the return buffer state command 916 is used to configure a set of return buffers for a corresponding pipeline to write data. Some pipeline operations require allocating, selecting, or configuring one or more return buffers into which the operation writes intermediate data during processing. In some embodiments, the graphics processor also uses one or more return buffers to store output data and perform cross-thread communication. In some embodiments, the return buffer state 916 includes selecting the size and number of return buffers to be used for the set of pipeline operations.

[0144] The remaining commands in the command sequence differ based on the active pipeline for operation.Based on pipeline decision 920 , the command sequence is tailored for either 3D pipeline 922 starting at 3D pipeline state 930 , or media pipeline 924 starting at media pipeline state 940 .

[0145] The commands for configuring the 3D pipeline state 930 include 3D state setting commands for vertex buffer state, vertex element state, constant color state, depth buffer state, and other state variables that will be configured before processing 3D primitive commands. The values ​​of these commands are determined at least in part based on the specific 3D API in use. In some embodiments, if certain pipeline elements will not be used, the 3D pipeline state 930 commands can also selectively disable or bypass those elements.

[0146] In some embodiments, 3D primitive 932 commands are used to submit 3D primitives to be processed by the 3D pipeline. The commands and associated parameters passed to the graphics processor via the 3D primitive 932 commands are forwarded to the vertex fetching function in the graphics pipeline. The vertex fetching function uses the 3D primitive 932 command data to generate multiple vertex data structures. The vertex data structure is stored in one or more return buffers. In some embodiments, 3D primitive 932 commands are used to perform vertex operations on 3D primitives via vertex shaders. In order to process vertex shaders, the 3D pipeline 922 dispatches the shader execution thread to the graphics processor execution unit.

[0147] In some embodiments, the 3D pipeline 922 is triggered via an execution 934 command or event. In some embodiments, a register write triggers the command execution. In some embodiments, the execution is triggered via a "go" or "kick" command in the command sequence. In one embodiment, the command execution is triggered using a pipeline synchronization command to dump and clear the command sequence through the graphics pipeline. The 3D pipeline will perform geometry processing for 3D primitives. Once the operation is completed, the resulting geometric objects are rasterized and the pixel engine shades the resulting pixels. For those operations, additional commands for controlling pixel shading and pixel backend operations may also be included.

[0148] In some embodiments, when performing media operations, the graphics processor command sequence 910 follows the media pipeline 924 path. Generally, the specific purpose and manner of programming for the media pipeline 924 depends on the media or computing operation to be performed. During media decoding, specific media decoding operations can be transferred to the media pipeline. In some embodiments, the media pipeline can also be bypassed, and the resources provided by one or more general processing cores can be used to perform media decoding in whole or in part. In one embodiment, the media pipeline also includes elements for general purpose graphics processor unit (GPGPU) operations, wherein the graphics processor is used to perform SIMD vector operations using compute shader programs, which are not explicitly related to the rendering of graphics primitives.

[0149] In some embodiments, the media pipeline 924 is configured in a similar manner to the 3D pipeline 922. A set of commands for configuring the media pipeline state 940 is dispatched or placed into a command queue before the media object commands 942. In some embodiments, the commands for the media pipeline state 940 include data for configuring the media pipeline elements that will be used to process the media objects. This includes data for configuring the video decoding and video encoding logic within the media pipeline, such as encoding or decoding formats. In some embodiments, the commands for the media pipeline state 940 also support the use of one or more pointers to "indirect" state elements that contain batches of state settings.

[0150] In some embodiments, media object commands 942 supply pointers to media objects for processing by the media pipeline. The media object includes a memory buffer that contains video data to be processed. In some embodiments, all media pipeline states must be valid before issuing media object commands 942. Once the pipeline state is configured and media object commands 942 are queued, media pipeline 924 is triggered via an execution command 944 or an equivalent execution event (e.g., a register write). The output from media pipeline 924 can then be post-processed by operations provided by 3D pipeline 922 or media pipeline 924. In some embodiments, GPGPU operations are configured and executed in a manner similar to media operations.

[0151] Graphics software architecture

[0152] Fig.10 An exemplary graphics software architecture for data processing system 1000 according to some embodiments is illustrated. In some embodiments, the software architecture includes 3D graphics application 1010, operating system 1020, and at least one processor 1030. In some embodiments, processor 1030 includes graphics processor 1032 and one or more general purpose processor cores 1034. Graphics application 1010 and operating system 1020 are each executed in system memory 1050 of the data processing system.

[0153] In some embodiments, the 3D graphics application 1010 includes one or more shader programs that include shader instructions 1012. The shader language instructions may be in a high-level shader language, such as High Level Shader Language (HLSL) or OpenGL Shader Language (GLSL). The application also includes executable instructions 1014 in a machine language suitable for execution by the general purpose processor core 1034. The application also includes graphics objects 1016 defined by vertex data.

[0154] In some embodiments, operating system 1020 is from Microsoft Corporation Operating system 1020 may be a graphics API 1022, such as Direct3D API, OpenGL API or Vulkan API. When Direct3D API is in use, operating system 1020 uses front-end shader compiler 1024 to compile any shader instruction 1012 using HLSL into a lower-level shader language. Compilation may be just-in-time (JIT) compilation, or application executable shader precompilation. In some embodiments, during the compilation of 3D graphics application 1010, high-level shaders are compiled into low-level shaders. In some embodiments, shader instructions 1012 are provided in an intermediate form, such as a certain version of the standard portable intermediate representation (SPIR) used by Vulkan API.

[0155] In some embodiments, the user mode graphics driver 1026 includes a backend shader compiler 1027 for converting shader instructions 1012 into hardware-specific representations. When the OpenGL API is in use, the shader instructions 1012 in the GLSL high-level language are passed to the user mode graphics driver 1026 for compilation. In some embodiments, the user mode graphics driver 1026 uses the operating system kernel mode function 1028 to communicate with the kernel mode graphics driver 1029. In some embodiments, the kernel mode graphics driver 1029 communicates with the graphics processor 1032 to dispatch commands and instructions.

[0156] IP Core Implementation

[0157] One or more aspects of at least one embodiment may be implemented by representative code stored on a machine-readable medium, which represents and / or defines logic within an integrated circuit (such as a processor). For example, a machine-readable medium may include instructions representing various logics within a processor. When read by a machine, the instructions may cause the machine to manufacture logic for performing the techniques described herein. Such representations (referred to as "IP cores") are reusable units of logic for an integrated circuit, which may be stored on a tangible, machine-readable medium as a hardware model describing the structure of the integrated circuit. The hardware model may be supplied to each consumer or manufacturing facility that loads the hardware model on a manufacturing machine that manufactures the integrated circuit. The integrated circuit may be manufactured so that the circuit performs the operations described in association with any of the embodiments described herein.

[0158] Fig.11A1 is a block diagram illustrating an IP core development system 1100 according to an embodiment, which can be used to manufacture integrated circuits to perform operations. The IP core development system 1100 can be used to generate modular, reusable designs that can be incorporated into a larger design or used to build an entire integrated circuit (e.g., a SOC integrated circuit). The design facility 1130 can generate a software simulation 1110 of the IP core design using a high-level programming language (e.g., C / C++). The software simulation 1110 can be used to design, test and verify the behavior of the IP core using a simulation model 1112. The simulation model 1112 can include functional simulation, behavioral simulation and / or timing simulation. A register transfer level (RTL) design 1115 can then be created or synthesized from the simulation model 1112. The RTL design 1115 is an abstraction of the behavior of an integrated circuit (including associated logic executed using the modeled digital signals) that models the flow of digital signals between hardware registers. In addition to the RTL design 1115, a lower level design at the logic level or transistor level can also be created, designed or synthesized. Thus, the specific details of the initial design and simulation can be different.

[0159] The RTL design 1115 or an equivalent solution can be further synthesized into a hardware model 1120 by the design facility, which can adopt a hardware description language (HDL) or some other representation of physical design data. The HDL can be further simulated or tested to verify the IP core design. The IP core design can be stored using a non-volatile memory 1140 (e.g., a hard disk, flash memory, or any non-volatile storage medium) for delivery to a third-party manufacturing facility 1165. Alternatively, the IP core design can be transmitted via a wired connection 1150 or a wireless connection 1160 (e.g., via the Internet). The manufacturing facility 1165 can then manufacture an integrated circuit based at least in part on the IP core design. The manufactured integrated circuit can be configured to perform operations according to at least one embodiment described herein.

[0160] Fig. 11BA cross-sectional side view of an integrated circuit package assembly 1170 according to some embodiments described herein is illustrated. The integrated circuit package assembly 1170 illustrates an implementation of one or more processors or accelerator devices as described herein. The package assembly 1170 includes a plurality of hardware logic units 1172, 1174 connected to a substrate 1180. The logic 1172, 1174 may be implemented at least in part in configurable logic or fixed-function logic hardware, and may include one or more portions of any of the processor cores, graphics processors, or other accelerator devices described herein (multiple) processor cores, (multiple) graphics processors, or other accelerator devices. Each logic unit 1172, 1174 may be implemented within a semiconductor die and coupled to the substrate 1180 via an interconnect structure 1173. The interconnect structure 1173 may be configured to route electrical signals between the logic 1172, 1174 and the substrate 1180, and may include interconnects such as, but not limited to, bumps or pillars. In some embodiments, the interconnect structure 1173 can be configured to route electrical signals, such as, for example, input / output (I / O) signals and / or power or ground signals associated with the operation of the logic 1172, 1174. In some embodiments, the substrate 1180 is a laminate substrate based on epoxy resin. In other embodiments, the package substrate 1180 may include other suitable types of substrates. The package assembly 1170 can be connected to other electrical devices via the package interconnect 1183. The package interconnect 1183 can be coupled to the surface of the substrate 1180 to route electrical signals to other electrical devices, such as a motherboard, other chipsets, or multi-chip modules.

[0161] In some embodiments, logic units 1172, 1174 are electrically coupled to a bridge 1182 configured to route electrical signals between logic 1172 and logic 1174. Bridge 1182 may be a dense interconnect structure that provides routing for electrical signals. Bridge 1182 may include a bridge substrate composed of glass or a suitable semiconductor material. Circuit features may be formed on the bridge substrate to provide a chip-to-chip connection between logic 1172 and logic 1174.

[0162] Although two logic cells 1172, 1174 and bridge 1182 are illustrated, the embodiments described herein may include more or fewer logic cells on one or more dies. The one or more dies may be connected by zero or more bridges, as bridge 1182 may be excluded when the logic is included on a single die. Alternatively, multiple dies or logic cells may be connected by one or more bridges. Additionally, multiple logic cells, dies, and bridges may be connected together in other possible configurations, including three-dimensional configurations.

[0163] Exemplary System-on-Chip Integrated Circuit

[0164] Figure 12-14B An exemplary integrated circuit and associated graphics processor that can be manufactured using one or more IP cores according to various embodiments described herein are illustrated. In addition to what is illustrated, other logic and circuits may also be included, including additional graphics processors / cores, peripheral interface controllers, or general purpose processor cores.

[0165] Fig.12 1 is a block diagram illustrating an exemplary system-on-chip integrated circuit 1200 that can be manufactured using one or more IP cores according to an embodiment. The exemplary integrated circuit 1200 includes one or more application processors 1205 (e.g., CPUs), at least one graphics processor 1210, and may additionally include an image processor 1215 and / or a video processor 1220, any of which may be modular IP cores from the same design facility or multiple different design facilities. The integrated circuit 1200 includes peripheral or bus logic, including a USB controller 1225, a UART controller 1230, a SPI / SDIO controller 1235, and an I2S / I2C controller 1240. In addition, the integrated circuit may include a display device 1245, which is coupled to one or more of a high-definition multimedia interface (HDMI) controller 1250 and a mobile industry processor interface (MIPI) display interface 1255. Storage may be provided by a flash memory subsystem 1260 (including flash memory and a flash memory controller). A memory interface may be provided via a memory controller 1265 to obtain access to SDRAM or SRAM memory devices. Some integrated circuits additionally include an embedded security engine 1270 .

[0166] Figure 13A-13B is a block diagram illustrating an exemplary graphics processor for use within a SoC according to embodiments described herein. Fig.13A An exemplary graphics processor 1310 of a system on chip integrated circuit that may be fabricated using one or more IP cores is illustrated in accordance with an embodiment. Fig. 13B An additional exemplary graphics processor 1340 is illustrated that may be fabricated using one or more IP cores of a system on a chip integrated circuit in accordance with an embodiment. Fig.13A Graphics processor 1310 is an example of a low-power graphics processor core. Fig. 13B The graphics processor 1340 of FIG. 1 is an example of a higher performance graphics processor core. Each of the graphics processors 1310 and 1340 may be Fig.12 A variant of graphics processor 1210.

[0167] like Fig.13AAs shown in , the graphics processor 1310 includes a vertex processor 1305 and one or more fragment processors 1315A-1315N (e.g., 1315A, 1315B, 1315C, 1315D, all the way to 1315N-1 and 1315N). The graphics processor 1310 can execute different shader programs via separate logic, so that the vertex processor 1305 is optimized to perform operations for the vertex shader program, while the one or more fragment processors 1315A-1315N perform fragment (e.g., pixel) shading operations for the fragment or pixel shader program. The vertex processor 1305 performs the vertex processing stage of the 3D graphics pipeline and generates primitive data and vertex data. (Multiple) fragment processors 1315A-1315N use the primitive data and vertex data generated by the vertex processor 1305 to generate a frame buffer that is displayed on the display device. In one embodiment, the fragment processor(s) 1315A-1315N are optimized to execute fragment shader programs as provided in the OpenGL API, which can be used to perform similar operations as pixel shader programs as provided in the Direct 3D API.

[0168] The graphics processor 1310 additionally includes one or more memory management units (MMUs) 1320A-1320B, cache(s) 1325A-1325B, and circuit interconnect(s) 1330A-1330B. The one or more MMUs 1320A-1320B provide virtual to physical address mappings for the graphics processor 1310 (including for the vertex processor 1305 and / or fragment processor(s) 1315A-1315N), which may reference vertex data or image / texture data stored in memory in addition to vertex data or image / texture data stored in the one or more caches 1325A-1325B. In one embodiment, the one or more MMUs 1320A-1320B may be synchronized with other MMUs within the system so that each processor 1205-1220 may participate in a shared or unified virtual memory system, including with other MMUs within the system. Fig.12 One or more MMUs associated with one or more application processors 1205, image processor 1215, and / or video processor 1220. According to an embodiment, one or more circuit interconnects 1330A-1330B enable graphics processor 1310 to interface with other IP cores within the SoC via an internal bus of the SoC or via a direct connection.

[0169] like Fig. 13B As shown in FIG. 1 , the graphics processor 1340 includes Fig.13AOne or more MMUs 1320A-1320B, caches 1325A-1325B, and circuit interconnects 1330A-1330B of a graphics processor 1310. Graphics processor 1340 includes one or more shader cores 1355A-1355N (e.g., 1355A, 1355B, 1355C, 1355D, 1355E, 1355F, all the way to 1355N-1 and 1355N), which provide a unified shader core architecture in which a single core or type or core can execute all types of programmable shader code, including shader program code for implementing vertex shaders, fragment shaders, and / or compute shaders. The exact number of shader cores present may vary by embodiment and implementation. In addition, the graphics processor 1340 includes an inter-core task manager 1345 that acts as a thread dispatcher for dispatching execution threads to one or more shader cores 1355A-1355N and a tiling unit 1358 for accelerating tiling operations for tile-based rendering, in which rendering operations for a scene are subdivided in image space, for example to exploit local spatial consistency within a scene or to optimize the use of internal caches.

[0170] Figure 14A-14B Additional exemplary graphics processor logic is illustrated in accordance with embodiments described herein. Fig.14A Graphics core 1400 is shown, which may be included in Fig.12 within the graphics processor 1210 and may be such as Fig. 13B Unified shader cores 1355A-1355N in. Fig. 14B An additional highly parallel general purpose graphics processing unit 1430 is illustrated, which is a highly parallel general purpose graphics processing unit suitable for deployment on a multi-chip module.

[0171] As in Fig.14A, the graphics core 1400 includes a shared instruction cache 1402, a texture unit 1418, and a cache memory / shared memory 1420 that are common to execution resources within the graphics core 1400. The graphics core 1400 may include multiple slices 1401A-1401N or partitions for each core, and the graphics processor may include multiple instances of the graphics core 1400. The slices 1401A-1401N may include support logic including local instruction caches 1404A-1404N, thread schedulers 1406A-1406N, thread dispatchers 1408A-1408N, and sets of registers 1410A-1440N. To perform logical operations, slices 1401A-1401N may include a set of additional functional units (AFU 1412A-1412N), floating point units (FPU 1414A-1414N), integer arithmetic logic units (ALU 1416-1416N), address calculation units (ACU 1413A-1413N), double precision floating point units (DPFPU 1415A-1415N), and matrix processing units (MPU1417A-1417N).

[0172] Some of these computing units operate with specific precision. For example, FPU 1414A-1414N can perform single-precision (32-bit) and half-precision (16-bit) floating-point operations, while DPFPU 1415A-1415N performs double-precision (64-bit) floating-point operations. ALU 1416A-1416N can perform variable-precision integer operations with 8-bit precision, 16-bit precision, and 32-bit precision, and can be configured for mixed-precision operations. MPU 1417A-1417N can also be configured for mixed-precision matrix operations, including half-precision floating-point operations and 8-bit integer operations. MPU 1417A-1417N can perform a variety of matrix operations to accelerate machine learning application frameworks, including enabling support for accelerated general matrix-to-matrix multiplication (GEMM). AFU 1412A-1412N can perform additional logical operations that are not supported by floating-point units or integer units, including trigonometric operations (e.g., sine, cosine, etc.).

[0173] like Fig. 14BAs shown in , a general purpose processing unit (GPGPU) 1430 can be configured to enable highly parallel computing operations to be performed by an array of graphics processing units. In addition, GPGPU 1430 can be directly linked to other instances of GPGPU to create a multi-GPU cluster, thereby improving the training speed of deep neural networks in particular. GPGPU 1430 includes a host interface 1432 for enabling connection to a host processor. In one embodiment, host interface 1432 is a PCI Express interface. However, the host interface can also be a communication interface or communication structure dedicated to the supplier. GPGPU 1430 receives commands from the host processor and distributes the execution threads associated with those commands to a collection 1436A-1436H of a computing cluster using a global scheduler 1434. Computing clusters 1436A-1436H share cache memory 1438. Cache memory 1438 can act as a higher level cache for cache memory within computing clusters 1436A-1436H.

[0174] GPGPU 1430 includes memory 1434A-1434B coupled to compute clusters 1436A-1436H via a set of memory controllers 1442A-1442B. In various embodiments, memory 1434A-1434B may include various types of memory devices, including dynamic random access memory (DRAM) or graphics random access memory, such as synchronous graphics random access memory (SGRAM), including graphics double data rate (GDDR) memory.

[0175] In one embodiment, computing clusters 1436A-1436H each include a collection of graphics cores, such as Fig.14A The graphics core 1400 may include multiple types of integer logic units and floating point logic units that can perform computational operations including those suitable for machine learning computations within a range of precision. For example and in one embodiment, at least a subset of the floating point units in each of the computing clusters 1436A-1436H may be configured to perform 16-bit or 32-bit floating point operations, while a different subset of the floating point units may be configured to perform 64-bit floating point operations.

[0176] Multiple instances of GPGPU 1430 may be configured to operate as a computing cluster. The communication mechanisms used by the computing cluster for synchronization and data exchange vary across embodiments. In one embodiment, multiple instances of GPGPU 1430 communicate via a host interface 1432. In one embodiment, GPGPU 1430 includes an I / O hub 1439 that couples GPGPU 1430 with a GPU link 1440 that enables direct connection to other instances of GPGPU. In one embodiment, GPU link 1440 is coupled to a dedicated GPU-GPU bridge that enables communication and synchronization between multiple instances of GPGPU 1430. In one embodiment, GPU link 1440 is coupled to a high-speed interconnect to transmit and receive data to other GPGPUs or parallel processors. In one embodiment, multiple instances of GPGPU 1430 are located in separate data processing systems and communicate via a network device that is accessible via a host interface 1432. In one embodiment, in addition to or in lieu of host interface 1432 , GPU link 1440 may be configured to enable connection to a host processor.

[0177] Although the illustrated configuration of GPGPU 1430 can be configured to train a neural network, one embodiment provides an alternative configuration of GPGPU 1430 that can be configured for deployment within a high-performance or low-power inference platform. In the inference configuration, relative to the training configuration, GPGPU 1430 includes fewer computing clusters in computing clusters 1436A-1436H. In addition, the memory technology associated with memory 1434A-1434B can be different between the inference configuration and the training configuration, wherein the higher bandwidth memory technology is dedicated to the training configuration. In one embodiment, the inference configuration of GPGPU 1430 can support inference-specific instructions. For example, the inference configuration can provide support for one or more 8-bit integer dot product instructions, which are typically used during inference operations for deployed neural networks.

[0178] Ray tracing using machine learning

[0179] As mentioned above, ray tracing is a graphics processing technique in which light transport is simulated through physically based rendering. One of the key operations in ray tracing is processing visibility queries, which require traversal and intersection testing of nodes in a bounding volume hierarchy (BVH).

[0180] Ray tracing and path tracing based techniques compute images by tracing rays and paths through each pixel and using random sampling to compute advanced effects such as shadows, glossiness, indirect lighting, etc. Using only a few samples is fast but produces noisy images, while using many samples produces high quality images but is prohibitively expensive.

[0181] In recent years, a breakthrough solution to ray / path tracing for real-time use has emerged in the form of "denoising" - the process of using image processing techniques to produce a high-quality, filtered / denoised image from a noisy, low-sample-count input. The most effective denoising techniques rely on machine learning techniques, where a machine learning engine learns what a noisy image would likely look like if it had been calculated with more samples. In one particular implementation, the machine learning is performed by a convolutional neural network (CNN); however, the underlying principles of the invention are not limited to CNN implementations. In such implementations, training data is generated with a low-sample-count input and ground truth. The CNN is trained to predict converged pixels based on a neighborhood of noisy pixel inputs surrounding the pixel under consideration.

[0182] Although not perfect, this AI-based denoising technique has proven to be surprisingly effective. However, the caveat is that good training data is needed, as otherwise the network may predict incorrect results. For example, if an animation film studio trains a denoising CNN on past films using scenes on land and then attempts to use the trained CNN to denoise frames from a new film set on water, the denoising operation will perform suboptimally.

[0183] To address this problem, one embodiment of the present invention dynamically collects learning data while rendering, and continuously trains a machine learning engine (such as a CNN) based on the data on which it is running, thereby continuously improving the machine learning engine for upcoming tasks. This embodiment may still perform the training phase before runtime, but continue to adjust the machine learning weights as needed during runtime. In addition, this embodiment avoids the high cost of calculating the reference data required for training by limiting the generation of learning data to sub-regions of each frame or every N frames of the image. Specifically, a noisy input of a frame is generated to denoise the complete frame using the current network. In addition, as described below, small areas of reference pixels are generated and used for continuous training.

[0184] Although CNN implementations are described with reference to certain embodiments, any form of machine learning engine may be used, including but not limited to systems that perform supervised learning (e.g., building a mathematical model of a data set that includes both inputs and desired outputs), unsupervised learning (e.g., which evaluates input data against certain types of structures), and / or a combination of supervised and unsupervised learning.

[0185] Existing denoising implementations operate in a training phase and a runtime phase. During the training phase, a network topology is defined that receives an area of ​​NxN pixels with various pixel-by-pixel data channels and generates the final pixel color, such as pixel color, depth, normal, normal deviation, primitive ID, and reflectivity. A set of "representative" training data is generated by using the amount of information of a low sample number input of a frame and referring to the "expected" pixel color calculated with a very high sample number. The network is trained on these inputs to generate a set of "ideal" weights for the network. In these implementations, the weights of the network are trained using reference data to most closely match the output of the network with the desired result.

[0186] At runtime, load the given, pre-computed ideal network weights and initialize the network. For each frame, generate a low-sample-count image of the denoised input (i.e., the same as used for training). For each frame, run a given neighborhood of pixel input through the network to predict the "denoised" pixel color, generating a denoised frame.

[0187] Fig.15 One embodiment of an initial training implementation is illustrated. A machine learning engine 1500 (e.g., CNN) receives a region of N x N pixels as high sample count image data 1502 with various pixel-by-pixel data channels such as pixel color, depth, normal, normal deviation, primitive ID, and reflectivity and generates final pixel colors. Representative training data is generated using the information content of a frame of low sample count input 1501. The network is trained on these inputs, thereby generating a set of "ideal" weights 1505, which the machine learning engine 1500 subsequently uses to denoise the low sample count image at runtime.

[0188] To improve the above technique, one embodiment of the present invention expands the denoising stage to generate new training data for each frame or to generate new training data for a subset of frames (e.g., for every N frames, where N=2, 3, 4, 10, 25, etc.). Specifically, as in Fig.16 As shown in FIG. 1 , the embodiment selects one or more regions in each frame (referred to herein as “new reference regions” 1602), which are rendered with a high sample count into a separate high sample count buffer 1604. A low sample count buffer 1603 stores low sample count input frames 1601 (including low sample regions 1604 corresponding to new reference regions 1602).

[0189] In one embodiment, the position of the new reference region 1602 is chosen arbitrarily. Alternatively, the position of the new reference region 1602 can be adjusted for each new frame in a pre-specified manner (e.g., using a predetermined movement of regions between frames, limited to a specified region at the center of a frame, etc.).

[0190] Regardless of how the new reference region is chosen, the new reference region is used by the machine learning engine 1600 to continually refine and update the trained weights 1605 for noise reduction. Specifically, the reference pixel colors from each new reference region 1602 are rendered along with the noisy reference pixel inputs from the corresponding low sample count region 1607. Supplemental training is then performed on the machine learning engine 1600 using the high sample count reference region 1602 and the corresponding low sample count region 1607. In contrast to the initial training, this training is performed for each new reference region 1602 during runtime - thereby ensuring that the machine learning engine 1600 is accurately trained. For example, pixel-by-pixel channels (e.g., pixel color, depth, normal, normal deviation, etc.) may be evaluated, and the machine learning engine 1600 uses these pixel-by-pixel data channels to make adjustments to the trained weights 1605. As in the training case ( Fig.15 ), the machine learning engine 1600 is trained on a set of ideal weights 1605 to remove noise from a low sample number input frame 1601, thereby generating a denoised frame 1620. However, in this embodiment, the trained weights 1605 are continuously updated based on new image characteristics of new types of low sample number input frames 1601.

[0191] In one embodiment, the retraining operation performed by the machine learning engine 1600 is performed concurrently in a background process on a graphics processor unit (GPU) or a host processor. A rendering loop, which may be implemented as a driver component and / or a GPU hardware component, continuously generates new training data (e.g., in the form of new reference regions 1602), which is placed in a queue. A background training process executed on the GPU or host processor continuously reads the new training data, retrains the machine learning engine 1600, and updates the machine learning engine 1600 with new weights 1605 at appropriate intervals.

[0192] Fig.17 An example of one such implementation is shown in which the background training process 1700 is implemented by the host CPU 1710. Specifically, in this embodiment, the background training process 1700 uses the high sample number new reference area 1602 and the corresponding low sample area 1604 to continuously update the trained weights 1605, thereby updating the machine learning engine 1600.

[0193] As in Fig.18AAs illustrated in , in one implementation, such as in a multiplayer online game, different host machines 1820-1822 each generate reference areas, and the background training processes 1700A-1700C transmit these reference areas to the server 1800 (such as, a game server). The server 1800 then performs training on the machine learning engine 1810 using the new reference areas received from each of the hosts 1820-1822, thereby updating the weights 1805 as previously described. The server 1800 transmits these weights 1805 to the host machines 1820-1822 storing the weights 1605A-1605C, thereby updating each individual machine learning engine (not shown). Because the server 1800 can be provided with a large number of reference areas in a short period of time, the server 1800 can efficiently and accurately update the weights for any given application (e.g., an online game) being executed by a user.

[0194] As in Fig.18B , different host machines may generate new trained weights (e.g., based on training / reference area 1602 as previously described) and share the new trained weights with server 1800 (e.g., such as a game server) or alternatively use a peer-to-peer sharing protocol. A machine learning management component 1810 on the server generates a set 1805 of combined weights using the new weights received from each of the host machines. The combined weights 1805 may, for example, be an average value generated from the new weights and continuously updated as described herein. Once generated, copies 1605A-1605C of the combined weights may be transmitted and stored on each of the host machines 1820-1822, which may then use the combined weights as described herein to perform noise reduction operations.

[0195] In one embodiment, this semi-closed loop update mechanism can be used by hardware manufacturers. For example, a reference network can be included as part of a driver distributed by a hardware manufacturer. As the driver generates new training data using the techniques described herein and continuously submits this data back to the hardware manufacturer, the hardware manufacturer uses this information to continue to improve its machine learning implementation for the next driver update.

[0196] In one implementation (e.g., in batch movie rendering on a render farm), the renderer transmits newly generated training regions to a dedicated server or database (in that studio's render farm) that aggregates this data from multiple render nodes over time. A separate process on a separate machine continuously improves the studio's dedicated denoising network, and new render jobs always use the latest trained network.

[0197] exist Fig.19A method according to one embodiment of the present invention is illustrated in FIG. The method may be implemented on the architecture described herein, but is not limited to any particular system or graphics processing architecture.

[0198] At 1901, as part of an initial training phase, low sample count image data and high sample count image data are generated for a plurality of image frames. At 1902, a machine learning denoising engine is trained using the high / low sample count image data. In one embodiment, for example, a set of convolutional neural network weights associated with pixel features may be updated based on the training. However, any machine learning architecture may be used.

[0199] At 1903, at runtime, a low sample count image frame and at least one reference region with a high sample count are generated. At 1904, the high sample count reference region is used by the machine learning engine and / or a separate training logic (e.g., background training module 1700) to continuously improve the training of the machine learning engine. For example, in one embodiment, the high sample count reference region is used in conjunction with a corresponding portion of the low sample count image to continue teaching the machine learning engine 1904 how to most effectively perform noise reduction. In a CNN implementation, for example, this may involve updating weights associated with the CNN.

[0200] Multiple variations of the embodiments described above may be implemented, such as the manner in which a feedback loop to a machine learning engine is configured; the entity that generates training data; the manner in which training data is fed back to the training engine; and how the improved network is provided to a rendering engine. Furthermore, while the above embodiments described above perform continuous training using a single reference region, any number of reference regions may be used. Furthermore, as previously mentioned, the reference regions may have different sizes, may be used for different numbers of image frames, and may be positioned in different locations within an image frame using different techniques (e.g., arbitrarily, according to a predetermined pattern, etc.).

[0201] Furthermore, while a convolutional neural network (CNN) is described as one example of a machine learning engine 1600, any form of machine learning engine that can continually improve its results using new training data may be used to implement the underlying principles of the invention. By way of example and not limitation, other machine learning implementations include grouping methods for data processing (GMDH), long short-term memory, deep storage computing, deep belief networks, tensor deep stacked networks, and deep predictive decoding networks, to name a few.

[0202] Apparatus and method for efficient distributed noise reduction

[0203] As mentioned above, denoising has become a critical feature for real-time ray tracing with smooth, noise-free images. Rendering can be done on multiple devices across a distributed system, but to date, existing denoising frameworks all operate on a single instance on a single machine. If rendering is being done across multiple devices, those devices may not make all rendered pixels accessible for computing the denoised portion of the image.

[0204] One embodiment of the present invention includes a distributed denoising algorithm that operates using both artificial intelligence (AI) based denoising techniques and non-AI based denoising techniques. Regions of an image are either already distributed across nodes from a distributed rendering operation or are split and distributed from a single frame buffer. Pseudo regions of neighboring regions required to compute sufficient denoising are collected from neighboring nodes when needed, and the resulting tiles are composited into the final image.

[0205] Distributed processing

[0206] Fig. 20 An embodiment of the present invention is illustrated in which multiple nodes 2021-2023 perform rendering. Although only three nodes are illustrated for simplicity, the underlying principles of the present invention are not limited to any particular number of nodes. In fact, a single node may be used to implement certain embodiments of the present invention.

[0207] Nodes 2021-2023 each render a portion of the image, resulting in regions 2011-2013 in this example. Fig. 20 Rectangular regions 2011-2013 are shown in FIG, but regions of any shape may be used, and any device may handle any number of regions. The regions required by the node to perform a sufficiently smooth noise reduction operation are referred to as pseudo regions 2001-2003. In other words, pseudo regions 2001-2003 represent all the data required to perform noise reduction at the specified quality level. Lowering the quality level reduces the size of the pseudo regions and thus reduces the amount of data required, and increasing the quality level increases the pseudo regions and the corresponding data required.

[0208] In one embodiment, if a node such as 2021 does have a local copy of the portion of the fake region 2001 needed to denoise the region 2011 of the node at a specified quality level, the node will retrieve the required data from one or more "neighboring" nodes, such as the illustrated node 2022 that has a portion of the fake region 2001. Similarly, if node 2022 does have a local copy of the portion of the fake region 2002 needed to denoise the region 2012 of the node 2022 at a specified quality level, node 2022 will retrieve the required fake region data 2032 from node 221. The retrieval can be performed over a bus, interconnect, high-speed memory structure, network (e.g., high-speed Ethernet), or even an on-chip interconnect in a multi-core chip capable of distributing rendering work among multiple cores (e.g., for rendering large images at extreme resolutions or in a time-varying manner). In one embodiment, each node 2021-2023 includes a respective execution unit or a specified set of execution units within a graphics processor.

[0209] The specific amount of data to be sent depends on the noise reduction technique used. In addition, the data from the false regions may include any data required to improve the noise reduction of each corresponding region. In one embodiment, for example, the false region data includes image color, wavelength, intensity / alpha data, and / or normals. However, the underlying principles of the present invention are not limited to any specific false region data set.

[0210] Additional details of one embodiment

[0211] For slower networks or interconnects, compression of the data can be utilized by using existing general-purpose lossless or lossy compression. Examples include, but are not limited to, zlib, gzip, and the Lempel–Ziv–Markov chain algorithm (LZMA). Further content-specific compression can be used by noting that the deltas of ray hit information between frames may be very sparse, and when a node already has the collected deltas from a previous frame, only the samples contributing to that delta need be sent. These can be selectively pushed to the node i that collected those samples, or node i can request samples from other nodes. In one embodiment, lossless compression is used for certain types of data and program code, while lossy compression is used for other types of data.

[0212] Fig.21The diagram illustrates additional details of the interaction between nodes 2021 and 2022 according to one embodiment of the present invention. Each node 2021-2022 includes a ray tracing rendering circuit 2081-2082, which is used to render the corresponding image region 2011-2012 and the false region 2001-2002. The denoisers 2100-2111 perform denoising operations on the regions 2011-2012, respectively, and each node 2021-2022 is responsible for rendering and denoising the regions 2011-2012, respectively. The denoisers 2021-2022 may, for example, include circuits, software, or any combination of circuits and software for generating denoised regions 2121-2122, respectively. As mentioned, when generating denoised regions, denoisers 2021-2022 may need to rely on data within false regions owned by different nodes (eg, denoiser 2100 may need data from false region 2002 owned by node 2022).

[0213] Thus, in one embodiment, the denoisers 2100-2111 generate denoised regions 2121-2122 using data from regions 2011-2012 and dummy regions 2001-2002, at least a portion of which may be received from another node. As described herein, the region data managers 2101-2102 manage the transfer of data from dummy regions 2001-2002. In one embodiment, the compressor / decompressor units 2131-2132 perform compression and decompression of dummy region data exchanged between nodes 2021-2022, respectively.

[0214] For example, the region data manager 2101 of the node 2021 may send data from the false region 2001 to the compressor / decompressor 2131 at the request from the node 2022, which compresses the data to generate compressed data 2106, which the compressor / decompressor 2131 transmits to the node 2022, thereby reducing the bandwidth through the interconnect, network, bus or other data communication link. The compressor / decompressor 2132 of the node 2022 then decompresses the compressed data 2106, and the denoiser 2111 uses the decompressed false data to generate a higher quality denoised region 2012 than would be possible using only data from the region 2012. The region data manager 2102 may store the decompressed data from the false region 2001 in a cache, memory, register file or other storage so that the data is available to the denoiser 2111 when generating the denoised region 2121. A similar set of operations may be performed to provide data from false region 2002 to denoiser 2100 on node 2021 , which uses that data in conjunction with data from region 2011 to generate a higher quality denoised region 2121 .

[0215] Fetch data or render

[0216] If the connection between devices such as nodes 2021-2022 is slow (i.e., slower than a threshold latency and / or a threshold bandwidth), it may be faster to render the fake region locally rather than requesting results from other devices. This can be determined at runtime by tracking network transaction speeds and linearly extrapolated render times for the fake region size. In such cases where the entire fake region is rendered, multiple devices may end up rendering multiple identical portions of the image. The resolution of the rendered portion of the fake region may be adjusted based on the variance of the base region and the determined blurriness.

[0217] Load Balancing

[0218] In one embodiment, static and / or dynamic load balancing schemes may be used to distribute the processing load among the nodes 2021-2022. For dynamic load balancing, the variance determined by the denoising filter may both require more time in denoising and drive the amount of samples used to render specific areas of the scene, where low variance and blurred areas of the image require fewer samples. The specific areas assigned to specific nodes may be dynamically adjusted based on data from previous frames, or when multiple devices are rendering, the specific areas assigned to specific nodes may be dynamically transferred across these devices so that all devices will have the same workload.

[0219] Fig. 22An embodiment in which monitors 2201-2202 running on each respective node 221-2022 collect performance metric data is illustrated, including but not limited to the time consumed by transmitting data through network interfaces 2211-2212, the time consumed when denoising the region (with or without false region data), and the time consumed by rendering each region / false region. Monitors 2201-2202 report these performance metrics back to a manager or load balancer node 2201, which analyzes the data to identify the current workload on each node 2221-2022 and potentially determines a more efficient mode for processing various denoised regions 2121-2122. Manager node 2201 then distributes the new workload for the new region to nodes 221-2022 according to the detected load. For example, manager node 2201 may transfer more work to those nodes that are not heavily loaded, and / or reassign work from those nodes that are overloaded. In addition, the load balancer node 2201 may transmit reconfiguration commands to adjust the specific manner in which rendering and / or denoising is performed by each of the nodes (some examples of which are described above).

[0220] Identify false areas

[0221] In one embodiment, the size and shape of the spurious regions 2001-2002 are determined based on a denoising algorithm implemented by the denoisers 2100-2111. The respective sizes of these spurious regions may then be dynamically modified based on the detected variance of the samples being denoised. The learning algorithm used for the AI ​​denoising itself may be used to determine the appropriate region size, or in other cases such as bilateral blur, a predetermined filter width will determine the size of the spurious regions 2001-2002. In implementations using a learning algorithm, a machine learning engine may be executed on the manager node 2201, and / or multiple portions of the machine learning may be executed on each of the individual nodes 221-2022 (see, e.g. Figure 18A-18B and the text associated with the above).

[0222] Aggregate the final image

[0223] In one embodiment, the final image may be generated by aggregating the rendered and denoised regions from each of the nodes 2021-2022 without the need for spurious regions or normals. Fig. 22, for example, the denoised regions 2121-2122 are transmitted to the region processor 2280 of the manager node 2201, which combines the regions to generate a final denoised image 2290, which is then displayed on the display 2290. The region processor 2280 can use various 2D synthesis techniques to combine the regions. Although illustrated as multiple separate components, the region processor 2280 and the denoised image 2290 can be integral to the display 2290. In this embodiment, each node 2021-2022 can use a direct send technique to transmit the denoised regions 2121-2122 and potentially use various lossy or lossless compressions of the region data.

[0224] As games move into the cloud, AI denoising remains an expensive operation. As a result, distributed processing of denoising across multiple nodes 2021-2022 may become necessary to achieve real-time frame rates for traditional games or virtual reality (VR) that requires higher frame rates. Movie studios also often render in large render farms, which can be used to perform faster denoising.

[0225] exist Fig.23 One embodiment of a method for performing distributed rendering and denoising is illustrated in The method may be implemented within the context of the system architecture described above, but is not limited to any particular system architecture.

[0226] At 2301, graphics work is dispatched to a plurality of nodes that perform ray tracing operations to render regions of an image frame. In one embodiment, each node may already have the data needed to perform the operation in memory. For example, two or more of the nodes may share a common memory, or the node's local memory may already store data from a previous ray tracing operation. Alternatively or additionally, certain data may be transferred to each node.

[0227] At 2302, a "pseudo region" required for a specified level of noise reduction (ie, at an acceptable level of performance) is determined. The pseudo region includes any data required to perform the specified level of noise reduction, including data possessed by one or more other nodes.

[0228] At 2303, data related to the false data (or portions thereof) is exchanged between nodes. At 2304, each node performs noise reduction on its corresponding region (eg, using the exchanged data), and at 2305, the results are combined to generate a final noise-reduced image frame.

[0229] In one embodiment, such as Fig. 22The manager node or master node shown in assigns work to the nodes and then combines the work performed by the nodes to generate the final image frame. In another embodiment, a peer-based architecture is used, where the nodes are peers that exchange data to render and denoise the final image frame.

[0230] The nodes described herein (e.g., nodes 2021-2022) can be graphics processing computing systems interconnected via a high-speed network. Alternatively, the nodes can be individual processing elements coupled to a high-speed memory structure. In this embodiment, all nodes can share a common virtual memory space and / or a common physical memory. In another embodiment, the node can be a combination of a CPU and a GPU. For example, the manager node 2201 described above can be a CPU and / or software executed on a CPU, and nodes 2021-2022 can be a GPU and / or software executed on a GPU. Various different types of nodes can be used while still complying with the underlying principles of the present invention.

[0231] Example Neural Network Implementation

[0232] There are many types of neural networks; one simple type of neural network is a feedforward network. A feedforward network can be implemented as an acyclic graph in which nodes are arranged in layers. Typically, a feedforward network topology includes an input layer and an output layer separated by at least one hidden layer. The hidden layer transforms the input received by the input layer into a representation useful for generating output in the output layer. The network nodes are fully connected to the nodes in the adjacent layers via edges, but there are no edges between the nodes within each layer. Data received at the nodes of the input layer of the feedforward network is propagated (i.e., "fed forward") to the nodes of the output layer via an activation function, which calculates the state of the nodes of each consecutive layer in the network based on coefficients ("weights") associated with each of the edges connecting these layers. The output from the neural network algorithm can take a variety of forms, depending on the specific model being represented by the algorithm being executed.

[0233] Before a machine learning algorithm can be used to model a particular problem, the algorithm is trained using a training data set. Training a neural network involves: selecting a network topology; using a set of training data that represents the problem being modeled by the network; and adjusting weights until the network model performs with minimal error for all instances of the training data set. For example, during a supervised learning training process for a neural network, the output generated by the network in response to an input representing an instance in the training data set is compared to the "correct" labeled output for that instance, an error signal representing the difference between the output and the labeled output is calculated, and as the error signal is propagated backward through the layers of the network, the weights associated with the connections are adjusted to minimize that error. The network is considered "trained" when the error for each of the outputs generated from the instances of the training data set is minimized.

[0234] The accuracy of a machine learning algorithm can be significantly affected by the quality of the data set used to train the algorithm. The training process can be computationally intensive and can take a significant amount of time on a conventional general-purpose processor. Therefore, many types of machine learning algorithms are trained using parallel processing hardware. This is particularly useful for optimizing the training of neural networks, as the calculations performed when adjusting the coefficients in a neural network are naturally suited to parallel implementations. In particular, many machine learning algorithms and software applications have been adapted to exploit the parallel processing hardware within general-purpose graphics processing devices.

[0235] Fig.24 is a generalized diagram of a machine learning software stack 2400. Machine learning applications 2402 may be configured to train a neural network using a training data set or to implement machine intelligence using a trained deep neural network. Machine learning applications 2402 may include training and inference functions for a neural network and / or specialized software that may be used to train a neural network prior to deployment. Machine learning applications 2402 may implement any type of machine intelligence, including but not limited to: image recognition, mapping and localization, autonomous navigation, speech synthesis, medical imaging, or language translation.

[0236] Hardware acceleration for machine learning applications 2402 can be enabled via a machine learning framework 2404. The machine learning framework 2404 can provide a library of machine learning primitives. Machine learning primitives are basic operations often performed by machine learning algorithms. Without the machine learning framework 2404, developers of machine learning algorithms would be required to create and optimize the main computational logic associated with the machine learning algorithm, and then re-optimize the computational logic when developing new parallel processors. Instead, machine learning applications can be configured to use primitives provided by the machine learning framework 2404 to perform the necessary calculations. Exemplary primitives include tensor convolutions, activation functions, and pooling, which are computational operations performed when training convolutional neural networks (CNNs). The machine learning framework 2404 can also provide primitives to implement basic linear algebra subroutines performed by many machine learning algorithms, such as matrix and vector operations.

[0237] The machine learning framework 2404 can process input data received from the machine learning application 2402 and generate appropriate inputs to the computation framework 2406. The computation framework 2406 can abstract the underlying instructions provided to the GPGPU driver 2408 to enable the machine learning framework 2404 to utilize hardware acceleration via the GPGPU hardware 2410 without requiring the machine learning framework 2404 to be intimately familiar with the architecture of the GPGPU hardware 2410. In addition, the computation framework 2406 can enable hardware acceleration for the machine learning framework 2404 across various types and generations of GPGPU hardware 2410.

[0238] GPGPU machine learning acceleration

[0239] Fig.25A multi-GPU computing system 2500 according to an embodiment is illustrated. The multi-GPU computing system 2500 may include a processor 2502 coupled to a plurality of GPGPUs 2506A-2506D via a host interface switch 2504. In one embodiment, the host interface switch 2504 is a PCI Express switch device that couples the processor 2502 to a PCI Express bus, through which the processor 2502 can communicate with a collection 2506A-2506D of GPGPUs. Each of the plurality of GPGPUs 2506A-2506D may be an instance of a GPGPU described above. The GPGPUs 2506A-2506D may be interconnected via a collection 2516 of high-speed point-to-point GPU-GPU links. The high-speed GPU-GPU links may be connected to each of the GPGPUs 2506A-2506D via dedicated GPU links. P2P GPU link 2516 enables direct communication between each of GPGPUs 2506A-2506D without requiring communication over a host interface bus to which processor 2502 is connected. With GPU-GPU traffic directed to the P2P GPU link, the host interface bus remains available for system memory access, or for communicating with other instances of multi-GPU computing system 2500, for example, via one or more network devices. While in the illustrated embodiment, GPGPUs 2506A-2506D are connected to processor 2502 via host interface switch 2504, in one embodiment, processor 2502 includes direct support for P2P GPU link 2516 and can connect directly to GPGPUs 2506A-2506D.

[0240] Machine Learning Neural Network Implementation

[0241] The computing architecture provided by the embodiments described herein can be configured to perform these types of parallel processing that are particularly suitable for training and deploying neural networks for machine learning. A neural network can be generalized as a network of functions with graph relationships. As is known in the art, there are many types of neural network implementations used in machine learning. One exemplary type of neural network is a feedforward network as previously described.

[0242] The second exemplary type of neural network is a convolutional neural network (CNN). CNN is a specialized feedforward neural network for processing data (such as image data) with a known, grid-like topology. Therefore, CNNs are commonly used in computer vision and image recognition applications, but they can also be used for other types of pattern recognition, such as speech and language processing. The nodes in the CNN input layer are organized as a set of "filters" (feature detectors stimulated by the receptive fields found in the retina), and the output of each filter set is propagated to the nodes in the successive layers of the network. The calculation for CNN includes applying the convolution mathematical operation to each filter to produce the output of that filter. Convolution is a specialized mathematical operation performed by two functions to produce a third function, which is a modified version of one of the two original functions. In convolutional network terminology, the first function of the convolution can be called the input, and the second function can be called the convolution kernel. The output can be called a feature map. For example, the input to the convolution layer can be a multidimensional data array defining the various color components of the input image. The convolution kernel can be a multidimensional parameter array, where the parameters are adapted by the training process for the neural network.

[0243] Recurrent Neural Networks (RNNs) are a class of feedforward neural networks that include feedback connections between layers. RNNs enable modeling of sequential data by sharing parameter data across different parts of the neural network. The architecture for RNNs includes loops. These loops represent the effect of the current value of a variable on its own value at future moments, as at least a portion of the output data from the RNN is used as feedback for processing subsequent inputs in the sequence. This feature makes RNNs particularly useful for language processing due to the mutable nature of language data that can be composed.

[0244] The figures described below present exemplary feedforward networks, CNN networks, and RNN networks, and describe the general process for training and deploying each of those types of networks, respectively. It will be understood that these descriptions are exemplary and non-limiting with respect to any specific embodiments described herein, and that the concepts illustrated are generally applicable to deep neural networks and machine learning techniques.

[0245] The exemplary neural network described above can be used to perform deep learning. Deep learning is machine learning using deep neural networks. In contrast to shallow neural networks that include only a single hidden layer, the deep neural networks used in deep learning are artificial neural networks composed of multiple hidden layers. Deeper neural networks are generally more computationally intensive to train. However, the additional hidden layers of the network enable multi-step pattern recognition that produces reduced output errors relative to shallow machine learning techniques.

[0246] The deep neural network used in deep learning typically includes a front-end network for performing feature recognition, which is coupled to a back-end network that represents a mathematical model that can perform operations (e.g., object classification, speech recognition, etc.) based on the feature representation provided to the model. Deep learning enables machine learning to be performed without the need to perform manual feature engineering for the model. Instead, a deep neural network can learn features based on statistical structures or correlations within the input data. The learned features can be provided to a mathematical model that can map the detected features to an output. The mathematical model used by the network is typically dedicated to a specific task to be performed, and different models will be used to perform different tasks.

[0247] Once the neural network is structured, a learning model can be applied to the network to train the network to perform a specific task. The learning model describes how to adjust the weights within the model to reduce the output error of the network. Back propagation of errors is a common method for training neural networks. An input vector is presented to the network for processing. The output of the network is compared to the desired output using a loss function, and an error value is calculated for each of the neurons in the output layer. The error values ​​are then propagated backward until each neuron has an associated error value that roughly represents the contribution of the neuron to the original output. The network can then learn through those errors using an algorithm (such as a stochastic gradient descent algorithm) to update the weights of the neural network.

[0248] Figure 26-Figure 27 Illustration of an example convolutional neural network. Fig.26 The diagram shows the various layers within a CNN. Fig.26 As shown in , an exemplary CNN for modeling image processing may receive an input 2602 that describes the red, green, and blue (RGB) components of an input image. The input 2602 may be processed by a plurality of convolutional layers (e.g., convolutional layer 2604, convolutional layer 2606). The outputs from the plurality of convolutional layers may optionally be processed by a collection 2608 of fully connected layers. As previously described for feedforward networks, neurons in a fully connected layer have full connections to all activations in the previous layer. The outputs from the fully connected layer 2608 may be used to generate output results from the network. Activations within the fully connected layer 2608 may be calculated using matrix multiplication instead of convolution. Not all CNN implementations utilize fully connected layers. For example, in some implementations, the convolutional layer 2606 may generate the output of the CNN.

[0249] The convolutional layers are sparsely connected, which is different from the traditional neural network configuration found in the fully connected layer 2608. Traditional neural network layers are fully connected so that each output unit interacts with each input unit. However, as shown, the convolutional layers are sparsely connected because the output of the convolution of the receptive field (rather than the corresponding state value of each node in the receptive field) is input to the nodes of the subsequent layer. The kernel associated with the convolutional layer performs a convolution operation, the output of which is sent to the next layer. The dimensionality reduction performed within the convolutional layer is one aspect that enables CNNs to scale to process large images.

[0250] Fig. 27 An exemplary computational stage within a convolutional layer of a CNN is shown. Input 2712 to a convolutional layer of a CNN may be processed in three stages of a convolutional layer 2714. The three stages may include a convolutional stage 2716, a detector stage 2718, and a pooling stage 2720. Subsequently, the convolutional layer 2714 may output data to a successive convolutional layer. The final convolutional layer of the network may generate output feature map data or provide input to a fully connected layer, for example to generate a classification value for input to the CNN.

[0251] Several convolutions are performed in parallel in the convolution stage 2716 to produce a set of linear activations. The convolution stage 2716 may include an affine transformation, which is any transformation that can be specified as a linear transformation plus a translation. Affine transformations include rotations, translations, scaling, and combinations of these transformations. The convolution stage calculates the output of a function (e.g., a neuron) connected to a specific area in the input, which can be determined as a local area associated with the neuron. The neuron calculates the dot product between the weight of the neuron and the weight of the area in the local input to which the neuron is connected. The output from the convolution stage 2716 defines a set of linear activations processed by the successive stages of the convolution layer 2714.

[0252] The linear activations may be processed by the detector stage 2718. In the detector stage 2718, each linear activation is processed by a non-linear activation function. The non-linear activation function increases the non-linear nature of the overall network without affecting the receptive field of the convolutional layer. Several types of non-linear activation functions may be used. One particular type is the rectified linear unit (ReLU), which uses an activation function defined as f(x) = max(0, x) such that the threshold of the activation function is zero.

[0253] Pooling stage 2720 uses a pooling function that replaces the output of convolutional layer 2706 with summary statistics of nearby outputs. Pooling functions can be used to introduce translation invariance into neural networks so that small translations to the input do not change the pooled output. Local translation invariance can be useful in scenarios where the presence of a feature in the input data is more important than the exact location of the feature. Various types of pooling functions can be used during pooling stage 2720, including maximum pooling, average pooling, and l2 norm pooling. In addition, some CNN implementations do not include a pooling stage. Instead, such implementations substitute additional convolution stages with an increased stride relative to the previous convolution stage.

[0254] Subsequently, the output from the convolutional layer 2714 may be processed by the next layer 2722. The next layer 2722 may be an additional convolutional layer or a layer in the fully connected layer 2708. For example, Fig. 27 The first convolutional layer 2704 can output to the second convolutional layer 2706, and the second convolutional layer can output to the first layer in the fully connected layer 2808.

[0255] Fig.28 An exemplary recurrent neural network 2800 is illustrated. In a recurrent neural network (RNN), the previous state of the network affects the output of the current state of the network. RNNs can be established in various ways using various functions. The use of RNNs generally revolves around the use of mathematical models to predict the future based on previous input sequences. For example, RNNs can be used to perform statistical language modeling to predict upcoming words given a previous word sequence. The illustrated RNN 2800 can be described as having the following: an input layer 2802, which receives an input vector; a hidden layer 2804, which is used to implement a recursive function; a feedback mechanism 2805, which is used to enable "memory" of previous states; and an output layer 2806, which is used to output a result. RNN 2800 operates based on time steps. The state of the RNN at a given time step is affected based on the previous time step via a feedback mechanism 2805. For a given time step, the state of the hidden layer 2804 is defined by the previous state and the input at the current time step. The initial input (x1) at the first time step can be processed by the hidden layer 2804. The second input (x2) can be processed by the hidden layer 2804 using the state information determined during the processing of the initial input (x1). A given state can be calculated as s_t=f(Ux_t+Ws_(t-1)), where U and W are parameter matrices. The function f is generally nonlinear, such as a variant of the hyperbolic tangent function (Tanh) or a modified function f(x)=max(0,x). However, the specific mathematical function used in the hidden layer 2804 may vary depending on the specific implementation details of the RNN 2800.

[0256] In addition to the basic CNN network and RNN network described, changes in those networks can also be implemented. An example RNN variant is a long short-term memory (LSTM) RNN. LSTM RNN is able to learn long-term dependencies that may be necessary for processing longer language sequences. A variant of CNN is a convolutional deep belief network, which has a structure similar to CNN and is trained in a manner similar to a deep belief network. A deep belief network (DBN) is a generative neural network composed of multiple layers of stochastic (random) variables. Greedy unsupervised learning can be used to train DBN layer by layer. Subsequently, the learned weights of the DBN can be used to provide a pre-trained neural network by determining the best initial weight set for the neural network.

[0257] Fig.29 The training and deployment of a deep neural network is illustrated. Once a given network has been structured for a task, the neural network is trained using a training data set 2902. Various training frameworks 2904 have been developed to enable hardware acceleration of the training process. For example, the machine learning framework described above can be configured as a training framework. The training framework 2904 can be hooked up to an untrained neural network 2906 and enables the untrained neural network to be trained using the parallel processing resources described herein to generate a trained neural network 2908.

[0258] To start the training process, initial weights can be chosen randomly or by pre-training using a deep belief network. Subsequently, a training cycle is performed in a supervised or unsupervised manner.

[0259] Supervised learning is a learning method in which training is performed as an intermediate operation, such as when the training data set 2902 includes an input paired with an expected output of the input, or when the training data set includes an input with a known output and the output of the neural network is manually graded. The network processes the input and compares the resulting output with the expected output or a set of expected outputs. Subsequently, the error is back-propagated through the system. The training framework 2904 can be adjusted to adjust the weights that control the untrained neural network 2906. The training framework 2904 can provide tools to monitor how well the untrained neural network 2906 converges to a model that is suitable for generating the correct answer based on the known input data. The training process occurs repeatedly as the weights of the network are adjusted to improve the output generated by the neural network. The training process can continue until the neural network reaches a statistically expected accuracy associated with the trained neural network 2908. Subsequently, the trained neural network 2908 can be deployed to implement any number of machine learning operations.

[0260] Unsupervised learning is a learning method in which a network attempts to train itself using unlabeled data. Thus, for unsupervised learning, the training data set 2902 will include input data without any associated output data. The untrained neural network 2906 can learn the groupings within the unlabeled inputs and can determine how the individual inputs are related to the entire data set. Unsupervised training can be used to generate a self-organizing map, which is a class of trained neural networks 2907 that can perform operations that help reduce the dimensionality of the data. Unsupervised training can also be used to perform anomaly detection, which allows identification of data points in the input data set that deviate from the normal pattern of the data.

[0261] Variations of supervised and unsupervised training may also be employed. Semi-supervised learning is a technique in which a mixture of labeled and unlabeled data having the same distribution is included in the training data set 2902. Incremental learning is a variation of supervised learning in which input data is used continuously to further train the model. Incremental learning enables a trained neural network 2908 to adapt to new data 2912 without forgetting the knowledge embedded in the network during initial training.

[0262] Whether supervised or unsupervised, the training process for particularly deep neural networks may be too computationally intensive for a single computing node. The training process can be accelerated using a distributed network of computing nodes rather than using a single computing node.

[0263] Fig. 30A 3002, data parallelism 3004, or a combination of model parallelism and data parallelism.

[0264] In model parallelism 3002, different computing nodes in a distributed system can perform training computations for different parts of a single network. For example, each layer of a neural network can be trained by a different processing node of the distributed system. Benefits of model parallelism include the ability to scale to particularly large models. Splitting the computations associated with different layers of a neural network enables training of very large neural networks where the weights of all layers will not fit into the memory of a single computing node. In some instances, model parallelism can be particularly useful in performing unsupervised training of large neural networks.

[0265] In data parallelism 3004, different nodes of the distributed network have complete instances of the model, and each node receives a different portion of the data. Subsequently, the results from different nodes are combined. Although different methods for data parallelism are possible, data parallel training methods all require techniques for combining results and synchronizing model parameters between each node. Exemplary methods for combining data include parameter averaging and update-based data parallelism. Parameter averaging trains each node on a subset of the training data, and sets global parameters (e.g., weights, biases) to the average value of the parameters from each node. Parameter averaging uses a central parameter server that maintains parameter data. Update-based data parallelism is similar to parameter averaging, with the exception that updates to the model are transmitted instead of parameters from the node being transmitted to the parameter server. In addition, update-based data parallelism can be performed in a decentralized manner, where updates are compressed and transmitted between nodes.

[0266] Combined model and data parallelism 3006 can be implemented, for example, in a distributed system where each computing node includes multiple GPUs. Each node can have a complete instance of the model, with multiple separate GPUs within each node used to train different parts of the model.

[0267] Distributed training has increased overhead relative to training on a single machine. However, the parallel processors and GPGPUs described herein can each implement various techniques for reducing the overhead of distributed training, including techniques for enabling high-bandwidth GPU-GPU data transfer and accelerated remote data synchronization.

[0268] Example Machine Learning Applications

[0269] Machine learning can be applied to solve various technical problems, including but not limited to computer vision, autonomous driving and navigation, speech recognition, and language processing. Computer vision has traditionally been one of the most active research areas for machine learning applications. The application range of computer vision is from reproducing human visual capabilities (such as recognizing faces) to creating new categories of visual capabilities. For example, computer vision applications can be configured to identify sound waves from vibrations induced in objects visible in a video. Parallel processor-accelerated machine learning enables the use of significantly larger training data sets than previously feasible training data sets to train computer vision applications, and enables the use of low-power parallel processors to deploy inference systems.

[0270] Parallel processor-accelerated machine learning has autonomous driving applications, including lane and road sign recognition, obstacle avoidance, navigation, and driving control. Accelerated machine learning techniques can be used to train driving models based on data sets that define appropriate responses to specific training inputs. The parallel processors described in this article enable rapid training of increasingly complex neural networks for autonomous driving solutions and enable the deployment of low-power inference processors in mobile platforms suitable for integration into autonomous vehicles.

[0271] Deep neural networks accelerated by parallel processors have enabled machine learning methods for automatic speech recognition (ASR). ASR involves creating a function that computes the most likely speech sequence given a sequence of input sounds. Accelerated machine learning using deep neural networks has been implemented to replace the hidden Markov models (HMMs) and Gaussian mixture models (GMMs) previously used for ASR.

[0272] Parallel processor accelerated machine learning can also be used to accelerate natural language processing. Automatic learning programs can utilize statistical inference algorithms to produce models that are robust to erroneous or unfamiliar inputs. Exemplary natural language processor applications include automatic machine translation between human languages.

[0273] The parallel processing platform for machine learning can be divided into a training platform and a deployment platform. The training platform is typically highly parallel and includes optimizations for accelerating multi-GPU single-node training and multi-node multi-GPU training. Exemplary parallel processors suitable for training include highly parallel general-purpose graphics processing units and / or multi-GPU computing systems described herein. In contrast, the deployed machine learning platform typically includes a low-power parallel processor suitable for use in products such as cameras, autonomous robots, and autonomous vehicles.

[0274] Fig. 30B An exemplary inference system on chip (SOC) 3100 suitable for performing inference using a trained model is shown. The SOC 3100 can integrate multiple processing components, including a media processor 3102, a visual processor 3104, a GPGPU 3106, and a multi-core processor 3108. The SOC 3100 may additionally include an on-chip memory 3105 that enables a shared on-chip data pool that can be accessed by each of the processing components. The processing components can be optimized for low-power operation to enable deployment to various machine learning platforms, including autonomous vehicles and autonomous robots. For example, an implementation of the SOC 3100 can be used as part of a master control system for an autonomous vehicle. In the case where the SOC 3100 is configured for use in an autonomous vehicle, the SOC is designed and configured to comply with the relevant functional safety standards of the deployment jurisdiction.

[0275] During operation, the media processor 3102 and the visual processor 3104 can work in unison to accelerate computer vision operations. The media processor 3102 can enable low latency decoding of multiple high-resolution (e.g., 4K, 8K) video streams. The decoded video stream can be written to a buffer in the on-chip memory 3105. Subsequently, the visual processor 3104 can parse the decoded video and perform preliminary processing operations on the frames of the decoded video in preparation for processing the frames using the trained image recognition model. For example, the visual processor 3104 can accelerate convolution operations for a CNN that is used to perform image recognition on high-resolution video data, while the back-end model calculations are performed by the GPGPU 3106.

[0276] The multi-core processor 3108 may include control logic to assist in sequencing and synchronization of data transfers and shared memory operations performed by the media processor 3102 and the vision processor 3104. The multi-core processor 3108 may also act as an application processor to execute software applications that may utilize the inference computing capabilities of the GPGPU 3106. For example, at least a portion of navigation and driving logic may be implemented in software executing on the multi-core processor 3108. Such software may issue computational workloads directly to the GPGPU 3106, or may issue computational workloads to the multi-core processor 3108, which may offload at least a portion of those operations to the GPGPU 3106.

[0277] GPGPU 3106 may include a computing cluster, such as a low-power configuration of computing clusters DPLAB06A-DPLAB06 within a highly parallel general-purpose graphics processing unit DPLAB00. The computing cluster within GPGPU 3106 may support instructions specifically optimized to perform inference calculations on trained neural networks. For example, GPGPU 3106 may support instructions for performing low-precision calculations, such as 8-bit and 4-bit integer vector operations.

[0278] Ray Tracing Architecture

[0279] In one implementation, a graphics processor includes circuitry and / or program code for performing real-time ray tracing. In some embodiments, a set of dedicated ray tracing cores are included in the graphics processor to perform various ray tracing operations described herein, including ray traversal and / or ray intersection operations. In addition to the ray tracing cores, one embodiment also includes multiple sets of graphics processing cores for performing programmable shading operations and multiple sets of tensor cores for performing matrix operations on tensor data.

[0280] Fig.31An exemplary portion of one such graphics processing unit (GPU) 3105 is illustrated, which includes a collection of dedicated graphics processing resources arranged as multi-core groups 3100A-3100N. Although details of only a single multi-core group 3100A are provided, it will be understood that other multi-core groups 3100B-3100N may be equipped with the same or similar collection of graphics processing resources.

[0281] As illustrated, the multi-core group 3100A may include a set 3130 of graphics cores, a set 3140 of tensor cores, and a set 3150 of ray tracing cores. The scheduler / dispatcher 3110 schedules and dispatches graphics threads for execution on the various cores 3130, 3140, 3150. A set 3120 of register files stores operand values ​​used by the cores 3130, 3140, 3150 when executing graphics threads. These register files may include, for example, integer registers for storing integer values, floating point registers for storing floating point values, vector registers for storing packed data elements (integer and / or floating point data elements), and slice registers for storing tensor / matrix values. In one embodiment, the slice registers are implemented as a set of combined vector registers.

[0282] One or more first level (L1) caches and texture units 3160 store graphics data locally within each multi-core group 3100A, such as texture data, vertex data, pixel data, ray data, bounding volume data, etc. A second level (L2) cache 3180 shared by all multi-core groups 3100A-3100N or a subset of multi-core groups 3100A-3100N stores data and / or instructions for multiple concurrent graphics threads. As illustrated, the L2 cache 3180 can be shared across multiple multi-core groups 3100A-3100N. One or more memory controllers 3170 couple the GPU 3105 to a memory 3198, which can be a system memory (e.g., DRAM) and / or a dedicated graphics memory (e.g., GDDR6 memory).

[0283] Input / output (IO) circuitry 3195 couples the GPU 3105 to one or more IO devices 3190, such as a digital signal processor (DSP), a network controller, or a user input device. On-chip interconnects may be used to couple the I / O devices 3190 to the GPU 3105 and memory 3198. One or more memory management units (IOMMUs) 3170 of the IO circuitry 3195 directly couple the IO devices 3190 to the system memory 3198. In one embodiment, the IOMMU 3170 manages page tables for mapping virtual addresses to multiple sets of physical addresses in the system memory 3198. In this embodiment, the IO devices 3190, the CPU(s) 3199, and the GPU(s) 3105 may share the same virtual address space.

[0284] In one implementation, the IOMMU 3170 supports virtualization. In this case, the IOMMU 3170 manages a first set of page tables for mapping guest / graphics virtual addresses to guest / physical addresses and a second set of page tables for mapping guest / graphics physical addresses to system / host physical addresses (e.g., within system memory 3198). The base address of each of the first set of page tables and the second set of page tables may be stored in a control register and swapped out upon context switch (e.g., so that the new context is provided with access to the associated set of page tables). Although not described in detail in the prior art, the IOMMU 3170 may be used to map guest / graphics virtual addresses to guest / physical addresses. Fig.31 , but each of the cores 3130, 3140, 3150 and / or multi-core groups 3100A-3100N may include a translation lookaside buffer (TLB) that is used to cache guest virtual to guest physical translations, guest physical to host physical translations, and guest virtual to host physical translations.

[0285] In one embodiment, the CPU 3199, GPU 3105, and IO devices 3190 are integrated on a single semiconductor chip and / or chip package. The illustrated memory 3198 may be integrated on the same chip, or may be coupled to the memory controller 3170 via an off-chip interface. In one implementation, the memory 3187 includes GDDR6 memory that shares the same virtual address space as other physical system-level memories, but the underlying principles of the invention are not limited to this particular implementation.

[0286] In one embodiment, the tensor core 3140 includes multiple execution units specifically designed to perform matrix operations, which are the basic computational operations used to perform deep machine operations. For example, synchronized matrix multiplication operations can be used for neural network training and inference. The tensor core 3140 can perform matrix processing using a variety of operand precisions, including single-precision floating point (e.g., 32 bits), half-precision floating point (e.g., 16 bits), integers (16 bits), bytes (8 bits), and half bytes (4 bits). In one embodiment, the neural network implementation extracts features of each rendered scene, potentially combining details from multiple frames to build a high-quality final image.

[0287] In a deep learning implementation, parallel matrix multiplication work can be scheduled for execution on the Tensor Core 3140. The training of neural networks in particular requires a large number of matrix dot product operations. To handle the inner product formulation of the N x N x N matrix multiplication, the Tensor Core 3140 may include at least N dot product processing elements. Before the matrix multiplication begins, a complete matrix is ​​loaded into the slice register, and for each of the N loops, at least one column of the second matrix is ​​loaded. For each loop, there are N dot products that are processed.

[0288] Depending on the specific implementation, matrix elements can be stored in different precisions, including 16-bit words, 8-bit bytes (e.g., INT8), and 4-bit nibbles (e.g., INT4). Different precision modes can be specified for the Tensor Core 3140 to ensure that the most efficient precision is used for different workloads (e.g., such as inference workloads, which can tolerate quantization to bytes and nibbles).

[0289] In one embodiment, the ray tracing core 3150 accelerates ray tracing operations for both real-time ray tracing implementations and non-real-time ray tracing implementations. Specifically, the ray tracing core 3150 includes a ray traversal / intersection circuit for performing ray traversals using a bounding volume hierarchy (BVH) and identifying intersections between rays and primitives enclosed within the BVH volume. The ray tracing core 3150 may also include circuits for performing depth testing and culling (e.g., using a Z buffer or similar arrangement). In one implementation, the ray tracing core 3150 performs traversal and intersection operations in concert with the image denoising techniques described herein, at least a portion of which may be performed on the tensor core 3140. For example, in one embodiment, the tensor core 3140 implements a deep learning neural network to perform denoising on frames generated by the ray tracing core 3150. However, the CPU(s) 3199, the graphics core 3130, and / or the ray tracing core 3150 may also implement all or part of the denoising and / or deep learning algorithms.

[0290] In addition, as described above, a distributed approach to noise reduction can be employed in which the GPU 3105 is in a computing device coupled to other computing devices via a network or high-speed interconnect. In this embodiment, the interconnected computing devices share neural network learning / training data to improve the speed at which the entire system learns to perform noise reduction for different types of image frames and / or different graphics applications.

[0291] In one embodiment, the ray tracing core 3150 handles all BVH traversals and ray-primitive intersections, saving the graphics core 3130 from being overloaded with thousands of instructions for each ray. In one embodiment, each ray tracing core 3150 includes a first set of specialized circuits for performing bounding box tests (e.g., for traversal operations) and a second set of specialized circuits for performing ray-triangle intersection tests (e.g., intersecting rays that have been traversed). Thus, in one embodiment, the multi-core group 3100A can simply initiate ray detection, and the ray tracing core 3150 independently performs ray traversals and intersections and returns hit data (e.g., hits, no hits, multiple hits, etc.) to the thread context. While the ray tracing core 3150 performs traversal and intersection operations, the other cores 3130, 3140 are freed up to perform other graphics or computational work.

[0292] In one embodiment, each ray tracing core 3150 includes a traversal unit for performing BVH test operations and an intersection unit for performing ray-primitive intersection tests. The intersection unit generates a "hit", "no hit", or "multiple hits" response, which it provides to the appropriate thread. During traversal and intersection operations, execution resources of other cores (e.g., graphics cores 3130 and tensor cores 3140) are freed up to perform other forms of graphics work.

[0293] In one particular embodiment described below, a hybrid rasterization / ray tracing approach is used in which the work is distributed between the graphics core 3130 and the ray tracing core 3150.

[0294] In one embodiment, the ray tracing core 3150 (and / or other cores 3130, 3140) includes hardware support for ray tracing instruction sets, such as Microsoft's DirectX Ray Tracing (DXR), which includes the DispatchRays command; and ray generation shaders, nearest hit shaders, any hit shaders, and miss shaders, which enable the assignment of a unique set of shaders and textures to each object. Another ray tracing platform that can be supported by the ray tracing core 3150, graphics core 3130, and tensor core 3140 is Vulkan 1.1.85. However, it is to be noted that the underlying principles of the present invention are not limited to any particular ray tracing instruction set architecture ISA.

[0295] In general, each core 3150, 3140, 3130 may support a ray tracing instruction set including instructions / functions for: ray generation, nearest hit, any hit, ray-primitive intersection, per-primitive and hierarchy bounding box construction, misses, visits, and exceptions. More specifically, one embodiment includes ray tracing instructions for performing the following functions:

[0296] Light Generation - Ray generation instructions can be executed per pixel, sample, or user-defined work distribution.

[0297] Recent Hits - A nearest hit command may be executed to locate the closest intersection of a ray with a primitive within the scene.

[0298] Any hit - Any hit instruction identifies multiple intersections of a ray with a primitive within the scene, thereby potentially identifying a new closest intersection point.

[0299] intersect ——The intersection instruction performs a ray-primitive intersection test and outputs the result.

[0300] Per-primitive bounding box construction - This instruction builds a bounding box around a given primitive or group of primitives (e.g., when building a new BVH or other acceleration data structure).

[0301] miss - Indicates that the ray missed the scene or any geometry within the specified region of the scene.

[0302] visit ——Indicates the subvolume that the ray will traverse.

[0303] abnormal -- Includes various types of exception handlers (e.g., called for various error conditions).

[0304] Hierarchical Beam Tracing

[0305] Bounding volume hierarchies are often used to improve the efficiency of performing operations on graphics primitives and other graphics objects. BVH is a hierarchical tree structure based on a collection of geometric objects. At the top of the tree structure is a root node that encloses all geometric objects in a given scene. Each geometric object is wrapped in a bounding volume that forms a leaf node of the tree. These nodes are then grouped into multiple small sets and enclosed in larger bounding volumes. These nodes are in turn recursively grouped and enclosed in other larger bounding volumes, ultimately resulting in a tree structure with a single bounding volume, which is represented by a root node at the top of the tree. The bounding volume hierarchy is used to efficiently support various operations on a collection of geometric objects, such as collision detection, primitive culling, and traversal / intersection operations used in ray tracing.

[0306] In a ray tracing framework, a ray is traversed through a BVH to determine ray-primitive intersections. For example, if a ray does not pass through the root node of a BVH, then the ray does not intersect any of the primitives enclosed by the BVH, and no further processing is required for the ray with respect to the set of primitives. If a ray passes through the first child node of a BVH but not the second child node, then there is no need to test the ray against the primitives enclosed by the second child node. In this way, the BVH provides an efficient mechanism for testing ray-primitive intersections.

[0307] In one embodiment of the invention, groups of adjacent rays, called "beams," are tested against the BVH rather than testing individual rays. Fig.32 An exemplary light beam 3201 is illustrated as being outlined by four different rays. Any ray that intersects the patch 3200 defined by the four rays is considered to be within the same light beam. Fig.32 The light beam 3201 in FIG. 3 is defined by a rectangular arrangement of light beams, but the light beams may be defined in various other ways while still conforming to the underlying principles of the invention (eg, circular, elliptical, etc.).

[0308] Fig.33 An exemplary embodiment is shown in which a ray tracing engine 3310 of a GPU 3320 implements the beam tracing techniques described herein. Specifically, a ray generation circuit 3304 generates a plurality of rays for which traversal and intersection operations are to be performed. However, rather than performing traversal and intersection operations on individual rays, the illustrated embodiment uses a hierarchy 3307 of beams generated by the beam hierarchy construction circuit 3304 to perform the traversal and intersection. In one embodiment, the beam hierarchy is similar to a bounding volume hierarchy (BVH). For example, Fig.34An example of a main beam 3400 is provided, which can be subdivided into a number of different components. Specifically, the main beam 3400 can be subdivided into quadrants 3401-3404, and each quadrant itself can be divided into a number of sub-quadrants, such as a number of sub-quadrants AD within quadrant 3404. The main beam can be subdivided in a variety of ways. For example, in one embodiment, the main beam can be divided into two halves (rather than quadrants), and each half can be divided into two halves, and so on. Regardless of how the subdivision is performed, in one embodiment, a hierarchical structure is generated in a manner similar to a BVH, for example, where the root node represents the main beam 3400 - a first-level child node, each first-level node is represented by a quadrant 3401-3404 (a second-level child node for each sub-quadrant AD), and so on.

[0309] In one embodiment, once the beam hierarchy 3307 is constructed, the traversal / intersection circuit 3306 performs a traversal / intersection operation using the beam hierarchy 3307 and the BVH 3308. Specifically, the traversal / intersection circuit 3306 tests the beam against the BVH and removes portions of the beam that do not intersect any portion of the BVH. Using, for example Fig.34 Based on the data shown in FIG. 34, if the sub-beams associated with sub-areas 3402 and 3403 do not intersect the BVH or a particular branch of the BVH, then these sub-beams can be eliminated relative to the BVH or the branch. The remaining portions 3401, 3404 can be tested against the BVH by performing a depth-first search or other search algorithm.

[0310] exist Fig.35 A method according to one embodiment of the present invention is illustrated in . The method may be implemented within the context of the graphics processing architecture described above, but is not limited to any particular architecture.

[0311] At 3500, a main beam including a plurality of rays is constructed, and at 3501, the beam is subdivided and a hierarchical data structure is generated to create a beam hierarchy. In one embodiment, operations 3500-3501 are performed as a single, integrated operation to construct a beam hierarchy from a plurality of rays. At 3502, the beam hierarchy is used using a BVH to cull rays (from the beam hierarchy) and / or cull nodes / primitives from the BVH. At 3503, ray-primitive intersections are determined for the remaining rays and primitives.

[0312] Lossy and lossless packet compression in distributed ray tracing systems

[0313] In one embodiment, ray tracing operations are distributed across multiple computing nodes coupled together by a network. Fig.36For example, a ray tracing cluster 3600 is shown, which includes multiple ray tracing nodes 3610-3613, which are used to perform ray tracing operations in parallel, thereby potentially combining the results on each of the nodes. In the illustrated architecture, the ray tracing nodes 3610-3613 are communicatively coupled to the client-side ray tracing application 3630 via a gateway.

[0314] One of the difficulties of a distributed architecture is the large amount of packetized data that must be transmitted between each of ray tracing nodes 3610-3613. In one embodiment, both lossless and lossy compression techniques are used to reduce the data transmitted between ray tracing nodes 3610-3613.

[0315] To achieve lossless compression, instead of sending packets filled with the results of certain types of operations, data or commands are sent that allow the receiving node to reconstruct the results. For example, randomly sampled surface lights and ambient occlusion (AO) operations do not necessarily require direction. As a result, in one embodiment, the transmitting node will simply send a random seed that is then used by the receiving node to perform random sampling. For example, if the scene is distributed across nodes 3610-3612, in order to sample light 1 at points p1-p3, only the light ID and origin need to be sent to nodes 3610-3612. Subsequently, each of the nodes can sample the light randomly and independently. In one embodiment, the random seed is generated by the receiving node. Similarly, for the main beam hit point, ambient occlusion (AO) and soft shadow samples can be calculated on nodes 3610-3612 without waiting for the original point for consecutive frames. In addition, if it is known that a set of rays will go to the same point light source, an instruction identifying the light source can be sent to the receiving node, which will apply the light source to the set of rays. As another example, if there are N ambient occlusion rays emitted from a single point, a command may be sent to generate N samples from that point.

[0316] Various additional techniques may be applied to lossy compression. For example, in one embodiment, a discretization factor may be employed to discretize all coordinate values ​​associated with BVHs, primitives, and rays. In addition, 32-bit floating point values ​​for data such as BVH nodes and primitives may be converted to 8-bit integer values. In one particular implementation, the limits of ray groups are stored with full precision, but the individual ray points P1-P3 are transmitted as indexed offsets to these limits. Similarly, multiple local coordinate systems may be generated that use 8-bit integer values ​​as local coordinates. The position of the origin of each of these local coordinate systems may be encoded using full precision (e.g., 32-bit floating point) values, thereby effectively connecting the global coordinate system and the local coordinate system.

[0317] The following is an example of lossless compression used in one embodiment of the present invention. An example of the ray data format used inside a ray tracing program is as follows:

[0318] struct Ray

[0319] {

[0320] uint32 pixId;

[0321] uint32 materialID;

[0322] uint32 instanceID;

[0323] uint64 primitiveID;

[0324] uint32 geometryID;

[0325] uint32 lightID;

[0326] float origin[3];

[0327] float direction[3];

[0328] float t0;

[0329] float t;

[0330] float time;

[0331] float normal[3]; / / used for geometric intersection

[0332] float u;

[0333] float v;

[0334] float wavelength;

[0335] float phase; / / interference measurement

[0336] float refractedOffset; / / Schlieren-esque

[0337] float amplitude;

[0338] float weight;

[0339] };

[0340] Instead of sending raw data for each and every node generated, this data can be compressed by grouping values ​​and creating implicit rays with applicable metadata where possible.

[0341] Binding and grouping ray data

[0342] One embodiment uses flags for common data or masks with modifiers.

[0343] struct RayPacket

[0344] {

[0345] uint32 size;

[0346] uint32 flags;

[0347] list <ray>rays;

[0348] }

[0349] For example:

[0350] RayPacket.rays=ray_1 to ray_256

[0351] All origins are shared

[0352] All ray data is packed, with the exception that only a single origin is stored across all rays. Set RayPacket.flags to RAYPACKET_COMMON_ORIGIN. When a RayPacket is unpacked upon receipt, fill in the origin from the single origin value.

[0353] The origin is only shared between some rays

[0354] All ray data is packed, except for rays that share an origin. For each group with multiple unique shared origins, an operator is packed that identifies the operation (the shared origin), stores the origin, and masks which rays share information. Such operations can be performed on any shared value between multiple nodes, such as material IDs, primitive IDs, origins, directions, normals, etc.

[0355] struct RayOperation

[0356] {

[0357] uint8 operationID;

[0358] void*value;

[0359] uint64 mask;

[0360] }

[0361] Sending Implicit Rays

[0362] From time to time it is often possible to derive ray data on the receiving end with minimal meta information used to generate the ray data. A very common example is to generate multiple secondary rays to randomly sample an area. Instead of the sender generating a secondary ray, sending it, and the receiver operating on it, the sender can send a command that requires a ray to be generated with any slave attribute information, and generate the ray on the receiving end. In cases where a ray needs to be generated first by the sender to determine which receiver to send it to, the ray is generated and a random seed can be sent to regenerate the exact same ray.

[0363] For example, to sample a hit point with 64 shadow rays sampling an area light, all 64 rays intersect the region from the same computation N4. A RayPacket is created with a common origin and normal. More data can be sent if the receiver is expected to shade the resulting pixel contribution, but for this example, let's assume we want to return only data on whether the ray hit another node. A RayOperation is created to generate the shadow ray operation and is assigned the value of the lightID to sample as well as a random seed. When N4 receives the ray packet, it generates a fully populated ray data by filling all rays with the shared origin data and setting the direction based on the lightID randomly sampled with the random seed, thereby generating the same rays generated by the original sender. When the results are returned, only the binary result for each ray needs to be returned, which can be passed by the mask to the ray.

[0364] Sending the original 64 rays in this example will use 104 bytes * 64 rays = 6656 bytes. If the return rays are also sent in their original form, this is further doubled to 13312 bytes. Using lossless compression in which only the common ray origin, normal, and ray generation operations with seeds and IDs are sent, only 29 bytes are sent, of which 8 bytes are returned for the intersected mask. This results in a data compression ratio of about 360:1 that needs to be sent over the network. This does not include the overhead for processing the message itself, which will need to be identified in some way, but that is determined by the implementation. Other operations can be performed for: recalculating the ray origin and direction from the pixelID of the main ray; recalculating the pixelID based on the range in the raypacket; and many other possible implementations for recalculating values. Similar operations can be used for any single ray or single group of rays sent, including shadows, reflections, refractions, ambient occlusion, intersections, volume intersections, shading, bounce reflections, etc. in path tracing.

[0365] Fig.37 Additional details are illustrated for two ray tracing nodes 3710-3711, which perform compression and decompression of ray tracing packets. Specifically, in one embodiment, when the first ray tracing engine 3730 is ready to transfer data to the second ray tracing engine 3731, the ray compression circuit 3720 performs lossy and / or lossless compression of the ray tracing data as described herein (e.g., converting 32-bit values ​​to 8-bit values, replacing the original data with instructions for reconstructing the data, etc.). The compressed ray packets 3701 are transmitted from the network interface 3725 to the network interface 3726 via a local network (e.g., a 10Gb / s, 100Gb / s Ethernet network). The ray decompression circuit then decompresses the ray packets where appropriate. For example, the ray decompression circuit may execute commands for reconstructing the ray tracing data (e.g., using a random seed to perform random sampling for lighting operations). The ray tracing engine 3731 then uses the received data to perform ray tracing operations.

[0366] In the reverse direction, compression circuitry 3741 compresses the ray data, network interface 3726 transmits the compressed ray data over the network (e.g., using the techniques described herein), ray decompression circuitry 3740 decompresses the ray data if necessary, and ray tracing engine 3730 uses the data in ray tracing operations. Fig.37 Although shown as separate units in FIG. 3 , ray decompression circuits 3740-3741 may be integrated into ray tracing engines 3730-3731, respectively. For example, to the extent that the compressed ray data includes commands for reconstructing the ray data, these commands may be executed by each respective ray tracing engine 3730-3731.

[0367] like Fig.38 As shown in FIG. 3 , the ray compression circuit 3720 may include a lossy compression circuit 3801 for performing the lossy compression techniques described herein (e.g., converting 32-bit floating point coordinates to 8-bit integer coordinates) and a lossless compression circuit 3803 for performing lossless compression techniques (e.g., transmitting commands and data to allow the ray decompression circuit 3821 to reconstruct the data). The ray decompression circuit 3721 includes a lossy decompression circuit 3802 and a lossless decompression circuit 3804 for performing lossless decompression.

[0368] exist Fig.39 A method according to one embodiment is illustrated in . The method may be implemented on the ray tracing architecture described herein, but is not limited to any particular architecture.

[0369] At 3900, ray data to be transmitted from a first ray tracing node to a second ray tracing node is received. At 3901, lossy compression circuitry performs lossy compression on the first ray tracing data, and at 3902, lossless compression circuitry performs lossless compression on the second ray tracing data. At 3903, the compressed ray tracing data is transmitted to the second ray tracing node. At 3904, lossy / lossless decompression circuitry performs lossy / lossless decompression of the ray tracing data, and at 3905, the second ray tracing node performs a ray tracing operation using the decompressed data.

[0370] GPU with hardware-accelerated hybrid ray tracing

[0371] One embodiment of the present invention includes a hybrid rendering pipeline that performs rasterization on the graphics core 3130 and performs ray tracing operations on the ray tracing core 3150, the graphics core 3130, and / or the CPU 3199 core. For example, rasterization and depth testing may be performed on the graphics core 3130 in place of the primary ray casting stage. The ray tracing core 3150 may then generate secondary rays for light reflections, refractions, and shadows. In addition, certain embodiments may select certain areas of the scene in which the ray tracing core 3150 will perform ray tracing operations (e.g., based on material property thresholds such as high reflectivity levels), while other areas of the scene will be rendered using rasterization on the graphics core 3130. In one embodiment, this hybrid implementation is used for real-time ray tracing applications, where latency is a critical issue.

[0372] One embodiment of the ray traversal architecture described below performs programmable shading and control of ray traversal using existing single instruction multiple data (SIMD) and / or single instruction multiple thread (SIMT) graphics processors while using dedicated hardware to accelerate key functions such as BVH traversal and / or intersection. In this embodiment, SIMD occupancy for irrelevant paths is improved by regrouping the generated shaders at specific points during traversal and before shading. This is achieved using dedicated hardware that dynamically sorts the shaders on-chip. Recursion is managed by splitting functions into continuations that are executed on return and regrouping the continuations before execution for improved SIMD occupancy.

[0373] Programmable control of ray traversal / intersection is achieved by decomposing the traversal functionality into an inner traversal, which can be implemented as fixed-function hardware, and an outer traversal, which is executed on the GPU processor and enables programmable control through user-defined traversal shaders. The cost of passing traversal context between hardware and software is reduced by carefully truncating the inner traversal state during the transition between the inner and outer traversals.

[0374] Programmable control of ray tracing can be expressed through different shader types listed below in Table A. There can be multiple shaders for each type. For example, each material can have a different hit shader.

[0375] Shader Types Function main Shooting the main ray Hit Bidirectional reflectance distribution function (BRDF) sampling, emitting secondary rays Any hit Calculating transmittance of alpha-textured geometry miss Calculate the radiance from a light source intersect Intersecting custom shapes Traversal Instance selection and transformation Callable Universal Functions

[0376] Table A

[0377] In one embodiment, recursive ray tracing is initiated by an API function that commands the graphics processor to start a set of primary shaders, or an intersection circuit that can generate ray-scene intersections for primary rays. This in turn generates other shaders, such as traversal shaders, hit shaders, or miss shaders. The shader that generates a sub-shader can also receive a return value from that sub-shader. A callable shader is a general function that can be directly generated by another shader and can also return a value to the shader that made the call.

[0378] Fig.40 An embodiment of a graphics processor architecture is shown, which includes a shader execution circuit 4000 and a fixed function circuit 4010. The general execution hardware subsystem includes multiple single instruction multiple data (SIMD) and / or single instruction multiple thread (SIMT) cores / execution units (EUs) 4001 (i.e., each core may include multiple execution units), one or more samplers 4002, a first level (L1) cache 4003 or other form of local memory. The fixed function hardware subsystem 4010 includes a message passing unit 4004, a scheduler 4007, a ray-BVH traversal / intersection circuit 4005, a sorting circuit 4008, and a local L1 cache 4006.

[0379] In operation, the master dispatcher 4009 dispatches a set of primary rays to the scheduler 4007, which schedules work to the shader executing on the SIMD / SIMT core / EU 4001. The SIMD core / EU 4001 can be the ray tracing core 3150 and / or the graphics core 3130 described above. The execution of the primary shader generates additional work to be executed (e.g., to be executed by one or more sub-shaders and / or fixed-function hardware). The message passing unit 4004 distributes the work generated by the SIMD core / EU 4001 to the scheduler 4007 (thereby accessing the free stack pool as needed), the sorting circuit 4008, or the ray-BVH intersection circuit 4005. If additional work is sent to the scheduler 4007, the additional work is scheduled to be processed on the SIMD / SIMT core / EU 4001. Prior to scheduling, sorting circuitry 4008 may sort rays into groups or into bins as described herein (e.g., grouping rays with similar characteristics). Ray-BVH intersection circuitry 4005 performs intersection testing of rays using BVH containers. For example, ray-BVH intersection circuitry 4008 may compare ray coordinates to the BVH at each level to identify containers that the ray intersects.

[0380] A shader record is a user-allocated structure that includes a pointer to an entry function, vendor-specific metadata, and global variables for the shader executed by the SIMD core / EU 4001. Each execution instance of a shader is associated with a call stack that can be used to store variables passed between parent and child shaders. The call stack can also store references to extension functions that are executed when a call returns.

[0381] Fig.41 An example set of assigned stacks 4101 is shown, including a main shader stack, a hit shader stack, a traversal shader stack, a continuation function stack, and a ray-BVH intersection stack (which, as described, may be executed by fixed function hardware 4010). A new shader call may implement a new stack from a free stack pool 4102. The call stack may be cached in a local L1 cache 4003, 4006 to reduce latency for access.

[0382] In one embodiment, there are a finite number of call stacks, each with a fixed maximum size "S" allocated in a contiguous area of ​​memory. 栈 Therefore, the base address of the stack can be calculated directly through the stack index (SID) as base address = SID * S 栈 In one embodiment, when work is scheduled to a SIMD core / EU 4001, a stack ID is allocated and deallocated by the scheduler 4007.

[0383] In one embodiment, the master dispatcher 4009 includes a graphics processor command processor that dispatches master shaders in response to dispatch commands from a host (e.g., a CPU). The scheduler 4007 receives these dispatch requests, and the scheduler starts the master shader on a SIMD processor thread if it can allocate a stack ID for each SIMD lane. The stack ID is allocated from the free stack pool 4102, which is initialized at the beginning of the dispatch command.

[0384] The execution shader may generate a child shader by sending a generate message to the messaging unit 4004. The command includes a stack ID associated with the shader and also includes a pointer to a child shader record for each active SIMD channel. The parent shader may issue this message only once for an active channel. In one embodiment, after issuing the generate message for all relevant channels, the parent shader terminates.

[0385] The shader executed on the SIMD core / EU 4001 can also generate fixed function tasks such as ray-BVH intersection using a generate message with a shader record pointer reserved for fixed function hardware. As mentioned, the message passing unit 4004 sends the generated ray-BVH intersection work to the fixed function ray-BVH intersection circuit 4005 and directly sends the callable shader to the sorting circuit 4008. In one embodiment, the sorting circuit groups the shaders by the shader record pointer to derive SIMD batches with similar characteristics. Accordingly, the stack IDs from different parent shaders can be grouped in the same batch by the sorting circuit 4008. The sorting circuit 4008 sends the grouped batches to the scheduler 4007, which accesses the shader record from the graphics memory 2511 or the last level cache (LLC) 4020 and starts the shader on the processor thread.

[0386] In one embodiment, the extension is treated as a callable shader and can also be referenced by a shader record. When a child shader is generated and returns a value to the parent shader, a pointer to the extension shader record is pushed onto the call stack 4101. When the child shader returns, the extension shader record is popped from the call stack 4101 and the extension shader is generated. The generated extension passes through a sorting unit similar to the callable shader and is started on the processor thread.

[0387] As in Fig.42 As shown in , one embodiment of the sorting circuit 4008 groups the generated tasks by shader record pointers 4201A, 4201B, 4201n to create SIMD batches to be shaded. Stack IDs or context IDs from different dispatches and different input SIMD channels can be grouped in sorted batches. In one embodiment, the grouping circuit 4210 uses a content addressable memory (CAM) structure 4201 to perform the sorting, and the CAM structure 4201 includes a plurality of entries, each of which is identified by a tag 4201. As mentioned, in one embodiment, the tag 4201 is a corresponding shader record pointer 4201A, 4201B, 4201n. In one embodiment, the CAM structure 4201 stores a limited number of (e.g., 32, 64, 128, etc.) tags, each of which is associated with an incomplete SIMD batch corresponding to a shader record pointer.

[0388] For incoming generation commands, each SIMD lane has a corresponding stack ID (shown as 16 context IDs 0-15 in each CAM entry) and a shader record pointer 4201A-B, ... n (serving as a tag value). In one embodiment, the grouping circuit 4210 compares the shader record pointer for each lane with the tag 4201 in the CAM structure 4201 to find a matching batch. If a matching batch is found, the stack ID / context ID is added to the batch. Otherwise, a new entry with a new shader record pointer tag is created, thereby possibly expelling an older entry with an incomplete batch.

[0389] When the call stack is empty, the executing shader can deallocate the call stack by sending a deallocation message to the messaging unit. The deallocation message is relayed to the scheduler, which returns the stack ID / context ID for the active SIMD lane to the free pool.

[0390] One embodiment of the present invention implements a hybrid approach for ray traversal operations using a combination of fixed-function ray traversal and software ray traversal. Thus, the approach provides the flexibility of software traversal while maintaining the efficiency of fixed-function traversal. Fig.43 An acceleration structure that can be used for hybrid traversal is shown, which is a two-level tree with a single top-level BVH 4300 and several bottom-level BVHs 4301 and 4302. Graphic elements are shown on the right to indicate inner traversal paths 4303, outer traversal paths 4304, traversal nodes 4305, leaf nodes with triangles 4306, and leaf nodes with custom primitives 4307.

[0391] The leaf nodes 4306 with triangles in the top level BVH 4300 may reference triangle intersection shader records for custom primitives, or reference traversal shader records. The leaf nodes 4306 with triangles in the bottom level BVH 4301-4302 may only reference triangle intersection shader records for custom primitives. The type of reference is encoded in the leaf node 4306. An inner traversal refers to a traversal within each BVH 4300-4302. An inner traversal operation includes the calculation of ray-BVH intersections, and a traversal across the BVH structures 4300-4302 is referred to as an outer traversal. The inner traversal operation can be efficiently implemented in fixed function hardware, while an outer traversal operation can be performed with acceptable performance using a programmable shader. Thus, one embodiment of the present invention uses fixed function circuitry 4010 to perform inner pass operations and uses shader execution circuitry 4000, which includes a SIMD / SIMT core / EU 4001 for executing a programmable shader, to perform outer pass operations.

[0392] Note that for simplicity, the SIMD / SIMT core / EU 4001 is sometimes referred to herein as simply a "core," "SIMD core," "EU," or "SIMD processor." Similarly, the ray-BVH traversal / intersection circuit 4005 is sometimes referred to herein as simply a "traversal unit," "traversal / intersection unit," or "traversal / intersection circuit." When alternative terminology is used, the specific name used to designate the corresponding circuit / logic does not change the underlying functionality performed by that circuit / logic as described herein.

[0393] In addition, although Fig.40 4004 is illustrated as a single component for purposes of explanation, but the traversal / intersection unit 4005 may include different traversal units and a separate intersection unit, each of which may be implemented in circuits and / or logic as described herein.

[0394] In one embodiment, when a ray intersects a traversal node during an inner traversal, a traversal shader is generated. The sorting circuit 4008 groups these shaders through shader record pointers 4201A-B,n to create a SIMD batch, which is started by the scheduler 4007 for SIMD execution on the graphics SIMD core / EU 4001. The traversal shader can modify the traversal in several ways, thereby allowing a wide range of applications. For example, the traversal shader can select a BVH or transform rays with a coarser level of detail (LOD) to enable strict body transformation. The traversal shader then generates an inner traversal for the selected BVH.

[0395] The inner pass computes ray-BVH intersections by traversing the BVH and computing ray-box intersections and ray-triangle intersections. The inner pass is generated in the same manner as the shader by sending messages to the message passing circuit 4004, which relays the corresponding generated messages to the ray-BVH intersection circuit 4005, which computes the ray-BVH intersections.

[0396] In one embodiment, the stack for the inner traversal is stored locally in the fixed function circuit 4010 (e.g., in the L1 cache 4006). When the ray intersects with the leaf node corresponding to the traversal shader or the intersection shader, the inner traversal is terminated and the inner stack is truncated. The truncated stack is written to the location specified by the calling shader in the memory together with the pointer to the ray and the BVH, and the corresponding traversal shader or intersection shader is generated. If the ray intersects with any triangle during the inner traversal, the corresponding hit information is provided as an input variable to these shaders, as shown in the following code. These generated shaders are grouped by the sorting circuit 4008 to create a SIMD batch for execution.

[0397] struct HitInfo{

[0398] float barycentrics[2];

[0399] float tmax;

[0400] bool innerTravComplete;

[0401] uint primID;

[0402] uint geomID;

[0403] ShaderRecord*leafShaderRecord;

[0404] }

[0405] Truncating the inner traversal stack reduces the cost of streaming the inner traversal stack out to memory. One embodiment of the present invention uses the method described in "Restart Trail for Stackless BVH Traversal, High Performance Graphics (2010)" pages 107-111 to truncate the stack to a small number of entries at the top of the stack, namely a 42-bit restart trail and a 6-bit depth value. The restart trail indicates the branches that have been taken inside the BVH, and the depth value indicates the depth of the traversal corresponding to the last stack entry. This is enough information to resume the inner traversal later.

[0406] When the inner stack is empty and there are no more BVH nodes to test, the inner traversal is complete. In this case, an outer stack handler is generated that pops the top of the outer stack and resumes traversal if the outer stack is not empty.

[0407] In one embodiment, the outer traversal executes the main traversal state machine and is implemented in program code executed by the shader execution circuit 4000. The program code generates an inner traversal query under the following conditions: (1) when a new ray is generated by a hit shader or a main shader; (2) when a traversal shader selects a BVH to traverse; and (3) when the outer stack handler resumes the inner traversal for the BVH.

[0408] As in Fig.44 As illustrated in , before generating the inner traversal, space is allocated on the call stack 4405 for the fixed function circuit 4010 to store the truncated inner stack 4410. Offsets 4403-4404 to the top of the call stack and the inner stack are maintained in the traversal state 4400, which is also stored in the memory 2511. The traversal state 4400 also includes the ray 4401 in the world space 4401 and the ray 4402 in the target space and the hit information for the nearest intersecting primitive.

[0409] Traversal shaders, intersection shaders, and outer stack handlers are all generated by the ray-BVH intersection circuit 4005. Traversal shaders are allocated on the call stack 4405 before initiating an inner traversal for the second-level BVH. The outer stack handler is the shader responsible for updating hit information and resuming any pending inner traversal tasks. The outer stack handler is also responsible for generating a hit shader or a miss shader when the traversal is completed. When there are no pending inner traversal queries to be generated, the traversal is completed. When the traversal is completed and an intersection is found, a hit shader is generated; otherwise a miss shader is generated.

[0410] Although the hybrid traversal scheme described above uses a two-level BVH hierarchy, embodiments of the invention described herein may use any number of BVH hierarchies, with corresponding changes in the outer traversal implementation.

[0411] Furthermore, while the fixed function circuit 4010 is described in the above embodiments as being used to perform ray-BVH intersections, other system components may also be implemented in fixed function circuits. For example, the outer stack handler described above may be an internal (non-user visible) shader that may potentially be implemented in the fixed function BVH traversal / intersection circuit 4005. This implementation may be used to reduce the number of shader stages dispatched and the number of round trips between the fixed function intersection hardware 4005 and the processor.

[0412] Embodiments of the invention described herein use user-defined functions to enable programmable shading and ray traversal control, which can be executed on existing and future GPU processors with higher SIMD efficiency. Programmable control of ray traversal enables several important features, such as procedural instancing, random level of detail selection, custom primitive intersection, and lazy BVH updates.

[0413] Ray tracing instructions

[0414] The ray tracing instructions described below are included in an instruction set architecture (ISA) supported by one embodiment of the CPU 3199 and / or GPU 3105. If executed by the CPU, single instruction multiple data (SIMD) instructions can utilize vector / packed source and destination registers to perform the described operations and can be decoded and executed by the CPU core. If executed by the GPU 3105, the instructions can be executed by the graphics core 3130. For example, any of the execution units (EU) 4001 described above can execute the instructions. Alternatively, the instructions can be executed by execution circuits on the tensor core 3140 or the ray tracing core 3150.

[0415] Fig.45 One embodiment of an architecture for executing the ray tracing instructions described below is illustrated. The illustrated architecture may be integrated into the cores 3130, 3140, 3150 described above (e.g., see Fig.31 and associated text), or may be included in different processor architectures.

[0416] In operation, instruction fetch unit 4503 fetches ray tracing instructions 4500 from memory 3198, and decoder 4504 decodes the instructions. In one implementation, decoder 4504 decodes the instructions to generate executable operations (e.g., micro-operations or uops in a microcode core). Alternatively, some or all of ray tracing instructions 4500 may be executed without decoding, and thus decoder 4504 is not required.

[0417] In either implementation, the scheduler / dispatcher 4505 schedules and dispatches instructions (or operations) across a collection of functional units (FUs) 4510-4512. The illustrated embodiment includes a vector FU 4510 for executing single instruction multiple data (SIMD) instructions that concurrently operate on multiple packed data elements stored in vector registers 4515, and a scalar FU 4511 for operating on scalar values ​​stored in one or more scalar registers 4516. An optional ray tracing FU 4512 may operate on packed data values ​​stored in vector registers 4515 and / or scalar values ​​stored in scalar registers 4516. In embodiments without a dedicated FU 4512, the vector FU 4510, and possibly the scalar FU 4511, execute ray tracing instructions described below.

[0418] The various FUs 4510-4512 access ray tracing data 4502 (e.g., traversal / intersection data) required to execute ray tracing instructions 4500 from vector registers 4515, scalar registers 4516, and / or a local cache subsystem 4508 (e.g., L1 cache). In one embodiment, FUs 4510-4512 may also perform accesses to memory 3198 via load and store operations, and cache subsystem 4508 may operate independently to cache data locally.

[0419] Although ray tracing instructions can be used to improve performance for ray traversal / intersection and BVH building, they can also be used in other areas such as high performance computing (HPC) and general purpose GPU (GPGPU) implementations.

[0420] In the following description, the term double word is sometimes abbreviated as dw, and unsigned byte is abbreviated as ub. In addition, the source and destination registers (e.g., src0, src1, dest, etc.) referred to below may refer to vector registers 4515 or in some cases a combination of vector registers 4515 and scalar registers 4516. Typically, if the source or destination value used by the instruction includes packed data elements (e.g., where the source or destination stores N data elements), vector registers 4515 are used. Other values ​​may use scalar registers 4516 or vector registers 4515.

[0421] De-discretization

[0422] One embodiment of a dequantize instruction "de-discretizes" a previously discretized value. As an example, in a ray tracing implementation, certain BVH subtrees may be discretized to reduce storage and bandwidth requirements. One embodiment of a dequantize instruction takes the form of dequantize dest src0 src1 src2, where source register src0 stores N unsigned bytes, source register src1 stores 1 unsigned byte, source register src2 stores 1 floating point value, and destination register dest stores N floating point values. All of these registers may be vector registers 4515. Alternatively, src0 and dest may be vector registers 4515, and src1 and src2 may be scalar registers 4516.

[0423] The following code sequence defines a specific implementation of the De-Discretize instruction:

[0424] for(int i=0; i <SIMD_WIDTH){

[0425] if (execMask[i]) {

[0426] dst[i]=src2[i]+ldexp(convert_to_float(src0[i]),src1);

[0427] }

[0428] }

[0429] In this example, ldexp multiplies a double-precision floating-point value by a specified integer power of 2 (ie, ldexp(x, exp) = x*2 exp In the above code, if the execution mask value (execMask[i]) associated with the current SIMD data element is set to 1, the SIMD data element at position i in src0 is converted to a floating point value and multiplied by an integer power (2 src1值 ) and that value is added to the corresponding SIMD data element in src2.

[0430] Selective minimum or maximum value

[0431] One embodiment of the selective minimum or maximum instruction performs a per-channel minimum or maximum operation (i.e., returns the minimum or maximum of a set of values), as indicated by the bits in a bitmask. The bitmask can utilize vector register 4515, scalar register 4516, or a set of separate mask registers (not shown). The following code sequence defines a particular implementation of the minimum / maximum instruction: sel_min_max dest src0 src1 src2, where src0 stores N doublewords, src1 stores N doublewords, src2 stores one doubleword, and the destination register stores N doublewords.

[0432] The following code sequence defines a particular implementation of the selective minimum / maximum instruction:

[0433] for(int i = 0; i < SIMD_WIDTH){

[0434] if(execMask[i]){

[0435] dst[i] = (1 << i) & src2? min(src0[i], src1[i]) : max(src0[i], src1[i]);

[0436] }

[0437] }

[0438] In this example, the value of (1 << i) & src2 (the bitwise AND of 1 shifted left by i and src2) is used to select the minimum of the ith data element in src0 and src1 or the maximum of the ith data element in src0 and src1. The operation is performed on the ith data element only if the execution mask value (execMask[i]) associated with the current SIMD data element is set to 1.

[0439] Shuffle Index Instructions

[0440] One embodiment of the shuffle index instruction can copy any set of input channels to output channels. For a SIMD width of 32, this instruction can be executed with lower throughput. This embodiment takes the following form: shuffle_index dest src0 src1 <optional flags>, where src0 stores N doublewords, src1 stores N unsigned bytes (i.e., index values), and dest stores N doublewords.

[0441] The following code sequence defines a particular implementation of the shuffle index instruction:

[0442] for(int i = 0; i < SIMD_WIDTH){

[0443] uint8_t srcLane=src1.index[i];

[0444] if (execMask[i]) {

[0445] bool invalidLane=srcLane<0||srcLane>=SIMD_WIDTH||! execMask[srcLaneMod];

[0446] if(FLAG){

[0447] invalidLane|=flag[srcLaneMod];

[0448] }

[0449] if(invalidLane){

[0450] dst[i] = src0[i];

[0451] }

[0452] else{

[0453] dst[i] = src0[srcLane];

[0454] }

[0455] }

[0456] }

[0457] In the above code, the index in src1 identifies the current lane. If the i-th value in the execution mask is set to 1, a check is performed to ensure that the source lane is in the range of 0 to the SIMD width. If so, a flag is set (srcLaneMod) and data element i of the destination is set equal to data element i of src0. If the lane is in range (i.e., valid), the index value from src1 (srcLane0) is used as an index into src0 (dst[i] = src0[srcLane]).

[0458] Immediate shuffle up / down / XOR instructions

[0459] In one embodiment, the immediate shuffle instruction shuffles the input data elements / channels based on the immediate value of the instruction. In one implementation, the immediate value can specify that the input channel is moved by 1, 2, 4, 8, or 16 positions based on the value of the immediate value. Alternatively, an additional scalar source register can be specified as a fill value. When the source channel index is invalid, the fill value (if provided) is stored in the data element position in the destination. If the fill value is not provided, the data element position is set to all 0s.

[0460] In one embodiment, a flag register is used as a source mask. If the flag bit of a source lane is set to 1, the source lane is marked as invalid and the instruction continues.

[0461] The following are examples of different implementations of the immediate shuffle instruction:

[0462] shuffle_ <up dn xor>_<1 / 2 / 4 / 8 / 16>dest src0<optional src1><optional flag>

[0463] shuffle_ <up dn xor>_<1 / 2 / 4 / 8 / 16>dest src0<optional src1><optional flags>

[0464] In this implementation, src0 stores N double words, src1 stores one double word for the fill value (if any), and dest stores N double words including the result.

[0465] The following code sequence defines a specific implementation of the immediate shuffle instruction:

[0466] for(int i=0; i <SIMD_WIDTH){

[0467] int8_t srcLane;

[0468] switch(SHUFFLE_TYPE){

[0469] case UP:

[0470] srcLane = i-SHIFT;

[0471] case DN:

[0472] srcLane = i + SHIFT;

[0473] case XOR:

[0474] srcLane = i^SHIFT;

[0475] }

[0476] if (execMask[i]) {

[0477] bool invalidLane=srcLane<0||srcLane>=SIMD_WIDTH||! execMask[srcLane];

[0478] if(FLAG){

[0479] invalidLane|=flag[srcLane];

[0480] }

[0481] if(invalidLane){

[0482] if(SRC1)

[0483] dst[i] = src1;

[0484] else

[0485] dst[i] = 0;

[0486] }

[0487] else{

[0488] dst[i] = src0[srcLane];

[0489] }

[0490] }

[0491] }

[0492] Here the input data element / channel is shifted 1, 2, 4, 8 or 16 positions based on the value of the immediate. Register src1 is an additional scalar source register that is used as a fill value for the data element position stored in the destination when the source channel index is invalid. If no fill value is provided and the source channel index is invalid, the data element position in the destination is set to 0. The flag register (FLAG) is used as a source mask. If the flag bit of the source channel is set to 1, the source channel is marked as invalid and the instruction continues as described above.

[0493] Indirect shuffle up / down / XOR instructions

[0494] The indirect shuffle instruction has a source operand (src1) that controls the mapping from the source lane to the destination lane. One embodiment of the indirect shuffle instruction takes the following form:

[0495] shuffle_ <up dn xor>dest src0 src1 <optional flags>

[0496] Where src0 stores N double words, src1 stores 1 double word, and dest stores N double words.

[0497] The following code sequence defines a specific implementation of the indirect shuffle instruction:

[0498] for(int i=0; i <SIMD_WIDTH){

[0499] int8_t srcLane;

[0500] switch(SHUFFLE_TYPE){

[0501] case UP:

[0502] srcLane = i-src1;

[0503] case DN:

[0504] srcLane = i + src1;

[0505] case XOR:

[0506] srcLane = i^src1;

[0507] }

[0508] if (execMask[i]) {

[0509] bool invalidLane=srcLane<0||srcLane>=SIMD_WIDTH||! execMask[srcLane];

[0510] if(FLAG){

[0511] invalidLane|=flag[srcLane];

[0512] }

[0513] if(invalidLane){

[0514] dst[i] = 0;

[0515] }

[0516]

[0517] Thus, the indirect shuffle instruction operates in a similar manner to the immediate shuffle instruction described above, but the mapping of source lanes to destination lanes is controlled by the source register src1 rather than an immediate value.

[0518] Cross-channel min / max instructions

[0519] Embodiments of cross-lane minimum / maximum instructions are supported for floating point and integer data types. The cross-lane minimum instruction takes the form of lane_min dest src0, and the cross-lane maximum instruction takes the form of lane_max destsrc0, where src0 stores N double words and dest stores 1 double word.

[0520] As an example, the following code sequence defines a specific implementation of cross-channel minima:

[0521]

[0522] In this embodiment, the doubleword value in data element position i of the source register is compared with the data element in the destination register and the minimum of the two values ​​is copied to the destination register. The cross lane maximum instruction operates in essentially the same manner, with the only difference being that the maximum of the data element in position i and the destination value is selected.

[0523] Cross-channel min / max indexing instructions

[0524] An embodiment of a cross-lane minimum index instruction takes the form of lane_min_index dest src0, and a cross-lane maximum index instruction takes the form of lane_max_index dest src0, where src0 stores N double words and dest stores 1 double word.

[0525] As an example, the following code sequence defines a specific implementation of the cross-lane minimum index instruction:

[0526]

[0527] In this embodiment, the destination index increases from 0 to the SIMD width, across the destination register.If the execute mask bit is set, the data element at position i in the source register is copied to a temporary storage location (tmp) and the destination index is set to data element position i.

[0528] Cross-channel sorting network instructions

[0529] In one embodiment, the cross-channel sorting network instruction sorts all N input elements in ascending order (sortnet_min) or in descending order (sortnet_max) using an N-wide (stable) sorting network. The min / max versions of the instruction take the form of sortnet_min dest src0 and sortnet_max dest src0, respectively. In one implementation, src0 and dest store N double words. A min / max sort is performed on the N double words of src0, and the elements in ascending order (for minimum values) or descending order (for maximum values) are stored in dest in their corresponding sorted order. An example of a code sequence defining the instruction is: dst = apply_N_wide_sorting_network_min / max(src0).

[0530] Cross-channel sorting network indexing instructions

[0531] In one embodiment, the cross-lane sorting network index instruction sorts all N input elements using an N-wide (stable) sorting network in ascending order (sortnet_min) or in descending order (sortnet_max), but returns a permuted index. The min / max version of the instruction takes the form of sortnet_min_index dest src0 and sortnet_max_index dest src0, where src0 and dest each store N double words. An example of a code sequence defining this instruction is dst = apply_N_wide_sorting_network_min / max_index(src0).

[0532] exist Fig.46 An embodiment of a method for executing any of the above instructions is illustrated in . The method can be implemented on the specific processor architecture described above, but is not limited to any specific processor or system architecture.

[0533] At 4601, instructions of the main graphics thread are executed on a processor core. This may include, for example, any of the cores described above (e.g., graphics core 3130). When it is determined at 4602 that ray tracing work has arrived within the main graphics thread, the ray tracing instructions are offloaded to the ray tracing execution circuitry, which may execute instructions such as described above with respect to Fig.45 The form of the functional unit (FU) described or can be in relation to Fig.31 Described in the dedicated ray tracing core 3150.

[0534] At 4603, the ray tracing instructions are fetched from memory and decoded into executable operations (in embodiments where a decoder is required) at 4603. At 4604, the ray tracing instructions are scheduled and dispatched for execution by the ray tracing circuitry. At 4605, the ray tracing instructions are executed by the ray tracing circuitry. For example, the instructions may be dispatched to and executed on the FUs described above (e.g., vector FU 4510, ray tracing FU 4512, etc.) and / or graphics core 3130 or ray tracing core 3150.

[0535] When execution of the ray tracing instructions is complete, the results are stored at 4606 (e.g., stored back to memory 3198) and the main graphics thread is notified at 4607. At 4608, the ray tracing results are processed within the context of the main thread (e.g., read from memory and integrated into the graphics rendering results).

[0536] In embodiments, the term "engine" or "module" or "logic" may refer to, be part of, or include an application specific integrated circuit ("ASIC"), electronic circuit, processor (shared, dedicated, or group) and / or memory (shared, dedicated, or group) that executes one or more software or firmware programs, combinational logic circuits, and / or other suitable components that provide the described functionality. In embodiments, the engine, module, or logic may be implemented in firmware, hardware, software, or any combination of firmware, hardware, and software.

[0537] Embodiments of the present invention may include the steps described above. These steps may be embodied as machine executable instructions, which may be used to cause a general or special processor to perform these steps. Alternatively, these steps may be performed by specific hardware components containing hard-wired logic for performing these steps, or by any combination of programmed computer components and custom hardware components.

[0538] As described herein, instructions may refer to a specific configuration of hardware, such as an application-specific integrated circuit (ASIC), which is configured to perform certain operations or have predetermined functions or software instructions stored in a memory embodied as a non-transitory computer-readable medium. Thus, the techniques shown in the accompanying drawings may be implemented using code and data stored and executed on one or more electronic devices (e.g., end stations, network elements, etc.). Such electronic devices use computer machine-readable media (internally and / or with other electronic devices via a network) to store and communicate code and data, such as non-transitory computer machine-readable storage media (e.g., magnetic disks; optical disks; random access memory; read-only memory; flash memory devices; phase change memory) and transient computer machine-readable communication media (e.g., electrical, optical, acoustic or other forms of propagated signals - such as carrier waves, infrared signals, digital signals, etc.).

[0539] In addition, such electronic devices typically include a set of one or more processors coupled to one or more other components, such as one or more storage devices (non-transitory machine-readable storage media), user input / output devices (e.g., keyboards, touch screens, and / or displays), and network connections. The coupling of the set of processors and other components is typically through one or more buses and bridges (also referred to as bus controllers). The storage devices and signals that carry network traffic represent one or more machine-readable storage media and machine-readable communication media, respectively. Therefore, the storage device of a given electronic device typically stores code and / or data for execution on the set of one or more processors of the electronic device. Of course, different combinations of software, firmware, and / or hardware can be used to implement one or more parts of an embodiment of the present invention. Throughout this specific embodiment, for the purpose of explanation, numerous specific details are set forth in order to provide a thorough understanding of the present invention. However, it will be apparent to those skilled in the art that the present invention can be practiced without some of these specific details. In some instances, well-known structures and functions are not described in detail to avoid obscuring the subject matter of the present invention. Therefore, the scope and spirit of the present invention should be determined according to the appended claims.< / up> < / up> < / up> < / ray>

Claims

1. A graphics processing unit, comprising: Multiple multi-core groups, where a multi-core group includes: Multiple graphics cores; Multiple Tensor Cores for performing matrix operations; a plurality of ray tracing cores for performing ray tracing operations to protect the plurality of graphics cores from being overloaded, wherein the ray tracing cores include specialized circuitry for performing a first set of bounding box tests and specialized circuitry for performing a second set of ray-triangle intersection tests, and wherein the ray tracing cores independently perform all computations for bounding box tests and ray traversal and ray intersection; and A set of register files, used to store operand values; and wherein execution circuitry of at least one of the graphics core, the tensor core, and the ray tracing core is configured to execute a first instruction to select a minimum value from a plurality of threads, the first instruction comprising a first operand specifying a value associated with the plurality of threads, the execution being configured to perform the following operations: A minimum value is returned from values ​​associated with a set of threads, the set of threads being selected from the plurality of threads based on a mask, the mask comprising a bit associated with each thread of the plurality of threads indicating whether the corresponding thread is included in the set of threads.

2. The graphics processing unit according to claim 1, wherein: The first bit value indicates that the corresponding thread is included in the set of threads, and the second bit value indicates that the thread is not included in the set of threads.

3. The graphics processing unit according to claim 1 or 2, wherein: Each of the values ​​associated with the plurality of threads comprises an integer.

4. The graphics processing unit according to claim 1 or 2, wherein: The collection of threads is synchronized.

5. The graphics processing unit according to claim 1 or 2, wherein: The execution circuitry of at least one of the graphics core, the tensor core, and the ray tracing core is to execute a second instruction including a second operand specifying a value associated with the plurality of threads to: A maximum value is returned from among the values ​​associated with the set of threads, the set of threads being selected from the plurality of threads based on the mask.

6. The graphics processing unit according to claim 1 or 2, wherein: Executing the ray tracing operation includes: Generate rays to traverse through the graphics scene; constructing a hierarchical acceleration data structure including a plurality of nodes arranged in a hierarchical structure; and One or more of the rays are traversed through the hierarchical acceleration data structure and intersected with primitives contained within the hierarchically arranged nodes.

7. A graphics processing unit, comprising: Multiple multi-core groups, where a multi-core group includes: Multiple graphics cores; Multiple Tensor Cores for performing matrix operations; a plurality of ray tracing cores for performing ray tracing operations to protect the plurality of graphics cores from being overloaded, wherein the ray tracing cores include specialized circuitry for performing a first set of bounding box tests and specialized circuitry for performing a second set of ray-triangle intersection tests, and wherein the ray tracing cores independently perform all computations for bounding box tests and ray traversal and ray intersection; and A set of register files, used to store operand values; and wherein execution circuitry of at least one of the graphics core, the tensor core, and the ray tracing core is configured to execute a first instruction to select a maximum value from a plurality of threads, the first instruction comprising a first operand specifying a value associated with the plurality of threads, the execution being configured to perform the following operations: A maximum value is returned from values ​​associated with a set of threads, the set of threads being selected from the plurality of threads based on a mask, the mask comprising a bit associated with each thread of the plurality of threads indicating whether the corresponding thread is included in the set of threads.

8. The graphics processing unit of claim 7, wherein: The first bit value indicates that the corresponding thread is included in the set of threads, and the second bit value indicates that the thread is not included in the set of threads.

9. The graphics processing unit according to claim 7 or 8, wherein: Each of the values ​​associated with the plurality of threads comprises an integer.

10. The graphics processing unit according to claim 7 or 8, wherein: The collection of threads is synchronized.

11. The graphics processing unit according to claim 7 or 8, wherein: The execution circuitry of at least one of the graphics core, the tensor core, and the ray tracing core is to execute a second instruction including a second operand specifying a value associated with the plurality of threads to: A minimum value is returned from among the values ​​associated with the set of threads, the set of threads being selected from the plurality of threads based on the mask.

12. The graphics processing unit according to claim 7 or 8, wherein: Executing the ray tracing operation includes: Generate rays to traverse through the graphics scene; constructing a hierarchical acceleration data structure including a plurality of nodes arranged in a hierarchical structure; and One or more of the rays are traversed through the hierarchical acceleration data structure and intersected with primitives contained within the hierarchically arranged nodes.

13. A method for instruction execution, comprising: Instructions for scheduling a plurality of threads to be executed by a processor, wherein the processor comprises: Multiple graphics cores; Multiple Tensor Cores for performing matrix operations; a plurality of ray tracing cores for performing ray tracing operations to protect the plurality of graphics cores from being overloaded, wherein the ray tracing cores include specialized circuitry for performing a first set of bounding box tests and specialized circuitry for performing a second set of ray-triangle intersection tests, and wherein the ray tracing cores independently perform all computations for bounding box tests and ray traversal and ray intersection; and executing a first instruction to select a minimum value from the plurality of threads, the first instruction comprising a first operand specifying a value associated with the plurality of threads, wherein executing the first instruction comprises: A minimum value is returned from values ​​associated with a set of threads, the set of threads being selected from the plurality of threads based on a mask, the mask comprising a bit associated with each thread of the plurality of threads indicating whether the corresponding thread is included in the set of threads.

14. The method of claim 13, wherein: The first bit value indicates that the corresponding thread is included in the set of threads, and the second bit value indicates that the thread is not included in the set of threads.

15. The method according to claim 13 or 14, wherein: Each of the values ​​associated with the plurality of threads comprises an integer.

16. The method according to claim 13 or 14, wherein: The collection of threads is synchronized.

17. The method according to claim 13 or 14, wherein: The method further comprises: executing a second instruction, the second instruction comprising a second operand specifying a value associated with the plurality of threads, wherein executing the second instruction comprises: A maximum value is returned from among the values ​​associated with the set of threads, the set of threads being selected from the plurality of threads based on the mask.

18. The method of claim 13 or 14, wherein: The method further comprises: Generate rays to traverse through the graphics scene; constructing a hierarchical acceleration data structure including a plurality of nodes arranged in a hierarchical structure; and One or more of the rays are traversed through the hierarchical acceleration data structure and the one or more of the rays are intersected with primitives contained within the hierarchically arranged nodes.

19. A method for instruction execution, comprising: Instructions for scheduling a plurality of threads to be executed by a processor, wherein the processor comprises: Multiple graphics cores; Multiple Tensor Cores for performing matrix operations; a plurality of ray tracing cores for performing ray tracing operations to protect the plurality of graphics cores from being overloaded, wherein the ray tracing cores include specialized circuitry for performing a first set of bounding box tests and specialized circuitry for performing a second set of ray-triangle intersection tests, and wherein the ray tracing cores independently perform all computations for bounding box tests and ray traversal and ray intersection; and Executing a first instruction to select a maximum value from the plurality of threads, the first instruction comprising a first operand specifying a value associated with the plurality of threads, wherein executing the first instruction comprises: A maximum value is returned from values ​​associated with a set of threads, the set of threads being selected from the plurality of threads based on a mask, the mask comprising a bit associated with each thread of the plurality of threads indicating whether the corresponding thread is included in the set of threads.

20. The method of claim 19, wherein: The first bit value indicates that the corresponding thread is included in the set of threads, and the second bit value indicates that the thread is not included in the set of threads.

21. The method of claim 19 or 20, wherein: Each of the values ​​associated with the plurality of threads comprises an integer.

22. The method of claim 19 or 20, wherein: The collection of threads is synchronized.

23. The method of claim 19 or 20, further comprising: executing a second instruction, the second instruction comprising a second operand specifying a value associated with the plurality of threads, wherein executing the second instruction comprises: A minimum value is returned from among the values ​​associated with the set of threads, the set of threads being selected from the plurality of threads based on the mask.

24. The method of claim 19 or 20, further comprising: Generate rays to traverse through the graphics scene; Constructing a hierarchical acceleration data structure including a plurality of nodes arranged in a hierarchical structure; as well as One or more of the rays are traversed through the hierarchical acceleration data structure and the one or more of the rays are intersected with primitives contained within the hierarchically arranged nodes.

25. A machine-readable medium storing code which, when executed, causes a machine to perform the method of any one of claims 13 to 24.

26. A system for instruction execution, comprising means for executing the method according to any one of claims 13 to 24.