Adding greater sense of reality to computer-generated images by smoothing sawtooth edges

Abstract

The invention relates to adding a greater sense of reality to a computer-generated image by smoothing a jagged edge. Techniques are disclosed for adding a greater sense of reality to a computer-generated image by smoothing sawtooth edges. During rendering of an image, a particular pixel in the image is identified, where anti-aliasing will be helpful. Anti-aliasing is then performed on these identified pixels, wherein anti-aliasing is a technique for adding a greater sense of reality to the digital image by smoothing the aliasing edge. This reduces the cost of performing anti-aliasing by reducing the number of pixels in the image on which anti-aliasing is performed.

Description

[0001] Divisional Application Information

[0002] This invention patent application is a divisional application of the invention patent application with application date of April 12, 2019, application number 201910292877.5, and invention name “Adding greater realism to computer-generated images by smoothing jagged edges”.

[0003] Claiming priority

[0004] This application claims the benefit of U.S. Provisional Application No. 62 / 656,951, filed April 12, 2018, entitled “Adaptive Temporal and Hybrid Deferred Anti-Aliasing for GPU Ray-Tracing” (Case No. 17-MU-0287US01), the entire contents of which are incorporated herein by reference. In addition, this application claims the benefit of U.S. Provisional Application No. 62 / 659,620, filed April 18, 2018, entitled “ADAPTIVE RAY TRACING FOR TEMPORAL ANTIALIASING” (Case No. NVIDP1221+ / 18DU0112US01), the entire contents of which are incorporated herein by reference. Technical Field

[0005] The present invention relates to image rendering and, more particularly, to performing anti-aliasing during image rendering. Background Art

[0006] Anti-aliasing is often used during scene rendering to remove image artifacts caused by insufficient sampling rates. However, current methods for performing anti-aliasing incur high storage costs and may suffer from high bandwidth usage in some cases. Furthermore, current techniques for performing anti-aliased deferred rendering of complex geometry using graphics hardware can be complex and may require several rendering passes for a scene. BRIEF DESCRIPTION OF THE DRAWINGS

[0007] Figure 1 A flow chart of a method for performing complex pixel identification to improve anti-aliasing of computer-generated images is shown according to one embodiment.

[0008] Figure 2 A parallel processing unit according to an embodiment is shown.

[0009] Figure 3AAccording to one embodiment, Figure 2 A general-purpose processing cluster within a parallel processing unit.

[0010] Figure 3B According to one embodiment, Figure 2 A memory partitioning unit for parallel processing units.

[0011] Figure 4A According to one embodiment, Figure 3A Stream multiprocessor.

[0012] Figure 4B According to one embodiment, the use Figure 2 Conceptual diagram of the processing system implemented by the PPU in FIG.

[0013] Figure 4C An exemplary system is shown in which various architectures and / or functions of the foregoing embodiments may be implemented.

[0014] Figure 5 According to an embodiment, Figure 2 Conceptual diagram of the graphics processing pipeline implemented by the PPU.

[0015] Figure 6 A flow chart of a method for performing hybrid deferred ray tracing anti-aliasing is shown according to one embodiment.

[0016] Figure 7 A flow chart for performing an adaptive ray tracing method for temporal anti-aliasing is shown according to one embodiment.

[0017] Figure 8 An exemplary adaptive temporal anti-aliasing algorithm in the context of the DXR ray tracing API is shown, according to one embodiment. Detailed Description

[0019] In computer graphics, anti-aliasing is a technique for adding realism to digital images by smoothing jagged edges, such as on curves and diagonal lines. These jagged edges may appear due to a low sampling rate during image rendering. However, performing anti-aliasing requires a lot of processing resources and time. In response, specific pixels in the digital image for which anti-aliasing is helpful are explicitly identified, and then anti-aliasing is performed on these identified pixels. By reducing the number of pixels in the image for which anti-aliasing is performed, the cost of performing anti-aliasing can be reduced.

[0020] Figure 1A flow chart of a method 100 for performing complex pixel identification to improve anti-aliasing of computer-generated images according to one embodiment is shown. Although the method 100 is described in the context of a processing unit, the method 100 may also be performed by a program, a custom circuit, or a combination of a custom circuit and a program. For example, the method 100 may be performed by a GPU (graphics processing unit), a CPU (central processing unit), or any processor capable of performing parallel path space filtering by hashing. Further, one of ordinary skill in the art will appreciate that any system that performs the method 100 is within the scope and spirit of the embodiments of the present invention.

[0021] As shown in operation 102, complex pixels in a scene to be rendered are identified. In one embodiment, each of the complex pixels may include a pixel that contains one or more discontinuities in pixel attributes (e.g., coverage, depth, material, etc.) that cause aliasing. In another embodiment, each of the complex pixels may include a visible pixel that is partially covered by one or more instances of geometry in the scene (e.g., one or more polygons, etc.).

[0022] Additionally, in one embodiment, conservative rasterization may be used to identify complex pixels. For example, conservative rasterization may identify intersections of one or more instances of geometry with any portion of a pixel in a scene, even if the geometry does not hit the center of the pixel or any samples within the pixel (if multiple samples are used). In another example, conservative rasterization may identify one or more non-zero intersections between instances of geometry and a pixel, and may rasterize the pixel.

[0023] Further, in one embodiment, conservative rasterization can be implemented through multiple passes of one or more instances of geometry in the scene. For example, the first pass of conservative rasterization can include a depth pre-pass. For example, the depth pre-pass can create a depth buffer that stores depth for the nearest fully covered pixel.

[0024] Further, in one embodiment, the second pass of conservative rasterization may include a pixel identification pass. For example, the pixel identification pass may identify and mark complex pixels (e.g., visible pixels in a scene that are partially covered by one or more instances of geometry). In another example, the pixels may be marked using an implemented pixel shader. In another example, the pixel identification pass may also increment a counter.

[0025] In addition, in one embodiment, the pixel identification channel can also count the number of instances of geometric figures that touch the pixel for each pixel. This can be stored as a raw count of the pixel. In another embodiment, the pixel identification channel can result in a two-dimensional surface containing non-zero values ​​for complex pixels, where the non-zero value can indicate the number of primitives potentially visible in the pixel.

[0026] In this way, during anti-aliasing, conservative rasterization can be used to identify pixels in the scene that deserve a higher sampling rate (eg, partially covered pixels, etc.).

[0027] In addition, in one embodiment, conservative rasterization may include internal conservative rasterization. For example, internal conservative rasterization may analyze one or more instances of geometry in a scene and may indicate, for each pixel in the scene, whether the one or more instances of geometry cover at least a portion of the pixel and whether the pixel is completely covered. In another example, complete coverage of a pixel may be indicated by a flag associated with the pixel. For example, a binary flag value of 0 may indicate that the associated pixel is partially covered, while a binary flag value of 1 may indicate that the associated pixel is completely covered.

[0028] Additionally, as shown in operation 104, anti-aliasing is performed on each complex pixel in the scene using ray tracing. In one embodiment, performing anti-aliasing may include performing ray tracing on the complex pixels in the scene. For example, performing anti-aliasing may include determining point sampled visibility of sub-pixel regions within each complex pixel using GPU ray tracing. This may include sampling regions of complex pixels via ray tracing and approximating the geometric coverage of the complex pixels.

[0029] For example, based on the raw count of each complex pixel, rays can be distributed over the complex pixels. In another example, more rays can be traced through complex pixels with larger pixel counts than complex pixels with smaller pixel counts. This can be performed using a ray tracing framework / engine.

[0030] Still further, in one embodiment, performing anti-aliasing may include, for each complex pixel, analytically solving the geometric surface of the complex pixel and its coverage. For example, performing anti-aliasing may include storing additional data about the geometry that intersects the complex pixel. For example, the additional data may include one or more edge equations. In another example, performing anti-aliasing may include using the additional data to calculate edge / ray intersections. This may be performed by the GPU without using a ray tracing framework / engine.

[0031] Furthermore, in one embodiment, performing anti-aliasing may include analytically evaluating visibility. For example, the analyzed visibility may indicate how much area of ​​a complex pixel is occupied by a particular primitive (eg, by solving a surface integral, etc.).

[0032] In this way, during anti-aliasing, pixels in the scene that merit a higher sampling rate (such as partially covered pixels, etc.) can be identified. Thus, during anti-aliasing, sampling costs can be reduced by focusing sampling on pixels with primitives that only partially cover the pixel. Additionally, during anti-aliasing, ray tracing can be used to adjust the number of samples sent to each pixel on a per-pixel basis.

[0033] In another embodiment, a parallel processing unit (PPU) is used (e.g. Figure 2 The PPU 200 shown in FIG. 1 performs anti-aliasing on each complex pixel in the scene.

[0034] Now, more illustrative information about various optional architectures and features that can be used to implement the above framework will be described according to the needs of the user. It should be noted that the following information is described for illustrative purposes and should not be interpreted as limiting in any way. Any of the following features can be optionally combined with other features described or do not exclude other features described.

[0035] Parallel processing architecture

[0036] Figure 2 A parallel processing unit (PPU) 200 according to one embodiment is shown. In one embodiment, the PPU 200 is a multi-threaded processor implemented on one or more integrated circuit devices. The PPU 200 is a latency-hiding architecture designed for processing multiple threads in parallel. A thread (i.e., an execution thread) is an instantiation of a set of instructions configured to be executed by the PPU 200. In one embodiment, the PPU 200 is a graphics processing unit (GPU) configured to implement a graphics rendering pipeline for processing three-dimensional (3D) graphics data to generate two-dimensional (2D) image data for display on a display device such as a liquid crystal display (LCD) device. In other embodiments, the PPU 200 can be used to perform general-purpose calculations. Although an example parallel processor is provided herein for illustrative purposes, it should be strongly noted that such a processor is described for illustrative purposes only, and any processor can be used to supplement and / or replace the same processor.

[0037] One or more PPUs 200 can be configured to accelerate thousands of high-performance computing (HPC), data center, and machine learning applications. PPU 200 can be configured to accelerate multiple deep learning systems and applications, including autonomous vehicle platforms, deep learning, high-precision speech, image, and text recognition systems, intelligent video analysis, molecular simulation, drug discovery, disease diagnosis, weather forecasting, big data analysis, astronomy, molecular dynamics simulation, financial modeling, robotics, factory automation, real-time language translation, online search optimization, and personalized user recommendations, etc.

[0038] like Figure 2 As shown, the PPU 200 includes an input / output (I / O) unit 205, a front-end unit 215, a scheduler unit 220, a work distribution unit 225, a hub 230, a crossbar switch (Xbar) 270, one or more general processing clusters (GPCs) 250, and one or more partition units 280. The PPU 200 can be connected to a host processor or other PPUs 200 via one or more high-speed NVLink 210 interconnects. The PPU 200 can be connected to a host processor or other peripheral devices via an interconnect 202. The PPU 200 can also be connected to a local memory including a plurality of memory devices 204. In one embodiment, the local memory can include several dynamic random access memory (DRAM) devices. The DRAM device can be configured as a high bandwidth memory (HBM) subsystem, with multiple DRAM dies stacked in each device.

[0039] The NVLink 210 interconnect enables the system to scale and include one or more PPUs 200 combined with one or more CPUs, supporting cache coherency between the PPU 200 and the CPU, and CPU mastering. Data and / or commands can be sent by the NVLink 210 through the hub 230 to or from other units of the PPU 200, such as one or more copy engines, video encoders, video decoders, power management units, etc. (not explicitly shown). Figure 4B NVLink 210 is described in more detail.

[0040] The I / O unit 205 is configured to send and receive communications (i.e., commands, data, etc.) from a host processor (not shown) via the interconnect 202. The I / O unit 205 may communicate with the host processor directly via the interconnect 202, or through one or more intermediate devices (e.g., a memory bridge). In one embodiment, the I / O unit 205 may communicate with one or more other processors (such as one or more PPUs 200) via the interconnect 202. In one embodiment, the I / O unit 205 implements a peripheral component interconnect express (PCIe) interface for communicating over a PCIe bus, and the interconnect 202 is a PCIe bus. In alternative embodiments, the I / O unit 205 may implement other types of known interfaces for communicating with external devices.

[0041] The I / O unit 205 decodes packets received via the interconnect 202. In one embodiment, the packets represent commands configured to cause the PPU 200 to perform various operations. The I / O unit 205 sends the decoded commands to various other units of the PPU 200 as specified by the commands. For example, some commands may be sent to the front end unit 215. Other commands may be sent to the hub 230 or other units of the PPU 200, such as one or more replication engines, video encoders, video decoders, power management units, etc. (not explicitly shown). In other words, the I / O unit 205 is configured to route communications between and among the various logical units of the PPU 200.

[0042] In one embodiment, a program executed by a host processor encodes a command stream in a buffer that provides a workload to the PPU 200 for processing. The workload may include several instructions and data to be processed by these instructions. A buffer is an area in memory that is accessible (i.e., read / write) by both the host processor and the PPU 200. For example, the I / O unit 205 may be configured to access a buffer in a system memory connected to the interconnect 202 via a memory request transmitted on the interconnect 202. In one embodiment, the host processor writes a command stream to the buffer and then sends a pointer to the starting point of the command stream to the PPU 200. The front end unit 215 receives a pointer to one or more command streams. The front end unit 215 manages one or more streams, reads commands from the streams, and forwards the commands to the various units of the PPU 200.

[0043] The front end unit 215 is coupled to a scheduler unit 220, which configures each GPC 250 to process tasks defined by one or more streams. The scheduler unit 220 is configured to track state information related to the various tasks managed by the scheduler unit 220. The state may indicate which GPC 250 the task is assigned to, whether the task is active or inactive, a priority associated with the task, etc. The scheduler unit 220 manages the execution of multiple tasks on one or more GPCs 250.

[0044] The scheduler unit 220 is coupled to a work distribution unit 225, which is configured to dispatch tasks for execution on the GPCs 250. The work distribution unit 225 may track a plurality of scheduled tasks received from the scheduler unit 220. In one embodiment, the work distribution unit 225 manages a pending task pool and an active task pool for each GPC 250. The pending task pool may include a number of time slots (e.g., 32 time slots) containing tasks assigned to be processed by a particular GPC 250. The active task pool may include a number of time slots (e.g., 4 time slots) for tasks being actively processed by the GPC 250. When a GPC 250 completes execution of a task, the task is evicted from the active task pool of the GPC 250, and one of the other tasks in the pending task pool is selected and scheduled for execution on the GPC 250. If an active task on a GPC 250 has become idle, such as while waiting for data dependencies to be resolved, then the active task may be evicted from the GPC 250 and returned to the pending task pool while another task in the pending task pool is selected and scheduled for execution on the GPC 250 .

[0045] Work distribution unit 225 communicates with one or more GPCs 250 via XBar 270. XBar 270 is an interconnect network that couples many units of PPU 200 to other units of PPU 200. For example, XBar 270 can be configured to couple work distribution unit 225 to a particular GPC 250. Although not explicitly shown, one or more other units of PPU 200 can also be connected to XBar 270 via hub 230.

[0046] Tasks are managed by the scheduler unit 220 and dispatched to the GPCs 250 by the work distribution unit 225. The GPCs 250 are configured to process tasks and generate results. The results can be used by other tasks in the GPC 250, routed to different GPCs 250 via the XBar 270, or stored in the memory 204. The results can be written to the memory 204 via the partition unit 280, which implements a memory interface for reading data from / writing data to the memory 204. The results can be sent to another PPU 200 or CPU via NVLink 210. In one embodiment, the PPU 200 includes a number U of partition units 280, which is equal to the number of independent and different storage devices 204 coupled to the PPU 200. The following will be combined with Figure 3B The partition unit 280 is described in more detail.

[0047] In one embodiment, the host processor executes a driver kernel that implements an application programming interface (API) that enables one or more applications executing on the host processor to schedule operations for execution on the PPU 200. In one embodiment, multiple computing applications are executed simultaneously by the PPU 200, and the PPU 200 provides isolation, quality of service (QoS), and independent address spaces for the multiple computing applications. The application may generate instructions (i.e., API calls) that cause the driver kernel to generate one or more tasks for execution by the PPU 200. The driver kernel outputs the tasks to one or more streams processed by the PPU 200. Each task may include one or more related groups of threads, referred to herein as warps. In one embodiment, a warp includes 32 related threads that can be executed in parallel. Collaborative threads may refer to multiple threads, including instructions to perform tasks and instructions that can exchange data through shared memory. In combination Figure 4A Threads and cooperative threads are described in more detail.

[0048] Figure 3A According to an embodiment, Figure 2 The PPU 200 and GPC 250. Figure 3A As shown, each GPC 250 includes several hardware units for processing tasks. In one embodiment, each GPC 250 includes a pipeline manager 310, a pre-raster operation unit (PROP) 315, a raster engine 325, a work distribution crossbar switch (WDX) 380, a memory management unit (MMU) 390 and one or more data processing clusters (DPCs) 320. It can be understood that Figure 3A The GPC 250 may include other hardware units to replace or supplement Figure 3A The unit shown in .

[0049] In one embodiment, the operation of the GPC 250 is controlled by a pipeline manager 310. The pipeline manager 310 manages the configuration of one or more DPCs 320 for processing tasks assigned to the GPC 250. In one embodiment, the pipeline manager 310 may configure at least one of the one or more DPCs 320 to implement at least a portion of a graphics rendering pipeline. For example, a DPC 320 may be configured to execute a vertex shading program on a programmable streaming multiprocessor (SM) 340. The pipeline manager 310 may also be configured to route packets received from the work distribution unit 225 to appropriate logic units in the GPC 250. For example, some packets may be routed to fixed-function hardware units in the PROP 315 and / or the raster engine 325, while other packets may be routed to the DPC 320 to be processed by the primitive engine 335 or the SM 340. In one embodiment, the pipeline manager 310 may configure at least one of the one or more DPCs 320 to implement a neural network model and / or a computational pipeline.

[0050] PROP unit 315 is configured to route data generated by raster engine 325 and DPC 320 to a raster operations (ROP) unit, which in conjunction with Figure 3B The PROP unit 315 may also be configured to perform optimization of color mixing, organize pixel data, perform address translation, and the like.

[0051] The raster engine 325 includes a number of fixed-function hardware units that are configured to perform various raster operations. In one embodiment, the raster engine 325 includes a setup engine, a coarse raster engine, a culling engine, a clipping engine, a fine raster engine, and a tile merging engine. The setup engine receives the converted vertices and generates plane equations associated with the geometric primitives defined by the vertices. The plane equations are sent to the coarse raster engine to generate coverage information for the primitives (e.g., x, y coverage masks for tiles). The output of the coarse raster engine is sent to the culling engine, in which fragments associated with primitives that fail the z test are culled, and sent to the clipping engine, in which fragments outside the viewing cone are clipped. Those fragments that remain after clipping and culling can be passed to the fine raster engine to generate attributes of pixel fragments based on the plane equations generated by the setup engine. The output of the raster engine 325 includes fragments to be processed by, for example, a fragment shader implemented in the DPC 320.

[0052] Each DPC 320 included in the GPC 250 includes an M-pipe controller (MPC) 330, a primitive engine 335, and one or more SMs 340. The MPC 330 controls the operation of the DPC 320, routing packets received from the pipeline manager 310 to appropriate units in the DPC 320. For example, packets associated with vertices may be routed to the primitive engine 335, which is configured to retrieve vertex attributes associated with the vertices from the memory 204. Conversely, packets associated with shader programs may be sent to the SM 340.

[0053] SM 340 includes a programmable stream processor configured to process tasks represented by several threads. Each SM 340 is multithreaded and configured to execute multiple threads (e.g., 32 threads) from a specific thread group simultaneously. In one embodiment, SM 340 implements a SIMD (single instruction, multiple data) architecture, wherein each thread in a thread group (i.e., a warp) is configured to process different data sets based on the same instruction set. All threads in a thread group execute the same instruction. In another embodiment, SM 340 implements a SIMT (single instruction, multiple threads) architecture, wherein each thread in a thread group is configured to process different data sets based on the same instruction set, but wherein each thread in a thread group may diverge during execution. In one embodiment, a program counter, a call stack, and an execution state are maintained for each warp, and when a thread in a warp diverges, concurrency between serial executions in the warp and the warp is enabled. In another embodiment, a program counter, a call stack, and an execution state are maintained for each individual thread, enabling equal concurrency between all threads, within a warp, and between warps. When the execution state is maintained for each individual thread, threads executing the same instructions can converge and execute in parallel to achieve maximum efficiency. Figure 4A SM 340 is described in more detail.

[0054] MMU 390 provides an interface between GPC 250 and partition unit 280. MMU 390 can provide translation of virtual addresses to physical addresses, memory protection, and arbitration of memory requests. In one embodiment, MMU 390 provides one or more translation lookaside buffers (TLBs) for performing translation of virtual addresses to physical addresses in memory 204.

[0055] Figure 3B According to an embodiment, Figure 2 The memory partition unit 280 of the PPU 200. Figure 3BAs shown, the memory partition unit 280 includes a raster operation (ROP) unit 350, a second level (L2) cache 360, and a memory interface 370. The memory interface 370 is coupled to the memory 204. The memory interface 370 can implement a 32-bit, 64-bit, 128-bit, 1024-bit, etc. data bus for high-speed data transmission. In one embodiment, the PPU 200 includes U memory interfaces 370, one memory interface 370 for each pair of partition units 280, wherein each pair of partition units 280 is connected to a corresponding memory device 204. For example, the PPU 200 can be connected to up to Y memory devices 204, such as a high bandwidth memory stack or a graphics double data rate, version 5, synchronous dynamic random access memory, or other types of permanent storage.

[0056] In one embodiment, the memory interface 370 implements an HBM2 memory interface, and Y is equal to half of U. In one embodiment, the HBM2 memory stack is located on the same physical package as the PPU 200, saving a lot of power and area compared to traditional GDDR5 SDRAM systems. In one embodiment, each HBM2 stack includes 4 memory dies, and Y is equal to 4, each die of the HBM2 stack includes two 128-bit channels, a total of 8 channels, and the data bus width is 1024 bits.

[0057] In one embodiment, memory 204 supports single error correction double error detection (SECDED) error correction code (ECC) to protect data. ECC provides higher reliability for computing applications that are sensitive to data corruption. Reliability is particularly important in large cluster computing environments where PPU 200 processes very large data sets and / or long-running applications.

[0058] In one embodiment, the PPU 200 implements a multi-level memory hierarchy. In one embodiment, the memory partition unit 280 supports unified memory to provide a single unified virtual address space for the CPU and PPU 200 memory, enabling data sharing between virtual memory systems. In one embodiment, the frequency of accesses by the PPU 200 to memory located on other processors is tracked to ensure that memory pages are moved to the physical memory of the PPU 200 that is accessing these pages more frequently. In one embodiment, the NVLink 210 supports address translation services that allow the PPU 200 to directly access the CPU's page tables and provide the PPU 200 with full access to the CPU's memory.

[0059] In one embodiment, the copy engine transfers data between multiple PPUs 200 or between a PPU 200 and a CPU. The copy engine can generate a page fault for an address that is not mapped in a page table. The memory partition unit 280 can service the page fault, map the address into a page table, and then the copy engine can perform the transfer. In conventional systems, the memory used for multiple copy engine operations between multiple processors is fixed (i.e., non-pageable), greatly reducing the available memory. When there is a hardware page fault, the address can be passed to the copy engine without having to worry about whether the memory page is resident, and the copy process is transparent.

[0060] Data from memory 204 or other system memory may be retrieved by memory partition unit 280 and stored in L2 cache 360, which is located on-chip and shared between different GPCs 250. As shown, each memory partition unit 280 includes a portion of L2 cache 360 ​​that is associated with a corresponding memory device 204. Lower levels of cache may be implemented in different units within GPC 250. For example, each SM 340 may implement a level 1 (L1) cache. The L1 cache is a dedicated memory dedicated to a particular SM 340. Data from L2 cache 360 ​​may be retrieved and stored in each cache in the L1 cache for processing in the functional units of SM 340. L2 cache 360 ​​is coupled to memory interface 370 and XBar 270.

[0061] The ROP unit 350 performs graphics raster operations related to pixel color, such as color compression, pixel blending, etc. The ROP unit 350 also implements depth testing in conjunction with the raster engine 325, receiving the depth of the sample position associated with the pixel fragment from the culling engine of the raster engine 325. For the sample position associated with the fragment, the depth is tested relative to the corresponding depth in the depth buffer. If the fragment passes the depth test for the sample position, the ROP unit 350 will update the depth buffer and send the result of the depth test to the raster engine 325. It can be appreciated that the number of partition units 280 may be different than the number of GPCs 250, and therefore, each ROP unit 350 may be coupled to each of the GPCs 250. The ROP unit 350 tracks packets from different GPCs 250 and determines to which GPC 250 the results generated by the ROP unit 350 are routed via the Xbar 270. Although the ROP unit 350 is included in Figure 3B 280, but in other embodiments, ROP unit 350 may be located outside memory partition unit 280. For example, ROP unit 350 may reside in GPC 250 or other units.

[0062] Figure 4A According to an embodiment, Figure 3A The stream multiprocessor 340. Figure 4A As shown, SM 340 includes an instruction cache 405, one or more scheduler units 410 (K), a register file 420, one or more processing cores 450, one or more special function units (SFU) 452, one or more load / store units (LSU) 454, an interconnection network 480, and a shared memory / L1 cache 470.

[0063] As described above, the work distribution unit 225 dispatches tasks to be executed on the GPCs 250 in the PPU 200. Tasks are assigned to specific DPCs 320 within the GPC 250, and if the task is associated with a shader program, the task may be assigned to the SM 340. The scheduler unit 410(K) receives tasks from the work distribution unit 225 and manages the scheduling of instructions assigned to one or more thread blocks assigned to the SM 340. The scheduler unit 410(K) schedules the thread blocks for execution as warps of parallel threads, where each thread block is assigned at least one warp. In one embodiment, each warp executes 32 threads. The scheduler unit 410(K) can manage a plurality of different thread blocks, assign warps to different thread blocks, and then dispatch instructions from a plurality of different cooperative groups to various functional units (i.e., cores 450, SFUs 452, and LSUs 454) during each clock cycle.

[0064] Cooperative Groups is a programming model for organizing groups of communicating threads that allows developers to express the granularity of thread communication, enabling richer and more efficient expressions of parallel decompositions. The cooperative launch API supports synchronization between thread blocks to execute parallel algorithms. Traditional programming models provide a single, simple construct for synchronizing cooperative threads: a barrier across all threads of a thread block (i.e., the syncthreads() function). However, programmers often want to define thread groups at a granularity finer than the thread block granularity, and synchronize within the defined group, in the form of a collective, group-wide functional interface to achieve greater performance, design flexibility, and software reuse.

[0065] Cooperative Groups allow programmers to explicitly define groups of threads at sub-block (i.e., as small as a single thread) and multi-block granularity, and perform collective operations such as synchronization on threads in a cooperative group. The programming model supports clean composition across software boundaries, so that libraries and utility functions can safely synchronize in their local context without making any assumptions about convergence. Cooperative Groups primitives support new cooperative parallelism patterns, including producer-consumer parallelism, opportunistic parallelism, and global synchronization across the entire grid of thread blocks.

[0066] The dispatch unit 415 is configured to send instructions to one or more functional units. In this embodiment, the scheduler unit 410 (K) includes two dispatch units 415, which enable two different instructions from the same thread warp to be dispatched during each clock cycle. In alternative embodiments, each scheduler unit 410 (K) may include a single dispatch unit 415 or additional dispatch units 415.

[0067] Each SM 340 includes a register file 420 that provides a set of registers for the functional units of the SM 340. In one embodiment, the register file 420 is divided between each functional unit such that each functional unit is assigned a dedicated portion of the register file 420. In another embodiment, the register file 420 is divided between different warps executed by the SM 340. The register file 420 provides temporary storage for operands for data paths connected to the functional units.

[0068] Each SM 340 includes L processing cores 450. In one embodiment, SM 340 includes a large number (e.g., 128, etc.) of different processing cores 450. Each core 450 may include a fully pipelined, single-precision, double-precision, and / or mixed-precision processing unit, including a floating-point arithmetic logic unit and an integer arithmetic logic unit. In one embodiment, the floating-point arithmetic logic unit implements the IEEE 754-2008 standard for floating-point arithmetic. In one embodiment, core 450 includes 64 single-precision (32-bit) floating-point cores, 64 integer cores, 32 double-precision (64-bit) floating-point cores, and 8 tensor cores.

[0069] The tensor core is configured to perform matrix operations, and in one embodiment, one or more tensor cores are included in core 450. Specifically, the tensor core is configured to perform deep learning matrix arithmetic, such as convolution operations for neural network training and inference. In one embodiment, each tensor core operates on a 4x4 matrix and performs a matrix multiplication and accumulation operation D=A×B+C, where A, B, C, and D are 4x4 matrices.

[0070] In one embodiment, the matrix multiplication inputs A and B are 16-bit floating point matrices, while the accumulation matrices C and D can be 16-bit floating point matrices or 32-bit floating point matrices. The tensor core operates on 16-bit floating point input data and accumulates with 32-bit floating points. The 16-bit floating point multiplication requires 64 operations and produces a full-precision product, which is then accumulated with other intermediate products of the 4×4×4 matrix multiplication using 32-bit floating point addition. In practice, tensor cores are used to perform larger two-dimensional or high-dimensional matrix operations, which are composed of these smaller elements. APIs, such as the CUDA 9C++ API, expose specialized matrix loads, matrix multiplications and accumulations, and matrix storage operations to efficiently use tensor cores from CUDA-C++ programs. At the CUDA level, the warp-level interface assumes that matrices of size 16×16 span all 32 threads of the warp.

[0071] Each SM 340 also includes M SFUs 452 that perform special functions (e.g., attribute evaluation, inverse square root, etc.). In one embodiment, the SFUs 452 may include a tree traversal unit configured to traverse a hierarchical tree data structure. In one embodiment, the SFUs 452 may include a texture unit configured to perform texture map filtering operations. In one embodiment, the texture unit is configured to load a texture map (e.g., a two-dimensional array of texture pixels) from the memory 204 and sample the texture map to produce sampled texture values ​​for use in a shader program executed by the SM 340. In one embodiment, the texture map is stored in the shared memory / L1 cache 370. The texture unit implements texture operations, such as filtering operations using mip-maps (i.e., texture maps at different levels of detail). In one embodiment, each SM 340 includes two texture units.

[0072] Each SM 340 also includes N LSUs 454 that implement load and store operations between the shared memory / L1 cache 470 and the register file 420. Each SM 340 includes an interconnect network 480 that connects each functional unit to the register file 420 and connects the LSUs 454 to the register file 420, the shared memory / L1 cache 470. In one embodiment, the interconnect network 480 is a crossbar switch that can be configured to connect any functional unit to any register in the register file 420 and to connect the LSUs 454 to memory locations in the register file and the shared memory / L1 cache 470.

[0073] The shared memory / L1 cache 470 is an array of on-chip memory that allows data storage and communication between the SM 340 and the primitive engine 335 and between threads in the SM 340. In one embodiment, the shared memory / L1 cache 470 includes 128KB of storage capacity and is located in the path from the SM 340 to the partition unit 280. The shared memory / L1 cache 470 can be used to cache reads and writes. One or more of the shared memory / L1 cache 470, the L2 cache 360, and the memory 204 are backing stores.

[0074] Combining data cache and shared memory functionality into a single memory block provides the best overall performance for both types of memory access. This capacity can be used as a cache by programs that do not use the shared memory. For example, if the shared memory is configured to use half of the capacity, texture and load / store operations can use the remaining capacity. Integration within the shared memory / L1 cache 470 enables the shared memory / L1 cache 470 to function as a high throughput pipeline for streaming data while providing high bandwidth and low latency access to frequently reused data.

[0075] When configuring for general-purpose parallel computing, a simpler configuration can be used than for graphics processing. Specifically, Figure 2 The fixed-function graphics processing unit shown in is bypassed, creating a much simpler programming model. In the general parallel computing configuration, the work distribution unit 225 assigns and distributes thread blocks directly to the DPC 320. The threads in the block execute the same program, use unique thread IDs in calculations to ensure that each thread generates unique results, use SM 340 to execute the program and perform calculations, use shared memory / L1 cache 470 to communicate between threads, and use LSU 454 to read and write global memory through shared memory / L1 cache 470 and memory partition unit 280. When configured for general parallel computing, SM 340 can also write commands that scheduler unit 220 can use to start new work on DPC 320.

[0076] The PPU 200 may be included in a desktop computer, a laptop computer, a tablet computer, a server, a supercomputer, a smartphone (e.g., a wireless handheld device), a personal digital assistant (PDA), a digital camera, a vehicle, a head-mounted display, a handheld electronic device, etc. In one embodiment, the PPU 200 is embodied as a single semiconductor substrate. In another embodiment, the PPU 200 is included in a system on a chip (SoC) together with one or more other devices (such as an additional PPU 200, a memory 204, a reduced instruction set computer (RISC) CPU, a memory management unit (MMU), a digital-to-analog converter (DAC), etc.).

[0077] In one embodiment, PPU 200 may be included on a graphics card that includes one or more memory devices 204. The graphics card may be configured to interface with a PCIe slot on a desktop computer motherboard. In another embodiment, PPU 200 may be an integrated graphics processing unit (iGPU) or parallel processor included in a motherboard chipset.

[0078] Exemplary Computing System

[0079] As developers expose and use more parallelism in applications such as artificial intelligence computing, systems with multiple GPUs and CPUs are used in a variety of industries. High-performance GPU-accelerated systems with tens to thousands of computing nodes are deployed in data centers, research institutions, and supercomputers to solve larger problems. As the number of processing devices in high-performance systems increases, communication and data transmission mechanisms need to scale to support the increased bandwidth.

[0080] Figure 4B According to an embodiment, using Figure 2 200 in FIG. 4. The exemplary system 465 may be configured to implement Figure 1 The processing system 400 includes a CPU 430, a switch 410, and a plurality of PPUs 200, each of which has its own memory 204. The NVLink 210 provides a high-speed communication link between each PPU 200. Figure 4B 4. A specific number of NVLink 210 and interconnect 202 connections are shown in FIG. 4, but the number of connections to each PPU 200 and CPU 430 may be different. Switch 410 interfaces between interconnect 202 and CPU 430. PPU 200, memory 204, and NVLink 210 are located on a single semiconductor platform to form a parallel processing module 425. In an embodiment, switch 410 supports two or more protocols for interfacing between various different connections and / or links.

[0081] In another embodiment (not shown), NVLink 210 provides one or more high-speed communication links between each PPU 200 and CPU 430, and switch 410 interfaces between interconnect 202 and each PPU 200. PPU 200, memory 204, and interconnect 202 may be located on a single semiconductor platform to form a parallel processing module 425. In another embodiment (not shown), interconnect 202 provides one or more communication links between each PPU 200 and CPU 430, and switch 410 interfaces between each PPU 200 using NVLink 210 to provide one or more high-speed communication links between PPU 200. In another embodiment (not shown), NVLink 210 provides one or more high-speed communication links between PPU 200 and CPU 430 through switch 410. In another embodiment (not shown), interconnect 202 provides one or more communication links directly between each PPU 200. One or more NVLink 210 high-speed communication links may be implemented as a physical NVLink interconnect, or may be implemented as an on-chip or on-die interconnect using the same protocol as NVLink 210 .

[0082] In the context of this specification, a single semiconductor platform may refer to a separate single semiconductor-based integrated circuit manufactured on a bare die or chip. It should be noted that the term "single semiconductor platform" may also refer to a multi-chip module with increased connectivity that simulates operations on a chip and is a significant improvement over the use of conventional bus implementations. Of course, various circuits or devices may also be installed separately or in combination with various semiconductor platforms, depending on the needs of the user. Optionally, the parallel processing module 425 may be implemented as a circuit board substrate, and each PPU 200 and / or memory 204 may be a packaged device. In one embodiment, the CPU 430, switch 410, and parallel processing module 425 are located on a single semiconductor platform.

[0083] In one embodiment, the signaling rate of each NVLink 210 is 20 to 25 Gb / s, and each PPU 200 includes six NVLink 210 interfaces (e.g., Figure 4B As shown, each PPU 200 includes five NVLink 210 interfaces. Each NVLink 210 provides a data transfer rate of 25 Gb / sec in each direction, where six links provide a data transfer rate of 300 Gb / sec. When the CPU 430 also includes one or more NVLink 210 interfaces, the NVLink 210 can be used exclusively for Figure 4BPPU to PPU communication as shown, or some combination of PPU to PPU and PPU to CPU.

[0084] In one embodiment, NVLink 210 allows direct load / store / atomic access from CPU 430 to memory 204 of each PPU 200. In one embodiment, NVLink 210 supports coherency operations, allowing data read from memory 204 to be stored in the cache hierarchy of CPU 430, thereby reducing cache access latency of CPU 430. In one embodiment, NVLink 210 includes support for address translation services (ATS), allowing PPU 200 to directly access page tables in CPU 430. One or more of NVLink 210 may also be configured to operate in a low power mode.

[0085] Figure 4C An exemplary system 465 is shown in which various architectures and / or functions of the various embodiments described above can be implemented. The exemplary system 465 can be configured to implement Figure 1 The method 100 shown in FIG.

[0086] As shown, system 465 includes at least one central processing unit 430 connected to a communication bus 475. Communication bus 475 can be implemented using any suitable protocol, such as PCI (Peripheral Component Interconnect), PCI-Express, AGP (Accelerated Graphics Port), HyperTransport, or any other bus or one or more point-to-point communication protocols. System 465 also includes main memory 440. Control logic (software) and data are stored in main memory 440, which can take the form of random access memory (RAM).

[0087] The system 465 also includes an input device 460, a parallel processing system 425, and a display device 445, i.e., a conventional CRT (cathode ray tube), LCD (liquid crystal display), LED (light emitting diode), plasma display, etc. User input can be received from the input device 460, such as a keyboard, a mouse, a touch pad, a microphone, etc. Each of the above modules and / or devices can even be located on a single semiconductor platform to form the system 465. In addition, according to the needs of the user, each module can also be provided separately or in the form of various combinations of semiconductor platforms.

[0088] Further, system 465 may be coupled to a network (eg, a telecommunications network, a local area network (LAN), a wireless network, a wide area network (WAN) such as the Internet, a peer-to-peer network, a cable network, etc.) through network interface 435 for communication purposes.

[0089] System 465 may also include auxiliary storage (not shown). Auxiliary storage includes, for example, a hard disk drive and / or a removable storage drive, which represents a floppy disk drive, a tape drive, an optical disk drive, a digital versatile disk (DVD) drive, a recording device, a universal serial bus (USB) flash memory. The removable storage drive reads from and / or writes to a removable storage unit in a well-known manner.

[0090] Computer programs or computer control logic algorithms may be stored in the main memory 440 and / or the secondary storage. Such computer programs, when executed, enable the system 465 to perform various functions. The memory 440, storage and / or any other storage are possible examples of computer readable media.

[0091] The architecture and / or functionality of the various preceding figures may be implemented in the context of a general purpose computer system, a circuit board system, an entertainment specific gaming system, an application specific system, and / or any other desired system. For example, system 465 may take the form of a desktop computer, a laptop computer, a tablet computer, a server, a supercomputer, a smartphone (e.g., a wireless handheld device), a personal digital assistant (PDA), a digital camera, a vehicle, a head mounted display, a handheld electronic device, a mobile phone device, a television, a workstation, a gaming console, an embedded system, and / or any other type of logic.

[0092] Although various embodiments have been described above, it should be understood that they are provided by way of example only and not limitation.Thus, the breadth and scope of a preferred embodiment should not be limited by any of the above exemplary embodiments, but should be defined only in accordance with the claims and their equivalents.

[0093] Graphics Processing Pipeline

[0094] In one embodiment, PPU 200 includes a graphics processing unit (GPU). PPU 200 is configured to receive commands specifying a shader program for processing graphics data. Graphics data may be defined as a set of primitives, such as points, lines, triangles, quadrilaterals, triangle strips, etc. Typically, a primitive includes data specifying a number of vertices for the primitive (e.g., in a model space coordinate system), and attributes associated with each vertex of the primitive. PPU 200 may be configured to process graphics primitives to generate a frame buffer (i.e., pixel data for each pixel of a display).

[0095] The application writes the model data (i.e., a collection of vertices and attributes) for the scene to a memory, such as system memory or memory 204. The model data defines each object that may be visible on the display. Then, the application makes an API call to the driver kernel, which requests rendering and displaying the model data. The driver kernel reads the model data and writes commands to one or more streams to perform operations that process the model data. These commands can reference different shader programs to be implemented on the SM 340 of the PPU 200, including one or more of a vertex shader, a hull shader, a domain shader, a geometry shader, and a pixel shader. For example, one or more of the SM 340 can be configured to execute a vertex shader program that processes multiple vertices defined by the model data. In one embodiment, different SMs 340 can be configured to execute different shader programs concurrently. For example, a first subset of SMs 340 can be configured to execute a vertex shader program, and a second subset of SMs 340 can be configured to execute a pixel shader program. The first subset of SMs 340 processes vertex data to generate processed vertex data and writes the processed vertex data to L2 cache 360 ​​and / or memory 204. After the processed vertex data is rasterized (i.e., converted from three-dimensional data to two-dimensional data in screen space) to produce fragment data, the second subset of SMs 340 executes pixel shaders to generate processed fragment data, which is then blended with other processed fragment data and written to a frame buffer in memory 204. Vertex shader programs and pixel shader programs can be executed concurrently, processing different data from the same scene in a pipelined manner until all model data for the scene is rendered to the frame buffer. The contents of the frame buffer are then sent to a display controller for display on a display device.

[0096] Figure 5 According to an embodiment, Figure 2 2. A conceptual diagram of a graphics processing pipeline 500 implemented by PPU 200 of FIG. 1. Graphics processing pipeline 500 is an abstract flow chart of processing steps implemented to generate a two-dimensional computer-generated image from three-dimensional geometric data. As is well known, pipeline architectures can perform long-latency operations more efficiently by dividing the operations into multiple stages, where the output of each stage is coupled to the input of the next consecutive stage. Thus, graphics processing pipeline 500 receives input data 501 sent from one stage of graphics processing pipeline 500 to the next stage to generate output data 502. In one embodiment, graphics processing pipeline 500 may represent a graphics processing pipeline composed of API Defined Graphics Processing Pipeline As an option, graphics processing pipeline 500 can be implemented in the functional and architectural context of the preceding figure and / or one or more of any subsequent figures.

[0097] like Figure 5 As shown, the graphics processing pipeline 500 includes a pipeline architecture that includes multiple stages. The stages include, but are not limited to, a data assembly stage 510, a vertex shading stage 520, a primitive assembly stage 530, a geometry shading stage 540, a viewport scaling, culling and clipping (VSCC) stage 550, a rasterization stage 560, a fragment shading stage 570, and a raster operation stage 580. In one embodiment, input data 501 includes commands for configuring a processing unit to implement each stage of the graphics processing pipeline 500 and the geometric primitives (such as points, lines, triangles, quadrilaterals, triangle strips or fans, etc.) to be processed by each stage. Output data 502 may include pixel data (i.e., color data), which is copied to a frame buffer or other type of surface data structure in a memory.

[0098] The data assembly stage 510 receives input data 501 that specifies vertex data for a high-order surface, primitive, etc. The data assembly stage 510 collects the vertex data in temporary storage or queues, such as by receiving a command from a host processor that includes a pointer to a buffer in memory and reads the vertex data from the buffer. The vertex data is then sent to the vertex shading stage 520 for processing.

[0099] The vertex shading stage 520 processes vertex data by executing a set of operations (i.e., a vertex shader or program) for each vertex at a time. A vertex may, for example, be specified as a 4-coordinate vector (i.e., a pixel) associated with one or more vertex attributes (e.g., color, texture coordinates, surface normal, etc.).<x,y,z,w> ). The vertex shading stage 520 can manipulate various vertex attributes, such as position, color, texture coordinates, etc. In other words, the vertex shading stage 520 performs operations on vertex coordinates or other vertex attributes associated with the vertex. These operations typically include lighting operations (i.e., modifying the color attributes of the vertex) and transformation operations (i.e., modifying the coordinate space of the vertex). For example, a vertex can be specified using coordinates in an object coordinate space, which are transformed by multiplying the coordinates by a matrix that transforms the coordinates in the object coordinate space into world space or normalized device coordinate (NCD) space. The vertex shading stage 520 generates the transformed vertex data, which is sent to the primitive assembly stage 530.

[0100] The primitive assembly stage 530 collects the vertices output by the vertex shading stage 520 and groups the vertices into geometric primitives for processing by the geometry shading stage 540. For example, the primitive assembly stage 530 may be configured to group every three consecutive vertices into a geometric primitive (i.e., a triangle) for sending to the geometry shading stage 540. In some embodiments, particular vertices may be reused for consecutive geometric primitives (e.g., two consecutive triangles in a triangle strip may share two vertices). The primitive assembly stage 530 sends the geometric primitives (i.e., a collection of associated vertices) to the geometry shading stage 540.

[0101] The geometry shading stage 540 processes geometric primitives by performing a set of operations (i.e., geometry shaders or programs) on the geometric primitives. The tessellation operation can generate one or more geometric primitives from each geometric primitive. In other words, the geometry shading stage 540 can subdivide each geometric primitive into a finer grid of two or more geometric primitives for processing by the rest of the graphics processing pipeline 500. The geometry shading stage 540 sends the geometric primitives to the viewport SCC stage 550.

[0102] In one embodiment, the graphics processing pipeline 500 may be executed within a streaming multiprocessor, and the vertex shading stage 520, the primitive assembly stage 530, the geometry shading stage 540, the fragment shading stage 570, and the hardware / software associated therewith may perform processing operations sequentially. Once the sequential processing operations are completed, in one embodiment, the viewport SCC stage 550 may use the data. In one embodiment, primitive data processed by one or more stages in the graphics processing pipeline 500 may be written to a cache (e.g., an L1 cache, a vertex cache, etc.). In this case, in one embodiment, the viewport SCC stage 550 may access the data in the cache. In one embodiment, the viewport SCC stage 550 and the rasterization stage 560 are implemented as fixed function circuits.

[0103] The viewport SCC stage 550 performs viewport scaling, culling, and clipping of geometric primitives. Each rendered surface is associated with an abstract camera position. The camera position represents the position from which the viewer views the scene and defines a view cone that surrounds the scene objects. The view cone may include a viewing plane, a back plane, and four clipping planes. Any geometric primitive that is completely outside the view cone can be culled (i.e., discarded) because the geometric primitive does not contribute to the final rendered scene. Any geometric primitive that is partially inside the view cone and partially outside the view cone can be clipped (i.e., converted to a new geometric primitive that is enclosed within the view cone). In addition, each geometric primitive can be scaled according to the depth of the view cone. All potentially visible geometric primitives are then sent to the rasterization stage 560.

[0104] The rasterization stage 560 converts three-dimensional geometric primitives into two-dimensional fragments (e.g., capable of being used for display, etc.). The rasterization stage 560 can be configured to use the vertices of the geometric primitives to set a set of plane equations from which various attributes can be interpolated. The rasterization stage 560 can also calculate a coverage mask for multiple pixels, which indicates whether one or more sample positions of the pixel intercept the geometric primitive. In one embodiment, a z test can also be performed to determine whether the geometric primitive is occluded by other geometric primitives that have been rasterized. The rasterization stage 560 generates fragment data (i.e., interpolated vertex attributes associated with a specific sample position for each covered pixel), which is sent to the fragment shading stage 570.

[0105] The fragment shading stage 570 processes the fragment data by performing a set of operations (i.e., a fragment shader or program) on each fragment. The fragment shading stage 570 may generate pixel data (i.e., color values) for the fragment, for example, by performing lighting operations or sampling a texture map using interpolated texture coordinates of the fragment. The fragment shading stage 570 generates pixel data, which is sent to the raster operations stage 580.

[0106] The raster operations stage 580 may perform various operations on the pixel data, such as performing alpha tests, stencil tests, and blending the pixel data with other pixel data corresponding to other fragments associated with the pixel. When the raster operations stage 580 completes processing the pixel data (i.e., output data 502), the pixel data may be written to a render target, such as a frame buffer, a color buffer, etc.

[0107] It will be appreciated that the graphics processing pipeline 500 may also include one or more additional stages to replace or supplement one or more of the stages described above. Various implementations of the abstract graphics processing pipeline may implement different stages. In addition, in some embodiments, one or more of the stages described above (e.g., the geometry shading stage 540) may be excluded from the graphics processing pipeline. Other types of graphics processing pipelines are considered to be within the scope of the present disclosure. Further, any stage of the graphics processing pipeline 500 may be implemented by one or more dedicated hardware units in a graphics processor (such as PPU 200). Other stages of the graphics processing pipeline 500 may be implemented by programmable hardware units, such as SM 340 of PPU 200.

[0108] The graphics processing pipeline 500 may be implemented via an application program executed by a host processor (such as a CPU). In one embodiment, a device driver may implement an application programming interface (API) that defines various functions that an application program may use in order to generate graphics data for display. The device driver is a software program that includes a plurality of instructions that control the operation of the PPU 200. The API provides an abstraction for programmers that allows programmers to use dedicated graphics hardware (such as the PPU 200) to generate graphics data without requiring the programmer to use a specific instruction set of the PPU 200. An application program may include an API call that is routed to a device driver of the PPU 200. The device driver interprets the API call and performs various operations in response to the API call. In some instances, the device driver may perform operations by executing instructions on the CPU. In other cases, the device driver may perform operations by initiating operations on the PPU 200 at least in part using an input / output interface between the CPU and the PPU 200. In one embodiment, the device driver is configured to implement the graphics processing pipeline 500 using the hardware of the PPU 200.

[0109] To implement the various stages of the graphics processing pipeline 500, various programs may be executed in the PPU 200. For example, a device driver may launch a kernel on the PPU 200 to perform the vertex shading stage 520 for one SM 340 (or multiple SMs 340). The device driver (or the initial kernel executed by the PPU 300) may also launch other kernels on the PPU 300 to perform other stages of the graphics processing pipeline 500, such as the geometry shading stage 540 and the fragment shading stage 570. In addition, some stages of the graphics processing pipeline 500 may be implemented on fixed unit hardware, such as a rasterizer or data assembler implemented in the PPU 300. It will be appreciated that the results of one kernel may be processed by one or more intermediate fixed function hardware units before being processed by subsequent kernels on the SM 340.

[0110] Machine Learning

[0111] Deep Neural Networks (DNNs) developed on processors such as the PPU 200 have been used for a variety of different use cases, from self-driving cars to faster drug development, from automatic image captioning in online image databases to intelligent real-time language translation in video chat applications. Deep learning is a technology that models the neural learning process of the human brain, which continuously learns, continuously gets smarter, and provides more accurate results faster over time. Children are initially taught by adults to correctly identify and classify various shapes, and eventually are able to identify shapes without any guidance. Similarly, deep learning or neural learning systems also need to be trained in object recognition and classification, making them smarter and more efficient in identifying basic objects, occluded objects, etc., while also assigning context to objects.

[0112] At the simplest level, neurons in the human brain see the various inputs they receive, assign a level of importance to each of those inputs, and then pass outputs to other neurons for action. An artificial neuron, or perceptron, is the most basic model of a neural network. In one example, a perceptron can receive one or more inputs representing various features of an object, and the perceptron is trained to recognize and classify those features, assigning each feature a certain weight based on how important they are in defining the shape of the object.

[0113] Deep neural network (DNN) models contain many connected perceptrons (e.g., nodes) in multiple layers and can be trained with large amounts of input data to solve complex problems quickly and with high accuracy. In one example, the first layer of a DLL model breaks down an input image of a car into its different parts and looks for basic patterns, such as lines and angles. The second layer assembles the lines to look for higher-level patterns, such as wheels, windshields, and mirrors. The next layer identifies the type of vehicle, and the final layers generate labels for the input image that identify the model as a specific car brand.

[0114] Once trained, a DNN can be deployed and used to identify and classify objects or patterns in a process called inference. Examples of inference (the process by which a DNN extracts useful information from a given input) include recognizing handwritten numbers on checks deposited into an ATM, identifying images of friends in photos, delivering movie recommendations to more than 50 million users, identifying and classifying different types of cars, pedestrians, and road hazards in self-driving cars, or translating human language in real time.

[0115] During training, data flows through the DNN in a forward propagation phase until a prediction is produced indicating the label corresponding to the input. If the neural network fails to label the input correctly, the error between the correct label and the predicted label is analyzed and the weights of each feature are adjusted during the backpropagation phase until the DNN correctly labels the input and other inputs in the training dataset. Training complex neural networks requires a large amount of parallel computing performance, including floating-point multiplication and addition supported by the PPU 200. Inference, which is less computationally intensive than training, is a latency-sensitive process where the trained neural network is applied to new inputs it has never seen before to classify images, translate speech, and generally infer new information.

[0116] Neural networks rely heavily on matrix math operations, and complex multi-layer networks require a lot of floating point performance and bandwidth for both efficiency and speed. With thousands of processing cores optimized for matrix math operations and delivering tens to hundreds of TFLOPS performance, the PPU 200 is a computing platform capable of providing the performance required for deep neural network-based artificial intelligence and machine learning applications.

[0117] Anti-aliasing for hybrid deferred ray tracing

[0118] In one embodiment, as part of anti-aliasing deferred rendering for a scene, anti-aliasing may be performed on the identified complex pixels. This may be performed by implementing hybrid deferred ray tracing anti-aliasing for the scene.

[0119] Figure 6 A flow chart of a method 600 for performing anti-aliasing for hybrid deferred ray tracing according to an embodiment is shown. Although the method 600 is described in the context of a processing unit, the method 600 may also be performed by a program, a custom circuit, or a combination of a custom circuit and a program. For example, the method 600 may be performed by a GPU (graphics processing unit), a CPU (central processing unit), or any processor capable of performing parallel path space filtering by hashing. In addition, those skilled in the art will appreciate that any system that performs the method 600 is within the scope and spirit of the embodiments of the present invention.

[0120] As shown in operation 602, a non-antialiased G-buffer is rendered for the scene. For example, the G-buffer may include a buffer that stores scene geometry for each pixel. In another example, a multi-sample antialiasing (MSAA) depth buffer may be rendered using target independent rasterization (TIR). In yet another example, MSAA normals may be rendered using TIR.

[0121] Further, as shown in operation 604, complex pixels in the scene are identified. For example, complex pixels may include pixels around one or more discontinuities (such as depth, brightness, etc.) in the scene. In another example, identifying complex pixels in the scene may use rendered MSAA depth and / or MSAA normals.

[0122] Still further, as shown in operation 606, rays are traced through the complex pixels to generate samples for the G-buffer. For example, storage may be allocated for a predetermined number of G-buffer data for the complex pixels. In another example, a predetermined number of rays may be traced through each complex pixel. In another example, the G-buffer data may be recorded for the most recent hit. In another example, additional rays may be iteratively traced after the results are received. In another example, ray hits may be calculated for the complex pixels during deferred illumination.

[0123] In another embodiment, a parallel processing unit (PPU) may be used to trace rays, such as Figure 2 The PPU 200 shown in FIG.

[0124] In this way, deferred ray traced anti-aliasing can be performed using a single rasterization pass through the scene. In addition, anti-aliasing can be achieved by combining single-pass deferred GPU rasterization and GPU ray tracing.

[0125] Adaptive ray tracing for temporal anti-aliasing

[0126] Temporal Anti-Aliasing (TAA) algorithms are widely used in today's video games. However, current TAA implementations are limited by the fact that when a particular pixel is identified as deviating from the correct value during TAA, there is no efficient way to accurately shade that pixel (only often broken heuristics based on the root cause of sampling problems).

[0127] Figure 7 A flow chart of a method 700 for performing adaptive ray tracing for temporal anti-aliasing according to an embodiment is shown. Although the method 700 is described in the context of a processing unit, the method 700 may also be performed by a program, a custom circuit, or a combination of a custom circuit and a program. For example, the method 700 may be performed by a GPU (graphics processing unit), a CPU (central processing unit), or any processor capable of performing parallel path space filtering by hashing. In addition, those skilled in the art will appreciate that any system that performs the method 700 is within the scope and spirit of embodiments of the present invention.

[0128] As shown in Figure 702, temporal anti-aliasing (TAA) is applied to the image. In one embodiment, the image can be one of a series of rasterized images. In another embodiment, the image can be created using one or more rendering techniques. For example, the image can be created using forward or delayed rasterization, ray tracing, point-based rendering, image-based rendering, and the like.

[0129] Additionally, in one embodiment, TAA may be applied to an image to remove aliasing effects (e.g., geometric aliasing, temporal aliasing, specular aliasing, etc.) within a rasterized image. For example, TAA may attempt to correct aliasing of a major visible surface in a rasterized image. In another embodiment, the result of applying TAA to a rasterized image may include a color buffer for pixels within the rasterized image.

[0130] Further, as shown in operation 704, failed pixels resulting from applying TAA to the image are identified. In one embodiment, the failed pixels may include pixels in the image for which TAA failed. In another embodiment, failed pixels may be identified during TAA. For example, TAA may compare a historical value of a pixel in a previous frame with a current value of an adjacent pixel in a current frame. In another example, a pixel having a historical value that differs from a current value of an adjacent pixel by more than a predetermined amount may be identified as a failed pixel. In this way, a pixel may be identified as complex and may be marked as a failed pixel.

[0131] Still further, in one embodiment, a failed pixel may be identified by comparing the depth value of the pixel with the depth values ​​of adjacent pixels. For example, in response to determining that the depth value of the pixel differs from the depth value of the adjacent pixel by more than a predetermined amount, the pixel may be identified as a failed pixel. In this manner, a pixel may be identified as being associated with a depth edge and may be marked as a failed pixel.

[0132] In addition, in one embodiment, failed pixels may be identified by comparing the brightness value of the pixel with the brightness values ​​of adjacent pixels. For example, in response to determining that the brightness value of the pixel differs from the brightness value of the adjacent pixels by more than a predetermined amount, the pixel may be identified as a failed pixel. In another embodiment, failed pixels may be marked (e.g., using a sentinel value, etc.). In another embodiment, failed pixels may be included in a segmentation mask.

[0133] Additionally, as shown in operation 706, anti-aliasing is performed for each failed pixel using a combination of ray tracing (e.g., GPU optimized ray tracing) and fast approximate anti-aliasing (FXAA). In one embodiment, the ray tracing may perform supersampling. In another embodiment, performing anti-aliasing may include tracing multiple rays through each failed pixel. For example, performing anti-aliasing may include using GPU ray tracing to determine point sampled visibility of sub-pixel regions within each failed pixel.

[0134] Further, in one embodiment, performing anti-aliasing for each failed pixel may include analytically solving the geometric surface and its coverage of failed pixels. In another embodiment, performing anti-aliasing may include analytically evaluating visibility. In another embodiment, performing anti-aliasing for each failed pixel may result in a ray-traced anti-aliased texture.

[0135] Still further, the result of performing anti-aliasing on the failed pixels (e.g., a texture of the first set of anti-aliased pixels) can be combined with the result of applying TAA to the image (e.g., a color buffer for the second set of anti-aliased pixels) to create the output image. For example, the anti-aliased texture result of performing anti-aliasing on the failed pixels can be blended with the color buffer produced by applying TAA to the image.

[0136] In this way, pixels in an image that fail antialiasing using TAA can be antialiased using GPU ray tracing, and the results can be combined. This potentially solves the problems with TAA (blurring, ghosting, etc.) entirely, while amortizing the cost of ray tracing across the rasterized image.

[0137] In addition, in one embodiment, one or more pixels associated with de-occlusion in the original image may be identified. For example, the identified pixels may include pixels for which temporal information does not exist (e.g., due to occlusion, data in the current frame is not available, and / or temporal data for the pixel is not available in a previous image). In one embodiment, these pixels may be identified by identifying one or more motion vector failures associated with the pixels. In another embodiment, anti-aliasing may be performed on these pixels using fast approximate anti-aliasing (FXAA). For example, FXAA may filter the color image values ​​of these pixels based on the degree to which the pixels are marked as edges.

[0138] Additionally, in one embodiment, pixels associated with de-occlusion may be included in a segmentation mask. In another embodiment, a segmentation mask including these pixels may be separate from a segmentation mask including failed pixels. In another embodiment, the results of performing anti-aliasing on failed pixels (e.g., a first set of anti-aliased pixels) may be combined with the results of applying TAA to the original image (e.g., a second set of anti-aliased pixels) and the results of performing anti-aliasing on pixels associated with de-occlusion (e.g., a third set of anti-aliased pixels) to create an output image.

[0139] In one embodiment, a renderer may be used to generate a rasterized image. Additionally, anti-aliasing may be applied to the rasterized image using TAA. Further, pixels that fail anti-aliasing using TAA may be identified and marked, and the TAA results for these pixels may be discarded. Additionally, anti-aliasing may be applied to the marked pixels, using super-sampled GPU ray tracing to create a ray-traced anti-aliased texture result. Additionally, the ray-traced anti-aliased texture result may be blended with a color buffer created using TAA.

[0140] In another embodiment, a parallel processing unit (PPU) may be used to perform anti-aliasing, such as Figure 2 The PPU 200 shown in FIG.

[0141] In this way, pixels that fail TAA can be identified and AA can be performed on those pixels using a combination of ray tracing and FXAA to improve the results of AA. In addition, the cost of ray tracing can be amortized by selectively applying it to the failed results of TAA. Further, using FXAA, cheaper and more efficient AA can be performed on pixels for which temporal information does not exist. Still further, AA can also be performed in a selective and intelligent manner using TAA, ray tracing, and FXAA in a single image.

[0142] Adaptive temporal and hybrid deferred anti-aliasing for GPU ray tracing

[0143] Anti-aliasing is a class of techniques used to remove image artifacts caused by insufficient sampling rates. Multi-sample anti-aliasing (MSAA) is a popular anti-aliasing technique that samples visibility at a rate different from the typical shading rate of per primitive per pixel. While somewhat effective in terms of geometric anti-aliasing, MSAA has a high storage cost due to storing depth and color samples at the sampling rate. In addition, it may suffer from higher bandwidth usage in cases where color compression does not compress color data well. Therefore, the high image quality produced by applying MSAA comes at a relatively high cost.

[0144] The image quality of MSAA is desirable without the high associated costs. When a primitive completely covers a pixel, no further visibility calculations need to be performed. When a pixel is partially covered by one or more primitives, it is necessary to determine how much of the pixel is covered by each intersecting primitive in order to calculate the correct visibility. Using this knowledge, embodiments of the present invention include a method for identifying "complex" pixels (pixels that benefit from more accurate visibility calculations than a single raster sample and / or pixels that contain depth or other discontinuities that cause aliasing). Other embodiments of the present invention provide methods for calculating visibility of identified pixels with increased accuracy.

[0145] Pixel classification using conservative rasterization

[0146] Complex pixels as described herein can be identified by analyzing depth and / or primitive ID buffers; however, this approach may miss small geometric features. A common problem is that standard rasterization does not adequately sample cables or fences at distances. Embodiments of the present invention can avoid using multiple sub-pixel samples during pixel classification by using conservative rasterization.

[0147] GPU hardware support for conservative rasterization usually refers to rasterization modes where pixels that are partially covered by a primitive are rasterized using this rasterization mode. There are different tiers of this feature. At tier 3, you can (optionally) pass in a system variable called SV_InnerCoverage, which has its least significant bit (LSB) set to 1 when the pixel is guaranteed to be fully covered by the primitive.

[0148] Exemplary Algorithm

[0149] Using a Tier 3 (or similar) conservative rasterizer, the preferred embodiment implements the following process for identifying pixels that would benefit from more accurate visibility calculations.

[0150] Pass 1: Depth pre-pass

[0151] From this pass, a depth buffer is generated which stores the depth of the nearest fully covered pixel. According to a preferred embodiment, the depth pre-pass may be performed as follows:

[0152] 1. Enable conservative rasterization and enable depth testing for both reading and writing.

[0153] 2. The pixel shader accepts SV_InnerCoverage as input. It checks this value to find pixels that are completely covered by the primitive and discards pixels that are not completely covered. The result is that only the depth value of completely covered pixels will be written, and because they are completely covered, it is the real depth (not the extrapolated / clamped depth). Pixels that are only partially covered are discarded and the depth value is not updated for these pixels.

[0154] Channel 2: Pixel identification channel

[0155] After this pass, a 2D surface is obtained that contains non-zero values ​​for pixels that have some partially visible primitives and require further processing to become visible. The values ​​in these positions represent the number of primitives that may be visible in these pixels. By incrementing the unordered access view (UAV) counter for the same UAV, the total number of primitives covering all pixels of interest can be tracked. Note that this is an approximate heuristic, as the actual visibility inside these pixels may be more complicated than the number of primitives touching these pixels.

[0156] According to a preferred embodiment, the pixel identification channel can be performed according to the following steps:

[0157] 1. Enable conservative rasterization and depth testing for reads, but disable for writes.

[0158] 2. Bind the depth-buffer generated in channel 1 as the depth buffer.

[0159] 3. Bind the unordered access view as the size and format of the render target UINT_8 / 16 / 32 and clear the UAV with all zeros. The size determines the maximum number of primitives that can touch a pixel (256, 64K or 4G respectively).

[0160] 4. Bind a pixel shader that accepts SV_InnerCoverage as input, but this pass skips processing pixels that are fully covered. Instead, for pixels that are partially covered and pass the depth test, the shader marks these pixels as interesting by incrementing the UAV by one at that location each time the UAV is called.

[0161] Improved visibility calculations

[0162] Once complex pixels are identified, more accurate methods can be applied to calculate the visibility of these pixels. This can be achieved in one of two ways: 1) using GPU ray tracing to increase the sub-pixel sampling rate, or 2) solving visibility analytically.

[0163] GPU ray tracing with DXR

[0164] According to a preferred embodiment, improved visibility estimates are calculated for complex pixels by point sampling sub-pixel regions using GPU ray tracing. Recently, Microsoft announced the DirectX Ray Tracing (DXR) API, which uses NVIDIA's RTX ray tracing technology and makes GPU ray tracing feasible in real-time applications. GPU ray tracing also eliminates the hardware limitations on the number and positioning of samples within a pixel in traditional raster-based pipelines.

[0165] Sampling visibility multiple times within a pixel provides a reasonable approximation (but not an exact solution) using enough samples. Traced rays (e.g., from a DXR ray generation shader) can use the aforementioned primitive count as a guide for how to distribute rays within the sub-pixel area. More rays can be dedicated to pixels touched by more primitives, or a dynamic number of visibility or colored sub-samples can be snapshotted for each complex pixel.

[0166] Visibility with non-DXR based ray tracing

[0167] It is possible to improve the sampling-based visibility determination technique using analytical methods. An example process can be performed as follows:

[0168] 1. Fill the buffer to 0xffffffffffffff with enough space for N (number of ray samples) 64-bit entries per pixel

[0169] 2. For complex pixels (simple pixels can be culled by e.g. stencil), re-render using conservative rasterization

[0170] 3. Use fast GS to pass the world space vertices of the triangle to the pixel shader

[0171] 4. In the pixel shader, for each sample s(1..N)

[0172] a. Set up a ray from the eye to the WS position of sample s in pixel

[0173] b. Calculate the intersection of the current triangle and the ray (which hits the triangle)

[0174] i. Construct a 64-bit word like this (DepthValue: ID) (32:32)

[0175] 1. ID is a 32-bit word that identifies the object and the primitives in the object

[0176] ii. Update the current pixel's sample s (old,

[0177] (DepthValue: ID))

[0178] iii. If 32 32-bit IDs are not enough, you need to use ROV

[0179] 5. Run on all complex pixels

[0180] a. Run on all N samples

[0181] i. Extract the ID and recalculate the ray intersection with the triangle

[0182] ii. Light rays cross

[0183] iii. Accumulated lighting

[0184] b. Divide the accumulated illumination by N

[0185] c. Write the result

[0186] Additional pixel classification methods

[0187] Further embodiments for classifying pixels and determining how many rays to cast when using GPU ray tracing may include:

[0188] 1. Calculate the brightness difference between adjacent pixels

[0189] 2. Calculate the difference of the 3-tuple (primitive ID, instance ID, drawCallID) of neighboring pixels. If tessellation is enabled, a unique ID can be generated for each triangle, and tessellation uses atomic increments in the GS and does not use the primitive ID to generate the triangle. When tessellation is enabled, the primitive ID corresponds to the patch ID.

[0190] 3. Calculate the differences in material ID or material parameters of adjacent pixels.

[0191] 4. Calculate the difference in depth or surface normals of adjacent pixels.

[0192] 5. Combine 1-4.

[0193] 6.1-4 binding time changes.

[0194] Anti-aliasing for hybrid deferred ray tracing

[0195] Typically antialiasing is performed using a temporal screen-space filter, or by applying techniques such as Aggregated G-Buffer Antialiasing (AGAA). AGAA is a technique that uses graphics hardware to implement deferred rendering of antialiased complex geometry. AGAA uses the rasterization pipeline to generate a compact, pre-filtered representation of geometry within each pixel. Shading is then performed at a fixed rate, independent of geometry complexity. By decoupling the shading rate from the geometry sampling rate, the algorithm reduces the storage and bandwidth costs of the geometry buffer and allows scaling to high visibility sampling rates for antialiasing. However, traditional AGAA techniques do not adequately handle complex pixels, and AGAA alone may not be sufficient to adaptively generate new information for each subsample for complex pixels, a complex process that typically requires multiple rendering passes for the scene.

[0196] Further embodiments of the present invention are directed to a new alternative approach that generates additional "sub-samples" only when necessary. Since an arbitrary number of rays can be incident, the quality is scalable. The solution is orthogonal to temporal filtering. According to these embodiments, multiple rendering / rasterization passes of the scene are no longer required. The user can selectively generate more samples only for certain geometry types (such as leaves, etc.).

[0197] In accordance with one or more embodiments, a fully scalable anti-aliasing solution is provided that perfectly combines the best of rasterization and accelerated ray tracing using co-processing (e.g., via a tree traversal unit (TTU)) to generate demand for a GPU that can perform fast and hardware-accelerated ray tracing.

[0198] An exemplary implementation is as follows:

[0199] 1. Rendering non-antialiased G-buffer (see below for details)

[0200] 2. Find the pixels surrounding the discontinuity in terms of depth, brightness, etc.

[0201] 3. Send several rays (e.g. ray tracing) through the pixels found in step 2 to generate G-buffer samples.

[0202] Another exemplary implementation is as follows:

[0203] 1. Rendering a non-antialiased G-buffer

[0204] a. Optionally render only the MSAA depth buffer using Target Independent Rasterization (TIR)

[0205] b. Optionally render MSAA normals using TIR

[0206] 2. Detecting complex pixels

[0207] a. Look for depth / brightness and other inconsistencies / discontinuities

[0208] b. Use MSAA depth and / or MSAA normals when necessary

[0209] 3. N-sample allocation storage for G-buffer data of complex pixels

[0210] a. You can also use a subsample list for each complex pixel

[0211] 4. Each complex pixel is incident with N rays (a time filter can be used here)

[0212] a. Record the most recently hit G-buffer data

[0213] b. Optionally, the result can be iterated to adaptively shoot more rays 5. During deferred lighting - the rays hitting the pixels are also calculated for complex pixels

[0214] Adaptive ray tracing for temporal anti-aliasing

[0215] Existing temporal anti-aliasing algorithms (TAA) are widely used in current video games. One of the main limitations of such algorithms is that when a particular pixel is identified as deviating from the correct value, there is no good way to shade that pixel, and only some heuristic rules, which break down depending on the root cause of the sampling problem.

[0216] Typically, a sample color is chosen heuristically when motion vectors and color between frames indicate that a normal algorithm will not do the right thing. Rather than using a heuristic rule to choose a color, one or more embodiments of the present invention project additional samples directly into the scene and gather the information needed to choose the correct color.

[0217] According to a preferred embodiment, temporal anti-aliasing may be performed according to the process described below:

[0218] 1. Use an existing renderer (forward or deferred... or even ray-traced raw rays) to generate a rasterized image as usual. Maybe MIP-biased oversharpening

[0219] 2. Run TAA channel

[0220] a.* Detect when TAA data is "bad" / color may not be part of the pixel

[0221] (This shows up as a point where "color clamping / clipping" is needed; it's also possible to keep historical EMWVariance values ​​which is smarter)

[0222] b. Don't use the TAA result for this pixel... instead, flag the pixel for supersampled ray tracing. This can be with a sentinel value or an alpha channel.

[0223] 3. During the TAA pass, dispatch AA rays for the marked pixels and write the output to the second texture

[0224] a.*MSAA click position + temporal sub-pixel offset, but never the center pixel that has been produced by the raster pass.

[0225] b.* Run the exact same "pixel" from the rasterizer pass on the ray hit

[0226] Shader HLSL code

[0227] c.* ...including using any AO / SSRT / subsurface / shadow maps etc. screen space inputs that pixels use for regular raster passes. Raster shading should match consistency

[0228] d. Depending on the embodiment, this process can be applied to opaque surfaces, assuming that transparency / grain / etc has its own pipeline as a separate pass to go through the pipeline and handle AA, just like they do in game engines today.

[0229] 4. Mixing ray traced anti-aliased texture results with the color buffer

[0230] a.* Use raster operations, because using atoms in a ray tracing pass can be slow.

[0231] Adaptive Temporal Anti-Aliasing

[0232] In one embodiment, a practical algorithm for real-time adaptive supersampling in games is provided. It extends temporal anti-aliasing of raster images through adaptive ray tracing and conforms to the constraints of commercial game engines and today's GPU ray tracing APIs. The algorithm eliminates blurring and ghosting artifacts associated with standard temporal anti-aliasing and achieves a quality close to 8x supersampling for geometry, shadows, and materials while staying within the 33ms frame budget required for most games.

[0233] introduce

[0234] Aliasing of the primary visible surface is one of the most fundamental and challenging limitations of computer graphics. Almost all rendering samples the surface at points within a pixel, and therefore errors arise when the sampled points are not representative of the pixel as a whole, i.e. when the primary surface is undersampled. This is true whether the points are tested by casting rays or by rasterizing amortized ray casting, and regardless of what shading algorithm is used. Of course, analytical renderers, such as perfect ray tracing in space and time, can completely avoid the ray (under)sampling problem; although there are some analytical solutions in limited cases, point sampling from ray or raster intersections remains the only fully developed approach for efficient rendering of complex geometry, materials, and shading. Even "point-based" renderers actually trace or splatter rays through rasterization.

[0235] This aliasing caused by undersampling manifests itself as jagged edges, spatial noise, and flickering (temporal noise). Attempts to hide these errors through more extensive and complex reconstruction filters in both space (such as MLAA and FXAA) and time (such as SMAA and TAA) may convert these artifacts into blurring (in space) or ghosting (blurring in time).

[0236] With a fixed sample count per pixel of the image, the only real solution to aliasing is to increase the sampling density and limit the bandwidth of the signal being sampled. Increasing density helps but does not solve the problem at a rate that is affordable for realtime: supersampling (SSAA) incurs a cost that is linearly proportional to the number of samples, while quality only increases with the square root; multisampling (MSAA, CSAA, SBAA, SRAA) samples geometry, materials, and shadows at different rates to heuristically reduce cost but also reduce quality; and aggregation reduces cost more aggressively but still limits quality at practical speeds. For bandwidth-constrained scenes, pre-filtering of materials, geometry level of detail, and shader level of detail improve undersampling through MIP mapping and variants such as LEAN. They also introduce other problems of excessive blur or pop-in (temporal and spatial discontinuities) while complicating the rendering system and ultimately fail to fully solve the problem.

[0237] The standard for real-time rendering is to use multiple of the above strategies simultaneously. In the best cases, this can nearly eliminate the viewer's perception of aliasing artifacts. While admiring these successes, the main aliasing challenges remain open to real-time because they are game-specific solutions that require a great deal of engineering complexity and careful hand-tuning of the scene by the designer.

[0238] These problems are caused by unnecessary costs or limitations of real-time under-sampling of rays and solutions due to the fixed sample count per pixel. The adversary can always place material, geometry, or shading features between samples, resulting in unbounded errors, such as very bright, very small light sources that are rarely sampled by the center of a pixel.

[0239] Offline ray tracing renderers have long used high and adaptive sample counts to simply and elegantly solve the aliasing problem, where the renderer tracks some high minimum number of samples per pixel (e.g., 64), and then tracks additional batches of samples within that pixel until a maximum threshold or stable distribution is reached. Since almost all real-time rendering is based on regular rasterization sampling rates, there has previously been no practical way to adaptively sample in real time. Even the simple approach of stenciling pixels that are likely to alias and doing a secondary high-density pass on them is inefficient: the rasterizer needs to process all geometry, even if only a few pixels are affected. For example, pixels that need antialiasing can be actively identified based on track shadows and high-resolution geometry passes, but nearly the same effect as SSAA can be achieved with a 10% frame time reduction, despite halving the number of shadow samples.

[0240] A method for real-time adaptive sampling using a hybrid of ray tracing and rasterization is proposed. This method can be enabled by the recently released DirectX Ray Tracing API (DXR). DXR allows full interoperability between data structures and shaders used for both types of rendering across the entire game engine, eliminating the impracticality of duplicating those between the ray and raster APIs previously used for hybrid approaches.

[0241] Adaptive sampling builds on this by showing how to effectively combine these techniques for modern graphics hardware, and how to use adaptive sampling in the context of temporal antialiasing to amortize the cost of rasterizing samples over time without causing blurring or ghosting.

[0242] Temporal Antialiasing

[0243] Temporal anti-aliasing [8] (TAA) is fast and quite good when it can be handled. TAA applies a sub-pixel displacement to the image plane for each frame, and then accumulates an exponentially weighted moving average over the previous frames, rendering each frame with only one sample per pixel. In static scenes, this is close to the quality of full-screen supersampling.

[0244] For dynamic scenes, TAA "re-projects" samples from the accumulated history buffer by fetching back along the per-pixel motion vector offset texture generated previously by the rasterizer pass.

[0245] TAA fails in some cases. When new screen areas are disoccluded (displayed) by object motion, they are not represented in the history buffer or are incorrectly represented by the motion vectors. Camera rotation and backward translation can also cause thick disocclusions at the edges of the screen. Sub-pixel features such as lines and fine material details can slide between consecutive offset samples and therefore cannot be represented by motion vectors in the next frame. Transparent surfaces create pixels where the motion vector from the opaque object does not match the total motion of the object being represented. Finally, shadows and reflections do not move in the direction of the motion vector of the surface they are disoccluding.

[0246] When TAA fails, it either produces ghosting (blurring due to integrating incorrect values) or displays the original as aliasing, flickering, and noise. Standard TAA attempts to detect these cases by comparing historical samples to the local neighbors of the corresponding pixels in the new frame. When they differ too much, TAA uses various heuristics to clip, clamp, or interpolate in color space. The best practices for these heuristics change frequently, and no universal solution has been found before.

[0247] Exemplary Algorithm

[0248] An exemplary approach is designed to be compatible with legacy game engines and use the strengths of TAA while explicitly and simply addressing its failings.

[0249] One illustrative idea is to run the basic scheme of TAA on most pixels, and then, instead of trying to combat its failures with a heuristic, output a conservative segmentation image that explains where and why it will fail. At the failing pixels, TAA's complex heuristics can be replaced by robust alternatives that adapt to the image content.

[0250] Figure 8 An exemplary adaptive temporal anti-aliasing algorithm 800 in the context of a DXR ray tracing API is shown according to an exemplary embodiment. In the figure, rectangular images 802A-F represent visualizations of buffers, and rounded rectangles 804-812 represent shader channels. Not all intermediate buffers are shown. For example, if the output of the previous frame is fed back as input to the TAA, the associated ping-pong buffer is not shown.

[0251] In one embodiment, the segmented image is actually stored in the alpha channel of the TAA output. In the image, ray tracing supersampling will be applied to the first group of pixels. The first group of pixels represents cases where the historical samples deviate significantly from the corresponding neighbors in the new frame (TAA's usual metric for color clipping and noise generation), as well as pixels identified as depth edges or high material gradients. The inclusion of additional terms ensures that conservative ray tracing is applied to potential sources of aliasing that history alone failed to detect. The second group of pixels in the image includes unoccluded pixels. These would be expensive to ray trace because they can be large and are typically only rendered for a single frame, such as the edges of a screen when rotating. The second group of pixels can be identified by motion vector failures, and FXAA is only run on those pixels. In the special case of the first frame after a camera cut or scene load, all pixels can be classified in the second group. The third group of pixels in the segmented image are pixels where TAA has succeeded in its base case.

[0252] Since in the third group, frames are almost always dominated by TAA pixels, the cost of ray tracing is highly amortized and requires a ray budget much lower than one sample per pixel. For example, if 6% of the segmentations fall within the first group, then supersampling with 8x ray tracing can be adaptively used at a cost of less than 0.5 rays per pixel. However, the quality is comparable to 8x supersampling everywhere; otherwise, the boundaries between segmented regions would flicker in the final result due to the different algorithms used. There is no need to fill transitions at these boundaries, indicating that all three integration strategies converge to similar results under this segmentation approach.

[0253] In another embodiment, the first group can be ray traced. For example, rays can be cast with 8xMSAA n-rooks sub-pixel sampling mode, the same sampling used for 8xTAA. Temporal jitter may not be used. Ray casting is implemented as a DXR ray generation shader. When the ray hits, a completely UE4 node-based material graph and shading pipeline can be run directly using the same HLSL code as the raster pipeline. Since forward differential derivatives are not available in ray generation shaders, they can be treated as infinite to force the highest resolution of the texture. Therefore, supersampling can be used alone to resolve material aliasing (most film renderers do this to achieve the highest quality); another approach is to use distance and azimuth to select the MIP level analytically, or use ray differentials instead of raster differentials.

[0254] Pixels processed with FXAA are less expensive than those in the third group of pixels successfully processed by TAA, but much cheaper than those processed by supersampled ray tracing. FXAA performs well here because it does not require history data, and it runs on the low dynamic range post-contour map output to save memory bandwidth. By running FXAA only on deoccluded pixels, the cost is reduced compared to a typical full-screen application; typically 15% lower, even for fast objects and camera movements.

[0255] in conclusion

[0256] Primary surface aliasing is a fundamental problem in computer graphics. An exemplary solution for offline rendering is adaptive supersampling. This has previously been impractical for rasterized renderers in the context of complex materials and scenes, because there is no way to efficiently rasterize sparse pixels. Even the most efficient GPU ray tracers require duplication of shader and scene data. While DXR solves the technical challenges of combining rasterization and ray tracing, applying ray tracing to solve aliasing through supersampling is not trivial: knowing which pixels to supersample when only 1spp input is given, and reducing the cost to a scalable level are not something that can be solved by pure ray tracing. A practical solution to the problem is demonstrated; it is practical enough to run in commercial game engines, even in real time on the first generation of real-time ray tracing commodity hardware and software, and is connected to the entire shader pipeline. When the movie renderer selects pixels to adaptively supersample by first casting multiple rays at each pixel, it amortizes the cost over many frames by using the history buffer of the TAA to detect aliasing. Additionally, large, transient aliased areas are identified where deocclusion and post-processing FXAA are used, rather than dilating rays. This hybrid strategy uses the strengths of the most sophisticated real-time anti-aliasing strategies, but avoids their limitations. By feeding the supersampled results back into the TAA buffer, the probability that these pixels will not trigger supersampling on subsequent frames is increased, further reducing the cost.

[0257] Although various embodiments have been described above, it should be understood that they are presented by way of example only and not limitation. Therefore, the breadth and scope of the preferred embodiment should not be limited by any of the above exemplary embodiments, but should be defined only in accordance with the following claims and their equivalents.

Claims

1. A method comprising: Apply temporal anti-aliasing (TAA) to the image; identifying failed pixels resulting from applying the TAA to the image; as well as Anti-aliasing is performed on each of the failing pixels using a combination of ray tracing and Fast Approximate Anti-Aliasing (FXAA).

2. The method according to claim 1, wherein: The failing pixels are identified using the results of applying the TAA to the image, using intermediate data, or using source geometry.

3. The method of claim 1, wherein the image is created using forward rendering or deferred rendering.

4. The method of claim 1 , wherein the failed pixels are identified during the TAA, wherein: The TAA compares the historical value of a pixel in the previous frame with the current value of an adjacent pixel in the current frame, and The pixel having the historical value that differs from the current value of the neighboring pixel by more than a predetermined amount is identified as a failed pixel. The method of claim 1 , wherein the failed pixel is identified by comparing a depth value of the pixel with depth values ​​of neighboring pixels. The method of claim 1 , wherein the failed pixel is identified by comparing a brightness value of the pixel with brightness values ​​of adjacent pixels. The method of claim 1 , wherein the failed pixels are marked using a sentinel value. The method of claim 1 , wherein the failed pixels are included in a segmentation mask.

9. The method of claim 1, wherein performing the anti-aliasing comprises determining point-sampled visibility of sub-pixel regions within each of the failed pixels using GPU ray tracing.

10. The method of claim 1, wherein a result of performing the anti-aliasing on the failed pixels is combined with a result of applying the TAA to the image to create an output image.

Images