Method and storage medium for streaming light field compression using lossless or lossy compression

Abstract

Streaming a light field compressed using lossless or lossy compression is disclosed. A remote device uses ray tracing to compute a light field for a scene to be rendered, where the light field includes information about light reflected from surfaces within the scene. The light field is then compressed using lossless or lossy compression and one or more video compression techniques that implement time multiplexing, such that only the differences between the light field of the scene and the light field of the previous scene are compressed. The compressed light field data is then sent to a client device, which decompresses the light field data and uses it to obtain the light field of the scene at the client device. The light field is then used by the client device to compute global illumination for the scene. The global illumination can be used to accurately render the scene at the mobile device, resulting in a realistic scene rendered by the mobile device.

Description

[0001] Claim priority

[0002] This application claims the benefit of U.S. Provisional Application No. 63 / 109,204, filed on November 3, 2020, the entire contents of which are incorporated herein by reference. Technical Field

[0003] The present invention relates to image rendering and, more particularly, to performing global illumination on a scene. Background Art

[0004] Today's high-performance graphics systems for gaming include multiple GPUs, hardware-accelerated ray tracing, and efficient algorithms, and can use ray-traced global illumination to achieve near-film-quality rendering in real time while delivering interactive latency measured in milliseconds. These high-end systems set expectations that are currently difficult to meet on lower-end consumer platforms constrained by thermal limitations, battery power, and limited GPU feature sets.

[0005] Current cloud-based graphics solutions stream the entire content as fully rendered images from a remote server to the client. On the client, the video stream is decompressed and displayed on the monitor. However, network latency, bandwidth limitations, and increasing screen resolutions and frame rates threaten the future viability of this approach. Therefore, there is a need for distributed rendering systems with reduced bandwidth and latency. BRIEF DESCRIPTION OF THE DRAWINGS

[0006] Figure 1 A flow chart illustrating a method for streaming a compressed light field according to one embodiment is shown.

[0007] Figure 2 A parallel processing unit according to one embodiment is shown.

[0008] Figure 3A According to one embodiment, Figure 2 A general-purpose processing cluster within a parallel processing unit.

[0009] Figure 3B According to one embodiment, Figure 2 A memory partitioning unit of a parallel processing unit.

[0010] Figure 4A According to one embodiment, Figure 3A Streaming multiprocessor.

[0011] Figure 4B According to one embodiment, the Figure 2 Conceptual diagram of the processing system implemented by the PPU.

[0012] Figure 4C Illustrated are exemplary systems in which the various architecture and / or functionality of various previous embodiments may be implemented.

[0013] Figure 5 According to one embodiment, Figure 2 Conceptual diagram of the graphics processing pipeline implemented by the PPU.

[0014] Figure 6 is a block diagram of an example game streaming system suitable for implementing some embodiments of the present disclosure.

[0015] Figure 7 is a block diagram of an example computing device suitable for implementing some embodiments of the present disclosure.

[0016] Figure 8 A flow chart illustrating a method for streaming a light field compressed using lossless or lossy compression, according to one embodiment.

[0017] Figure 9 A flow chart of a method for receiving and decompressing a compressed light field compressed using color conversion is shown, according to one embodiment.

[0018] Figure 10 An exemplary distributed rendering system according to one embodiment is shown.

[0019] Figure 11 An exemplary data flow according to one embodiment is shown. DETAILED DESCRIPTION

[0020] A remote device (such as a server or one or more distributed computing nodes) uses ray tracing to calculate the light field of the scene to be rendered, where the light field includes information about light reflected from surfaces within the scene. The light field is then compressed in an efficient manner using one or more video compression techniques that implement temporal reuse, so that only the differences between the light field of the scene and the light field of the previous scene are compressed. This minimizes the amount of data to be sent. The compressed light field data is then sent to a client device (such as a mobile computing device), which decompresses the light field data and uses such data at the client device to obtain the light field of the scene. The light field is then used by the client device to calculate the global illumination of the scene. The global illumination can be used to accurately render the scene at the mobile device, resulting in a realistic scene presented to one or more users by the mobile device.

[0021] Figure 1A flow chart of method 100 for streaming a compressed light field, according to one embodiment, is shown. Although method 100 is described in the context of a processing unit, method 100 may also be performed by a program, custom circuitry, or a combination of custom circuitry and a program. For example, method 100 may be performed by a GPU (graphics processing unit), a CPU (central processing unit), or any processor capable of performing parallel path spatial filtering via hashing. Furthermore, one of ordinary skill in the art will appreciate that any system that performs method 100 is within the scope and spirit of embodiments of the present invention.

[0022] It should be understood that this and other arrangements described herein are only set forth as examples.Except those arrangements and elements shown or instead of those arrangements and elements shown, other arrangements and elements (such as, machines, interfaces, functions, sequences, functional groups etc.) can be used, and some elements can be omitted together. Further, many elements described herein are functional entities that can be implemented as discrete or distributed components or in combination with other components and in any suitable combination and position. The different functions described herein as being performed by an entity can be performed by hardware, firmware and / or software. For example, different functions can be performed by a processor that performs an instruction stored in a memory.

[0023] As shown in operation 102, a scene to be rendered is identified at a remote device. In one embodiment, the scene may include a still image, a frame within a video, a frame within a video game, etc. In another embodiment, the remote device may include a computing node within a distributed computing system, one or more nodes within a cloud-based computing environment, a server computing device, etc. In yet another embodiment, the remote device may be physically separate from the client device.

[0024] For example, the client device may display the rendered scene to one or more users (e.g., using one or more displays, etc.). In another example, the remote device and the client device may communicate via one or more wired and / or wireless communication networks (e.g., a wireless Internet connection, a cellular communication network, etc.).

[0025] Furthermore, as shown in operation 104, ray tracing is utilized at the remote device to calculate the light field of the scene. In one embodiment, the remote device may perform ray tracing within the scene to determine the light field of the scene. For example, ray tracing may include tracing the path of light as pixels within an image plane and simulating its effects upon objects within the scene.

[0026] Further, in one embodiment, ray tracing can be performed in real time. In one embodiment, the light field of a scene may include one or more vectors describing the amount of light flowing through one or more points in space within the scene in one or more directions. For example, the light field of a scene may include a vector function that describes the amount of light flowing through each point in space within the scene in each direction. In another embodiment, the light field may include multiple light probes. In yet another embodiment, each light probe within the light field may store information about light reflected from a surface within the scene. In yet another embodiment, the light field may include an array of blocks (e.g., multiple blocks of texel pixels) representing light probes.

[0027] Furthermore, in one embodiment, each block within the array may include color texture information and visibility texture information. For example, the color texture information may include lighting color information within the block. In another example, the visibility texture information may include distance information (e.g., the distance to the closest object / surface within the block). In another embodiment, the light field may be calculated using a processor (e.g., a graphics processing unit (GPU)).

[0028] Furthermore, as shown in operation 106, the light field is compressed at the remote device to create compressed light field data of the scene. In one embodiment, the remote device may compress the light field using a video encoding and / or video compression unit. For example, the unit may be located within a GPU.

[0029] Additionally, in one embodiment, the light field may be compressed using one or more video compression algorithms / techniques. For example, the light field may be compressed using GPU-accelerated high dynamic range (HDR) video compression implemented via a predetermined video coding / compression standard. In another embodiment, temporal multiplexing may be implemented during the compression of the light field. For example, information from the compression of a previously rendered scene may be used during the compression of the current scene. In another example, only the difference between the light field of a scene and the light field of the previous scene may be compressed / transmitted as light field data.

[0030] In this way, the temporal dependencies between consecutive scenes can be exploited to reduce the amount of data that needs to be compressed. This can improve compression performance (e.g., by reducing the time required to compress the light field) and can reduce the amount of data sent over the network to the client device when compared to current compression implementations that do not exploit temporal dependencies.

[0031] Furthermore, in one embodiment, compression can take into account the current user view of the scene. For example, the current user view can be received from a client device. In another example, only the portion of the light field that is within the current user view can be calculated. In yet another example, only the portion of the calculated light field that is within the current user view that is likely to be visible can be compressed to create the compressed light field data. This can reduce the amount of data being compressed, which can further improve the performance of the system performing the compression and can further reduce the amount of compressed data sent over the network to the client device.

[0032] Furthermore, the compressed light field data is transmitted from the remote device to the client device, as shown in operation 108. In one embodiment, the compressed light field data may be transmitted (e.g., streamed, etc.) from the remote device to the client device via one or more wired and / or wireless communication networks (e.g., a wireless Internet connection, a cellular communication network, etc.).

[0033] In this way, compressed light field data can be remotely calculated and provided to a client device, which can decompress the compressed light field data to obtain a decompressed light field of the scene. This decompressed light field can be used by the client device to perform global illumination of the scene at the client device. Implementing time multiplexing during light field compression can reduce the amount of data sent / bandwidth used over a communication network, which can improve the performance of computing devices implementing communication over such a network. This multiplexing can also result in reduced latency in both the creation and transmission of the compressed light field data. Furthermore, when calculating the light field for a scene, illumination calculations can be calculated dynamically and independently of a static scene or static lighting. This can effectively account for dynamic changes in scene geometry and lighting, and can improve the quality of rendered results.

[0034] Likewise, compressed light field data of the scene is received at the client device, as shown in operation 110. In one embodiment, the compressed light field data may be received at the client device from a remote device via one or more wired and / or wireless communication networks (e.g., a wireless Internet connection, a cellular communication network, etc.).

[0035] Furthermore, in one embodiment, the client device may include a device for at least partially rendering a scene and displaying the rendered scene. In another embodiment, the client device may include a mobile device (e.g., a smartphone, a tablet device, etc.), a portable gaming platform, a virtual reality (VR) head-mounted display, etc.

[0036] Furthermore, as shown in operation 112, the compressed light field data is decompressed at the client device to obtain the light field of the scene. In one embodiment, the client device may use a video decoder and / or a video decompression unit to decompress the compressed light field data. In another embodiment, the video decoder and / or the video decompression unit may comprise a dedicated hardware unit within the client device. In yet another embodiment, the video decoder and / or the video decompression unit may be located within the GPU of the client device. In another embodiment, the light field data of the previous scene may be combined with the decompressed light field data to obtain the light field of the current scene.

[0037] Furthermore, as shown in operation 114, global illumination of the scene is calculated at the client device using the light field of the scene. In one embodiment, when calculating global illumination, the client device may perform a lookup within the light field (e.g., instead of performing ray tracing or other resource-intensive operations). In another embodiment, performing global illumination may include modeling indirect light (e.g., how light reflects from surfaces onto other surfaces within the scene) as well as direct light (e.g., light directly from a light source hitting a surface).

[0038] Furthermore, in one embodiment, the calculated global illumination may be used to render the scene at the client device. In another embodiment, the rendered scene may be displayed by the client device (e.g., using a display of the client device, etc.).

[0039] In this way, the performance of global illumination of a scene can improve the realism of the scene as displayed by a client device. Additionally, global illumination can be performed more efficiently using light field lookups, which can reduce the amount of processing required at the client device. This can improve the performance of the client device when rendering the scene and can result in high-quality, dynamically rendered images with low bandwidth usage.

[0040] In yet another embodiment, a parallel processing unit (PPU) (e.g. Figure 2 ) to perform one or more parts of the light field computation and compression.

[0041] Now, more illustrative information will be provided regarding various optional architectures and features that may be employed to implement the aforementioned framework, depending on the user's desires. It should be noted that the following information is provided for illustrative purposes only and should not be construed as limiting in any way. The optional features below may optionally be incorporated with or without excluding the other features described.

[0042] Parallel processing architecture

[0043] Figure 2A parallel processing unit (PPU) 200 is shown according to one embodiment. In one embodiment, the PPU 200 is a multi-threaded processor implemented on one or more integrated circuit devices. The PPU 200 is a latency-hiding architecture designed for processing many threads in parallel. A thread (i.e., an execution thread) is an instance of an instruction set configured to be executed by the PPU 200. In one embodiment, the PPU 200 is a graphics processing unit (GPU) that is configured to implement a graphics rendering pipeline for processing three-dimensional (3D) graphics data to generate two-dimensional (2D) image data for display on a display device (such as a liquid crystal display (LCD) device). In other embodiments, the PPU 200 can be used to perform general-purpose computations. Although an exemplary parallel processor is provided herein for illustrative purposes, it should be specifically noted that the processor is described for illustrative purposes only and any processor may be used in addition to and / or in place of the processor.

[0044] One or more PPUs 200 can be configured to accelerate thousands of high-performance computing (HPC), data center, and machine learning applications. PPUs 200 can be configured to accelerate numerous deep learning systems and applications, including autonomous vehicle platforms, deep learning, high-precision speech, image, and text recognition systems, intelligent video analysis, molecular simulations, drug discovery, disease diagnosis, weather forecasting, big data analytics, astronomy, molecular dynamics simulations, financial modeling, robotics, factory automation, real-time language translation, online search optimization, and personalized user recommendations, among others.

[0045] like Figure 2 As shown, the PPU 200 includes an input / output (I / O) unit 205, a front-end unit 215, a scheduler unit 220, a work distribution unit 225, a hub 230, a crossbar switch (Xbar) 270, one or more general processing clusters (GPCs) 250, and one or more partition units 280. The PPU 200 can be connected to a host processor or other PPUs 200 via one or more high-speed NVLink 210 interconnects. The PPU 200 can be connected to a host processor or other peripheral devices via interconnect 202. The PPU 200 can also be connected to a local memory including a plurality of memory devices 204. In one embodiment, the local memory can include a plurality of dynamic random access memory (DRAM) devices. The DRAM devices can be configured as a high-bandwidth memory (HBM) subsystem in which multiple DRAM dies are stacked within each device.

[0046] The NVLink 210 interconnect enables the system to scale and include one or more PPUs 200 in conjunction with one or more CPUs, supporting cache coherency between the PPU 200 and the CPU, and CPU mastering. Data and / or commands can be sent by the NVLink 210 through the hub 230 to or from other units of the PPU 200, such as one or more copy engines, video encoders, video decoders, power management units, etc. (not explicitly shown). Figure 4B NVLink 210 is described in more detail.

[0047] I / O unit 205 is configured to send and receive communications (i.e., commands, data, etc.) from a host processor (not shown) via interconnect 202. I / O unit 205 can communicate with the host processor directly via interconnect 202, or through one or more intermediary devices (such as a memory bridge). In one embodiment, I / O unit 205 can communicate with one or more other processors (e.g., one or more PPUs 200) via interconnect 202. In one embodiment, I / O unit 205 implements a Peripheral Component Interconnect Express (PCIe) interface for communicating over a PCIe bus, and interconnect 202 is a PCIe bus. In alternative embodiments, I / O unit 205 can implement other types of known interfaces for communicating with external devices.

[0048] I / O unit 205 decodes data packets received via interconnect 202. In one embodiment, the data packets represent commands configured to cause PPU 200 to perform various operations. I / O unit 205 sends the decoded commands to various other units of PPU 200 as specified by the commands. For example, some commands may be sent to front-end unit 215. Other commands may be sent to hub 230 or other units of PPU 200, such as one or more copy engines, video encoders, video decoders, power management units, etc. (not explicitly shown). In other words, I / O unit 205 is configured to route communications between and among the various logical units of PPU 200.

[0049] In one embodiment, a program executed by a host processor encodes a command stream in a buffer that provides a workload to the PPU 200 for processing. The workload may include many instructions and data to be processed by those instructions. A buffer is an area of ​​memory that is accessible (i.e., read / write) by both the host processor and the PPU 200. For example, the I / O unit 205 may be configured to access a buffer in system memory connected to the interconnect 202 via a memory request transmitted over the interconnect 202. In one embodiment, the host processor writes a command stream into the buffer and then sends a pointer to the start of the command stream to the PPU 200. The front end unit 215 receives pointers to one or more command streams. The front end unit 215 manages the one or more streams, reads commands from the streams, and forwards the commands to the various units of the PPU 200.

[0050] The front-end unit 215 is coupled to a scheduler unit 220, which configures the various GPCs 250 to process tasks defined by one or more streams. The scheduler unit 220 is configured to track state information associated with the various tasks managed by the scheduler unit 220. The state may indicate which GPC 250 the task is assigned to, whether the task is active or inactive, the priority associated with the task, and the like. The scheduler unit 220 manages the execution of multiple tasks on one or more GPCs 250.

[0051] Scheduler unit 220 is coupled to work distribution unit 225, which is configured to dispatch tasks for execution on GPCs 250. Work distribution unit 225 may track a number of scheduled tasks received from scheduler unit 220. In one embodiment, work distribution unit 225 manages a pending task pool and an active task pool for each GPC 250. The pending task pool may include a number of time slots (e.g., 32 time slots) that include tasks assigned to be processed by a particular GPC 250. The active task pool may include a number of time slots (e.g., 4 time slots) for tasks that are being actively processed by GPC 250. When a GPC 250 completes execution of a task, the task is evicted from the active task pool of GPC 250, and one of the other tasks from the pending task pool is selected and scheduled for execution on GPC 250. If an active task on GPC 250 has become idle, such as while waiting for a data dependency to be resolved, the active task may be evicted from GPC 250 and returned to the pending task pool, while another task in the pending task pool is selected and scheduled for execution on GPC 250 .

[0052] Work distribution unit 225 communicates with one or more GPCs 250 via XBar (crossbar) 270. XBar 270 is an interconnect network that couples many units of PPU 200 to other units of PPU 200. For example, XBar 270 can be configured to couple work distribution unit 225 to a specific GPC 250. Although not explicitly shown, one or more other units of PPU 200 can also be connected to XBar 270 via hub 230.

[0053] Tasks are managed by the scheduler unit 220 and dispatched to the GPCs 250 by the work distribution unit 225. The GPCs 250 are configured to process tasks and generate results. The results can be consumed by other tasks within the GPC 250, routed to different GPCs 250 via the XBar 270, or stored in the memory 204. The results can be written to the memory 204 via the partition unit 280, which implements a memory interface for reading data from and writing data to the memory 204. The results can be sent to another PPU 200 or CPU via the NVLink 210. In one embodiment, the PPU 200 includes a number U of partition units 280, which is equal to the number of independent and different memory devices 204 coupled to the PPU 200. Figure 3B The partition unit 280 is described in more detail.

[0054] In one embodiment, the host processor executes a driver kernel that implements an application programming interface (API) that enables one or more applications to be executed on the host processor to schedule operations for execution on the PPU 200. In one embodiment, multiple computing applications are executed simultaneously by the PPU 200, and the PPU 200 provides isolation, quality of service (QoS), and independent address spaces for the multiple computing applications. The application can generate instructions (e.g., API calls) that cause the driver kernel to generate one or more tasks to be executed by the PPU 200. The driver kernel outputs the tasks to one or more streams being processed by the PPU 200. Each task can include one or more related groups of threads, referred to herein as warps. In one embodiment, a warp includes 32 related threads that can execute in parallel. Collaborating threads can refer to multiple threads that include instructions to perform tasks and that can exchange data through shared memory. In combination Figure 4A Describes threads and cooperative threads in more detail.

[0055] Figure 3A According to one embodiment, Figure 2 PPU 200 GPC 250. Figure 3AAs shown, each GPC 250 includes multiple hardware units for processing tasks. In one embodiment, each GPC 250 includes a pipeline manager 310, a pre-raster operation unit (PROP) 315, a raster engine 325, a work distribution crossbar (WDX) 380, a memory management unit (MMU) 390, and one or more data processing clusters (DPCs) 320. It should be understood that Figure 3A The GPC 250 may include instead Figure 3A Other hardware units or other than the units shown in Figure 3A Other hardware units besides those shown in .

[0056] In one embodiment, the operation of GPC 250 is controlled by pipeline manager 310. Pipeline manager 310 manages the configuration of one or more DPCs 320 for processing tasks assigned to GPC 250. In one embodiment, pipeline manager 310 can configure at least one of one or more DPCs 320 to implement at least a portion of a graphics rendering pipeline. For example, DPC 320 can be configured to execute vertex shading programs on programmable streaming multiprocessors (SMs) 340. Pipeline manager 310 can also be configured to route packets received from work distribution unit 225 to appropriate logic units within GPC 250. For example, some packets can be routed to fixed-function hardware units within PROP 315 and / or raster engine 325, while other packets can be routed to DPC 320 for processing by primitive engine 335 or SM 340. In one embodiment, pipeline manager 310 can configure at least one of one or more DPCs 320 to implement a neural network model and / or a compute pipeline.

[0057] PROP unit 315 is configured to route data generated by raster engine 325 and DPC 320 to the raster operations (ROP) unit, in conjunction with Figure 3B The PROP unit 315 may also be configured to perform color blending optimizations, organize pixel data, perform address translations, and the like.

[0058] The raster engine 325 includes several fixed-function hardware units configured to perform various raster operations. In one embodiment, the raster engine 325 includes a setup engine, a coarse raster engine, a culling engine, a clipping engine, a fine raster engine, and a tile aggregation engine. The setup engine receives the transformed vertices and generates plane equations associated with the geometric primitives defined by the vertices. The plane equations are sent to the coarse raster engine to generate coverage information for the primitives (e.g., the x, y coverage mask of the tile). The output of the coarse raster engine is sent to the culling engine, where fragments associated with primitives that fail the z-test are culled, and to the clipping engine, where fragments outside the viewing frustum are clipped. Those fragments that remain after clipping and culling can be passed to the fine raster engine to generate attributes for the pixel fragments based on the plane equations generated by the setup engine. The output of the raster engine 325 includes, for example, fragments to be processed by the fragment shader implemented in the DPC 320.

[0059] Each DPC 320 included in the GPC 250 includes an M-pipe controller (MPC) 330, a primitive engine 335, and one or more SMs 340. The MPC 330 controls the operation of the DPC 320 and routes packets received from the pipeline manager 310 to appropriate units in the DPC 320. For example, packets associated with vertices may be routed to the primitive engine 335, which is configured to fetch vertex attributes associated with the vertices from the memory 204. Conversely, packets associated with shading programs may be sent to the SMs 340.

[0060] SM 340 includes a programmable streaming processor configured to process tasks represented by multiple threads. Each SM 340 is multi-threaded and configured to simultaneously execute multiple threads (e.g., 32 threads) from a particular thread group. In one embodiment, SM 340 implements a SIMD (single instruction, multiple data) architecture, in which each thread in a thread group (i.e., a warp) is configured to process a different data set based on the same instruction set. All threads in a thread group execute the same instructions. In another embodiment, SM 340 implements a SIMT (single instruction, multiple thread) architecture, in which each thread in a thread group is configured to process a different data set based on the same instruction set, but in which individual threads in a thread group are allowed to diverge during execution. In one embodiment, a program counter, call stack, and execution state are maintained for each warp, enabling concurrency between warps and serial execution within a warp when threads within the warp diverge. In another embodiment, a program counter, call stack, and execution state are maintained for each individual thread, thereby achieving equal concurrency between all threads within and between warps. When execution state is maintained for each individual thread, threads executing the same instruction can be converged and executed in parallel for maximum efficiency. Figure 4A SM 340 is described in more detail.

[0061] The MMU 390 provides an interface between the GPC 250 and the partition unit 280. The MMU 390 can provide virtual to physical address translation, memory protection, and arbitration of memory requests. In one embodiment, the MMU 390 provides one or more translation lookaside buffers (TLBs) for performing translations from virtual addresses to physical addresses in the memory 204.

[0062] Figure 3B According to one embodiment, Figure 2 The memory partition unit 280 of the PPU 200. Figure 3B As shown, the memory partition unit 280 includes a raster operations (ROP) unit 350, a level 2 (L2) cache 360, and a memory interface 370. The memory interface 370 is coupled to the memory 204. The memory interface 370 can implement a 32-, 64-, 128-, or 1024-bit data bus for high-speed data transfer. In one embodiment, the PPU 200 incorporates U memory interfaces 370, one for each pair of partition units 280, where each pair of partition units 280 is connected to a corresponding memory device 204. For example, the PPU 200 can be connected to up to Y memory devices 204, such as a high-bandwidth memory stack or graphics double data rate version 5 synchronous dynamic random access memory or other types of persistent memory.

[0063] In one embodiment, memory interface 370 implements an HBM2 memory interface, and Y is equal to half of U. In one embodiment, the HBM2 memory stack is located on the same physical package as PPU 200, providing significant power and area savings compared to conventional GDDR5 SDRAM systems. In one embodiment, each HBM2 stack includes four memory dies and Y is equal to 4, where the HBM2 stack includes two 128-bit channels per die, for a total of 8 channels and a data bus width of 1024 bits.

[0064] In one embodiment, memory 204 supports single-error correction, double-error detection (SECDED) error correction code (ECC) to protect data. ECC provides increased reliability for computing applications sensitive to data corruption. Reliability is particularly important in large cluster computing environments where PPU 200 processes very large data sets and / or long-running applications.

[0065] In one embodiment, the PPU 200 implements a multi-level memory hierarchy. In one embodiment, the memory partitioning unit 280 supports unified memory to provide a single, unified virtual address space for the CPU and PPU 200 memory, enabling data sharing between virtual memory systems. In one embodiment, the frequency of PPU 200 accesses to memory located on other processors is tracked to ensure that memory pages are moved to the physical memory of the PPU 200 that accesses them more frequently. In one embodiment, NVLink 210 supports address translation services that allow the PPU 200 to directly access the CPU's page tables and provide full access to the CPU's memory by the PPU 200.

[0066] In one embodiment, the copy engine transfers data between multiple PPUs 200 or between a PPU 200 and a CPU. The copy engine can generate a page fault for an address that is not mapped to a page table. The memory partition unit 280 can then service the page fault, map the address into a page table, and then the copy engine can perform the transfer. In conventional systems, multiple copy engines operate on fixed memory (i.e., non-pageable) between multiple processors, which significantly reduces the available memory. Due to hardware page faults, addresses can be passed to the copy engine without worrying about whether the memory page is resident, and the copy process is transparent.

[0067] Data from memory 204 or other system memory can be retrieved by memory partition unit 280 and stored in L2 cache 360, which is located on-chip and shared between various GPCs 250. As shown, each memory partition unit 280 includes a portion of L2 cache 360 ​​associated with the corresponding memory device 204. Lower-level caches can then be implemented in multiple units within a GPC 250. For example, each SM 340 can implement a level 1 (L1) cache. The L1 cache is a dedicated memory dedicated to a particular SM 340. Data from the L2 cache 360 ​​can be retrieved and stored in each L1 cache for processing in the functional units of the SM 340. The L2 cache 360 ​​is coupled to the memory interface 370 and the XBar 270.

[0068] The ROP unit 350 performs graphics raster operations related to pixel color such as color compression, pixel blending, etc. The ROP unit 350 also implements depth testing in conjunction with the raster engine 325, receiving the depth of the sample position associated with the pixel fragment from the culling engine of the raster engine 325. The depth of the sample position associated with the fragment is tested against the corresponding depth in the depth buffer. If the fragment passes the depth test for the sample position, the ROP unit 350 updates the depth buffer and sends the result of the depth test to the raster engine 325. It will be understood that the number of partition units 280 can be different than the number of GPCs 250, and therefore each ROP unit 350 can be coupled to each GPC 250. The ROP unit 350 tracks packets received from different GPCs 250 and determines to which GPC 250 the results generated by the ROP unit 350 are routed via the Xbar 270. Although in Figure 3B In some embodiments, ROP unit 350 is included within memory partition unit 280, but in other embodiments, ROP unit 350 may be external to memory partition unit 280. For example, ROP unit 350 may reside in GPC 250 or another unit.

[0069] Figure 4A According to one embodiment, Figure 3A The streaming multiprocessor 340. Figure 4A As shown, SM 340 includes an instruction cache 405, one or more scheduler units 410(K), a register file 420, one or more processing cores 450, one or more special function units (SFUs) 452, one or more load / store units (LSUs) 454, an interconnection network 480, and a shared memory / L1 cache 470.

[0070] As described above, the work distribution unit 225 schedules tasks for execution on the GPCs 250 of the PPU 200. Tasks are assigned to specific DPCs 320 within the GPCs 250 and, if the task is associated with a shader program, may be assigned to an SM 340. The scheduler unit 410(K) receives tasks from the work distribution unit 225 and manages the scheduling of instructions for one or more thread blocks assigned to the SMs 340. The scheduler unit 410(K) schedules the thread blocks for execution as warps of parallel threads, where each thread block is assigned at least one warp. In one embodiment, each warp executes 32 threads. The scheduler unit 410(K) can manage multiple different thread blocks, assign warps to different thread blocks, and then dispatch instructions from multiple different cooperative groups to various functional units (i.e., cores 450, SFUs 452, and LSUs 454) during each clock cycle.

[0071] Cooperative Groups is a programming model for organizing groups of communicating threads that allows developers to express the granularity at which threads are communicating, enabling the expression of richer and more efficient decompositions of parallelism. The cooperative launch API supports synchronization between thread blocks to execute parallel algorithms. The conventional programming model provides a single simple structure for synchronizing cooperating threads: a barrier across all threads of a thread block (i.e., the syncthreads() function). However, programmers often want to define thread groups at a granularity smaller than the thread block granularity and synchronize within the defined group, enabling higher performance, design flexibility, and software reuse in the form of a collective group-wide function interface.

[0072] Cooperative Groups enable programmers to explicitly define thread groups at sub-block (i.e., as small as a single thread) and multi-block granularity and perform collective operations, such as synchronization, on threads in a cooperative group. The programming model supports clean composition across software boundaries so that libraries and utility functions can safely synchronize in their local environment without making assumptions about convergence. Cooperative Group primitives enable new patterns of cooperative parallelism, including producer-consumer parallelism, opportunistic parallelism, and global synchronization across the entire grid of thread blocks.

[0073] The dispatch unit 415 is configured to deliver instructions to one or more functional units. In this embodiment, the scheduler unit 410(K) includes two dispatch units 415, which enable scheduling of two different instructions from the same warp during each clock cycle. In alternative embodiments, each scheduler unit 410(K) may include a single dispatch unit 415 or additional dispatch units 415.

[0074] Each SM 340 includes a register file 420 that provides a set of registers for the functional units of the SM 340. In one embodiment, register file 420 is divided between each functional unit so that each functional unit is allocated a dedicated portion of register file 420. In another embodiment, register file 420 is divided between the different warps executed by the SM 340. Register file 420 provides temporary storage for operands connected to the data paths of the functional units.

[0075] Each SM 340 includes L processing cores 450. In one embodiment, the SM 340 includes a large number (e.g., 128, etc.) of different processing cores 450. Each core 450 may include a fully pipelined, single-precision, double-precision, and / or mixed-precision processing unit, including a floating-point arithmetic logic unit (FLU) and an integer arithmetic logic unit (ALU). In one embodiment, the FLU implements the IEEE 754-2008 standard for floating-point operations. In one embodiment, the cores 450 include 64 single-precision (32-bit) floating-point cores, 64 integer cores, 32 double-precision (64-bit) floating-point cores, and 8 tensor cores.

[0076] Tensor cores are configured to perform matrix operations, and in one embodiment, one or more tensor cores are included in core 450. Specifically, the tensor cores are configured to perform deep learning matrix operations, such as convolution operations for neural network training and inference. In one embodiment, each tensor core operates on a 4×4 matrix and performs a matrix multiplication and accumulation operation D=A×B+C, where A, B, C, and D are 4×4 matrices.

[0077] In one embodiment, the matrix multiplication inputs A and B are 16-bit floating point matrices, while the accumulation matrices C and D can be 16-bit floating point or 32-bit floating point matrices. The tensor cores operate on 16-bit floating point input data as well as 32-bit floating point accumulations. The 16-bit floating point multiplication requires 64 operations to produce a full-precision product, which is then accumulated using 32-bit floating point additions with other intermediate products of the 4×4×4 matrix multiplication. In practice, tensor cores are used to perform larger two-dimensional or higher-dimensional matrix operations built from these smaller elements. APIs (such as the CUDA9 C++ API) expose specialized matrix load, matrix multiplication and accumulation, and matrix store operations to efficiently use tensor cores from CUDA-C++ programs. At the CUDA level, the warp-level interface assumes that a 16×16 size matrix spans all 32 threads of the warp.

[0078] Each SM 340 also includes M SFUs 452 that perform special functions (e.g., attribute evaluation, reciprocal square root, etc.). In one embodiment, the SFUs 452 may include a tree traversal unit configured to traverse a hierarchical tree data structure. In one embodiment, the SFUs 452 may include a texture unit configured to perform texture map filtering operations. In one embodiment, the texture unit is configured to load a texture map (e.g., a 2D array of texels) from memory 204 and sample the texture map to generate sampled texture values ​​for use in shader programs executed by the SM 340. In one embodiment, the texture map is stored in shared memory / L1 cache 370. The texture unit implements texture operations, such as filtering operations using mip maps (i.e., texture maps at different levels of detail). In one embodiment, each SM 340 includes two texture units.

[0079] Each SM 340 also includes N LSUs 454, which implement load and store operations between shared memory / L1 cache 470 and register file 420. Each SM 340 includes an interconnect network 480 that connects each functional unit to register file 420 and connects the LSUs 454 to register file 420 and shared memory / L1 cache 470. In one embodiment, interconnect network 480 is a crossbar switch that can be configured to connect any functional unit to any register in register file 420 and to connect the LSUs 454 to memory locations in the register file and shared memory / L1 cache 470.

[0080] Shared memory / L1 cache 470 is an on-chip memory array that allows data storage and communication between the SM 340 and the primitive engine 335, as well as between threads in the SM 340. In one embodiment, shared memory / L1 cache 470 includes 128KB of storage capacity and is in the path from the SM 340 to the partition unit 280. Shared memory / L1 cache 470 can be used to cache reads and writes. One or more of the shared memory / L1 cache 470, L2 cache 360, and memory 204 is a backing store.

[0081] Combining data cache and shared memory functionality into a single memory block provides the best overall performance for both types of memory access. This capacity can be used by programs as a cache that does not utilize the shared memory. For example, if the shared memory is configured to use half of its capacity, texture and load / store operations can use the remaining capacity. Integration within shared memory / L1 cache 470 enables shared memory / L1 cache 470 to function as a high-throughput pipeline for streaming data, while providing high-bandwidth and low-latency access to frequently reused data.

[0082] When configured for general parallel computing, a simpler configuration can be used compared to graphics processing. Specifically, Figure 2 The fixed-function graphics processing unit is shown bypassed, creating a simpler programming model. In a general-purpose parallel computing configuration, the work distribution unit 225 assigns and distributes thread blocks directly to the DPC 320. The threads in the block execute the same program, using unique thread IDs in computations to ensure that each thread produces unique results, using the SM 340 to execute the program and perform computations, using the shared memory / L1 cache 470 to communicate between threads, and using the LSU 454 to read and write global memory through the shared memory / L1 cache 470 and the memory partition unit 280. When configured for general-purpose parallel computing, the SM 340 can also write commands that the scheduler unit 220 can use to start new work on the DPC 320.

[0083] The PPU 200 may be included in a desktop computer, a laptop computer, a tablet computer, a server, a supercomputer, a smartphone (e.g., wireless, handheld device), a personal digital assistant (PDA), a digital camera, a vehicle, a head-mounted display, a handheld electronic device, etc. In one embodiment, the PPU 200 is included on a single semiconductor substrate. In another embodiment, the PPU 200 is included on a system-on-chip (SoC) along with one or more other devices (such as an additional PPU 200, memory 204, a reduced instruction set computer (RISC) CPU, a memory management unit (MMU), a digital-to-analog converter (DAC), etc.).

[0084] In one embodiment, PPU 200 may be included on a graphics card that includes one or more memory devices 204. The graphics card may be configured to interface with a PCIe slot on a motherboard of a desktop computer. In another embodiment, PPU 200 may be an integrated graphics processing unit (iGPU) or parallel processor included in a chipset of the motherboard.

[0085] Exemplary Computing System

[0086] Systems with multiple GPUs and CPUs are being used across various industries as developers expose and exploit greater parallelism in applications such as artificial intelligence computing. High-performance GPU-accelerated systems with tens to thousands of computing nodes are deployed in data centers, research institutions, and supercomputers to solve larger problems. As the number of processing devices within high-performance systems increases, communication and data transmission mechanisms need to scale to support this increased bandwidth.

[0087] Figure 4B According to one embodiment, the Figure 2 Conceptual diagram of a processing system 400 implemented by a PPU 200. The exemplary system 400 may be configured to implement Figure 1 The processing system 400 includes a CPU 430, a switch 410, and each of a plurality of PPUs 200 and a corresponding memory 204. The NVLink 210 provides a high-speed communication link between each PPU 200. Figure 4B A specific number of NVLink 210 and interconnect 202 connections are shown in FIG, but the number of connections connected to each PPU 200 and CPU 430 can vary. Switch 410 interfaces between interconnect 202 and CPU 430. PPU 200, memory 204, and NVLink 210 can be located on a single semiconductor platform to form a parallel processing module 425. In one embodiment, switch 410 supports two or more protocols that interface between various different connections and / or links.

[0088] In another embodiment (not shown), NVLink 210 provides one or more high-speed communication links between each PPU 200 and CPU 430, and switch 410 interfaces between interconnect 202 and each PPU 200. The PPUs 200, memory 204, and interconnect 202 may be located on a single semiconductor platform to form a parallel processing module 425. In yet another embodiment (not shown), interconnect 202 provides one or more communication links between each PPU 200 and CPU 430, and switch 410 interfaces between each PPU 200 using NVLink 210 to provide one or more high-speed communication links between the PPUs 200. In another embodiment (not shown), NVLink 210 provides one or more high-speed communication links between the PPUs 200 and CPU 430 through switch 410. In yet another embodiment (not shown), interconnect 202 provides one or more communication links directly between each PPU 200. One or more NVLink 210 high-speed communication links may be implemented as a physical NVLink interconnect or as an on-chip or on-die interconnect using the same protocol as NVLink 210 .

[0089] In the context of this specification, a single semiconductor platform may refer to a unique, single semiconductor-based integrated circuit fabricated on a die or chip. It should be noted that the term single semiconductor platform may also refer to a multi-chip module with increased connectivity that emulates on-chip operation and is substantially improved by utilizing conventional bus implementations. Of course, various circuits or devices may also be placed separately or in various combinations of semiconductor platforms, depending on the needs of the user. Alternatively, the parallel processing module 425 may be implemented as a circuit board substrate, and each of the PPU 200 and / or memory 204 may be a packaged device. In one embodiment, the CPU 430, switch 410, and parallel processing module 425 are located on a single semiconductor platform.

[0090] In one embodiment, the signaling rate of each NVLink 210 is 20 to 25 Gbit / s, and each PPU 200 includes six NVLink 210 interfaces (e.g., Figure 4B As shown, each PPU 200 includes five NVLink 210 interfaces. Each NVLink 210 provides a data transfer rate of 25 Gbit / s in each direction, with six links providing 300 Gbit / s. When the CPU 430 also includes one or more NVLink 210 interfaces, the NVLink 210 can be used exclusively for Figure 4B PPU to PPU communication shown, or some combination of PPU to PPU and PPU to CPU.

[0091] In one embodiment, NVLink 210 allows direct load / store / atomic access from CPU 430 to memory 204 of each PPU 200. In one embodiment, NVLink 210 supports coherency operations, allowing data read from memory 204 to be stored in the cache hierarchy of CPU 430, reducing cache access latency for CPU 430. In one embodiment, NVLink 210 includes support for Address Translation Services (ATS), allowing PPU 200 to directly access page tables within CPU 430. One or more NVLinks 210 can also be configured to operate in a low-power mode.

[0092] Figure 4C An exemplary system 465 is shown in which various architectures and / or functionalities of various previous embodiments may be implemented. The exemplary system 465 may be configured to implement Figure 1 The method 100 shown in FIG.

[0093] As shown, a system 465 is provided that includes at least one central processing unit 430 connected to a communication bus 475. The communication bus 475 can be implemented using any suitable protocol, such as PCI (Peripheral Component Interconnect), PCI-Express, AGP (Accelerated Graphics Port), HyperTransport, or any other bus or one or more point-to-point communication protocols. The system 465 also includes a main memory 440. Control logic (software) and data are stored in the main memory 440, which can take the form of random access memory (RAM).

[0094] System 465 also includes input device 460, parallel processing system 425, and display device 445, i.e., a conventional CRT (cathode ray tube), LCD (liquid crystal display), LED (light emitting diode), plasma display, etc. User input can be received from input device 460 (e.g., keyboard, mouse, touchpad, microphone, etc.). Each of the aforementioned modules and / or devices can even be located on a single semiconductor platform to form system 465. Alternatively, the modules can be located separately or in various combinations of semiconductor platforms, depending on user needs.

[0095] Furthermore, system 465 can be coupled to a network (e.g., a telecommunications network, a local area network (LAN), a wireless network, a wide area network (WAN) such as the Internet, a peer-to-peer network, a cable network, etc.) through network interface 435 for communication purposes.

[0096] System 465 may also include auxiliary storage (not shown). Auxiliary storage includes, for example, a hard drive and / or a removable storage drive, representative of a floppy disk drive, a tape drive, an optical drive, a digital versatile disk (DVD) drive, a recording device, and a universal serial bus (USB) flash memory. The removable storage drive reads from and / or writes to the removable storage unit in a well-known manner.

[0097] Computer programs or computer control logic algorithms may be stored in the main memory 440 and / or the secondary storage. These computer programs, when executed, enable the system 465 to perform various functions. The memory 440, storage, and / or any other storage are possible examples of computer-readable media.

[0098] The architecture and / or functionality of the various preceding figures can be implemented in the context of a general-purpose computer system, a circuit board system, a game console system dedicated for entertainment purposes, a dedicated system, and / or any other desired system. For example, system 465 can take the form of a desktop computer, a laptop computer, a tablet computer, a server, a supercomputer, a smartphone (e.g., a wireless, handheld device), a personal digital assistant (PDA), a digital camera, a vehicle, a head-mounted display, a handheld electronic device, a mobile telephone device, a television, a workstation, a game console, an embedded system, and / or any other type of logic.

[0099] Although various embodiments have been described above, it should be understood that they are presented by way of example only, not limitation. Therefore, the breadth and scope of a preferred embodiment should not be limited by any of the above-described exemplary embodiments, but should be defined only in accordance with the following claims and their equivalents.

[0100] Graphics processing pipeline

[0101] In one embodiment, the PPU 200 comprises a graphics processing unit (GPU). The PPU 200 is configured to receive commands specifying a shader for processing graphics data. Graphics data can be defined as a set of primitives, such as points, lines, triangles, quadrilaterals, triangle strips, and the like. Typically, a primitive includes data specifying a plurality of vertices of the primitive (e.g., in a model space coordinate system) and attributes associated with each vertex of the primitive. The PPU 200 can be configured to process the primitives to generate a frame buffer (i.e., pixel data for each of the pixels of the display).

[0102] An application writes model data for a scene (e.g., a collection of vertices and attributes) to memory (such as system memory or memory 204). The model data defines each of the objects that may be visible on the display. The application then makes an API call to the driver kernel, requesting the model data to be rendered and displayed. The driver kernel reads the model data and writes commands to one or more streams to perform operations to process the model data. These commands may reference different shading programs to be implemented on the SMs 340 of the PPU 200, including one or more of vertex shading, hull shading, domain shading, geometry shading, and pixel shading. For example, one or more of the SMs 340 may be configured to execute a vertex shading program that processes multiple vertices defined by the model data. In one embodiment, different SMs 340 may be configured to execute different shading programs simultaneously. For example, a first subset of SMs 340 may be configured to execute a vertex shading program, while a second subset of SMs 340 may be configured to execute a pixel shading program. The first subset of SMs 340 processes the vertex data to generate processed vertex data and writes the processed vertex data to the L2 cache 360 ​​and / or memory 204. After the processed vertex data is rasterized (e.g., converted from three-dimensional data to two-dimensional data in screen space) to generate fragment data, a second subset of SMs 340 performs pixel shading to generate processed fragment data, which is then blended with other processed fragment data and written to a frame buffer in memory 204. Vertex shading programs and pixel shading programs can be executed simultaneously, processing different data from the same scene in a pipelined manner, until all model data for the scene has been rendered to the frame buffer. The contents of the frame buffer are then transmitted to a display controller for display on a display device.

[0103] Figure 5 According to one embodiment, Figure 2 2. A conceptual diagram of a graphics processing pipeline 500 implemented by the PPU 200 of FIG. The graphics processing pipeline 500 is an abstract flow diagram of the processing steps implemented to generate a 2D computer-generated image from 3D geometric data. As is well known, pipeline architectures can perform long-latency operations more efficiently by dividing the operations into multiple stages, where the output of each stage is coupled to the input of the next consecutive stage. Thus, the graphics processing pipeline 500 receives input data 501 that is passed from one stage of the graphics processing pipeline 500 to the next stage to generate output data 502. In one embodiment, the graphics processing pipeline 500 may represent a graphics processing pipeline composed of API-defined graphics processing pipeline. Alternatively, graphics processing pipeline 500 can be implemented within the functional and architectural context of the previous figures and / or one or more of any subsequent figures.

[0104] like Figure 5As shown, graphics processing pipeline 500 includes a pipeline architecture comprising multiple stages. These stages include, but are not limited to, a data assembly stage 510, a vertex shading stage 520, a primitive assembly stage 530, a geometry shading stage 540, a viewport scale, cull, and clip (VSCC) stage 550, a rasterization stage 560, a fragment shading stage 570, and a raster operation stage 580. In one embodiment, input data 501 includes commands that configure a processing unit to implement the stages of graphics processing pipeline 500 and configure geometric primitives (e.g., points, lines, triangles, quads, triangle strips, or fans, etc.) to be processed by these stages. Output data 502 may include pixel data (i.e., color data), which is copied to a frame buffer or other type of surface data structure in memory.

[0105] The data assembly stage 510 receives input data 501, which specifies vertex data for high-level surfaces, primitives, etc. The data assembly stage 510 collects the vertex data in temporary storage or queues, such as by receiving a command from the host processor including a pointer to a buffer in memory and reading the vertex data from the buffer. The vertex data is then passed to the vertex shading stage 520 for processing.

[0106] The vertex shading stage 520 processes vertex data by executing a set of operations (i.e., a vertex shader or program) once for each vertex. A vertex may be specified, for example, as a 4-coordinate vector (i.e.,<x,y,z,w> ). The vertex shading stage 520 can manipulate various vertex attributes, such as position, color, texture coordinates, etc. In other words, the vertex shading stage 520 performs operations on the vertex coordinates or other vertex attributes associated with the vertex. These operations typically include lighting operations (i.e., modifying the color attributes of the vertex) and transformation operations (i.e., modifying the coordinate space of the vertex). For example, a vertex can be specified using coordinates in object coordinate space, which are transformed by multiplying the coordinates by a matrix that converts the coordinates from the object coordinate space to world space or normalized-device-coordinate (NCD) space. The vertex shading stage 520 generates transformed vertex data that is passed to the primitive assembly stage 530.

[0107] The primitive assembly stage 530 collects the vertices output by the vertex shading stage 520 and groups the vertices into geometric primitives for processing by the geometry shading stage 540. For example, the primitive assembly stage 530 can be configured to group every three consecutive vertices into geometric primitives (i.e., triangles) for transmission to the geometry shading stage 540. In some embodiments, particular vertices can be reused for consecutive geometric primitives (e.g., two consecutive triangles in a triangle strip can share two vertices). The primitive assembly stage 530 transmits the geometric primitives (i.e., the collection of associated vertices) to the geometry shading stage 540.

[0108] The geometry shading stage 540 processes geometric primitives by executing a set of operations (i.e., geometry shaders or programs) on the geometric primitives. Tessellation operations can generate one or more geometric primitives from each geometric primitive. In other words, the geometry shading stage 540 can subdivide each geometric primitive into a finer mesh of two or more geometric primitives for processing by the rest of the graphics processing pipeline 500. The geometry shading stage 540 passes the geometric primitives to the viewport SCC stage 550.

[0109] In one embodiment, the graphics processing pipeline 500 may operate within a streaming multiprocessor and vertex shading stage 520, primitive assembly stage 530, geometry shading stage 540, fragment shading stage 570, and / or hardware / software associated therewith, to sequentially perform processing operations. Once the sequential processing operations are complete, in one embodiment, the viewport SCC stage 550 may utilize the data. In one embodiment, primitive data processed by one or more stages in the graphics processing pipeline 500 may be written to a cache (e.g., an L1 cache, a vertex cache, etc.). In this case, in one embodiment, the viewport SCC stage 550 may access the data in the cache. In one embodiment, the viewport SCC stage 550 and the rasterization stage 560 are implemented as fixed-function circuits.

[0110] The viewport SCC stage 550 performs viewport scaling, culling, and clipping of geometric primitives. Each surface being rendered is associated with an abstract camera position. The camera position represents the position of the viewer viewing the scene and defines a viewing frustum that encloses the objects in the scene. The viewing frustum can include a viewing plane, a back plane, and four clipping planes. Any geometric primitives that are completely outside the viewing frustum can be culled (i.e., discarded) because they will not contribute to the final rendered scene. Any geometric primitives that are partially inside the viewing frustum and partially outside the viewing frustum can be clipped (i.e., converted to new geometric primitives that are enclosed within the viewing frustum). In addition, each geometric primitive can be scaled based on the depth of the viewing frustum. All potentially visible geometric primitives are then passed to the rasterization stage 560.

[0111] The rasterization stage 560 converts 3D geometric primitives into 2D fragments (e.g., capable of being used for display, etc.). The rasterization stage 560 can be configured to use the vertices of the geometric primitives to set a set of plane equations from which various attributes can be interpolated. The rasterization stage 560 can also calculate a coverage mask for multiple pixels, which indicates whether one or more sample positions of the pixel intercept the geometric primitive. In one embodiment, a z test can also be performed to determine whether the geometric primitive is occluded by other geometric primitives that have already been rasterized. The rasterization stage 560 generates fragment data (i.e., the interpolated vertex attributes associated with the specific sample position of each covered pixel), which is transmitted to the fragment shading stage 570.

[0112] The fragment shading stage 570 processes the fragment data by executing a set of operations (i.e., a fragment shader or program) on each of the fragments. The fragment shading stage 570 can generate pixel data (i.e., color values) for the fragment, such as by performing lighting operations or sampling a texture map using the fragment's interpolated texture coordinates. The fragment shading stage 570 generates pixel data, which is sent to the raster operations stage 580.

[0113] The raster operations stage 580 may perform various operations on the pixel data, such as performing alpha tests, stencil tests, and blending the pixel data with other pixel data corresponding to other fragments associated with the pixel. When the raster operations stage 580 has completed processing the pixel data (i.e., output data 502), the pixel data may be written to a render target, such as a frame buffer, a color buffer, etc.

[0114] It should be appreciated that one or more additional stages may be included in the graphics processing pipeline 500 in addition to or in place of one or more of the above-described stages. Various implementations of the abstract graphics processing pipeline may implement different stages. Furthermore, in some embodiments, one or more of the above-described stages may be excluded from the graphics processing pipeline (such as the geometry shading stage 540). Other types of graphics processing pipelines are contemplated within the scope of the present disclosure. Furthermore, any stage of the graphics processing pipeline 500 may be implemented by one or more dedicated hardware units within a graphics processor (such as the PPU 200). Other stages of the graphics processing pipeline 500 may be implemented by programmable hardware units (such as the SM 340 of the PPU 200).

[0115] The graphics processing pipeline 500 can be implemented via an application program executed by a host processor (such as a CPU). In one embodiment, a device driver can implement an application programming interface (API) that defines various functions that can be utilized by the application program to generate graphics data for display. A device driver is a software program that includes multiple instructions that control the operation of the PPU 200. The API provides an abstraction for programmers, allowing them to utilize specialized graphics hardware (such as the PPU 200) to generate graphics data without requiring them to utilize the specific instruction set of the PPU 200. An application program can include API calls that are routed to the device driver of the PPU 200. The device driver interprets the API calls and performs various operations in response to the API calls. In some cases, the device driver can perform operations by executing instructions on the CPU. In other cases, the device driver can perform operations at least in part by initiating operations on the PPU 200 using an input / output interface between the CPU and the PPU 200. In one embodiment, the device driver is configured to implement the graphics processing pipeline 500 using the hardware of the PPU 200.

[0116] Various programs can be executed within the PPU 200 to implement the various stages of the graphics processing pipeline 500. For example, a device driver can launch a kernel on the PPU 200 to execute the vertex shading stage 520 on one SM 340 (or multiple SMs 340). The device driver (or the initial kernel executed by the PPU 300) can also launch other kernels on the PPU 300 to execute other stages of the graphics processing pipeline 500, such as the geometry shading stage 540 and the fragment shading stage 570. In addition, some of the stages of the graphics processing pipeline 500 can be implemented on fixed unit hardware (such as a rasterizer or data assembler implemented within the PPU 300). It should be appreciated that the results from one kernel can be processed by one or more intermediate fixed-function hardware units before being processed by subsequent kernels on the SM 340.

[0117] Sample game streaming system

[0118] Now refer to Figure 6 , Figure 6 is an example system diagram of a game streaming system 600 according to some embodiments of the present disclosure. Figure 6 Includes one or more game servers 602 (which may include Figure 7 ), one or more client devices 604 (which may include components, features, and / or functionality similar to the example computing device 700 of Figure 7 ) and one or more networks 606 (which may be similar to the one or more networks herein). In some embodiments of the present disclosure, system 600 may be implemented.

[0119] In system 600, for a game session, one or more client devices 604 may receive only input data in response to input to one or more input devices, send the input data to one or more game servers 602, receive encoded display data from the one or more game servers 602, and display the display data on a display 624. In this way, more computationally intensive calculations and processing are offloaded to one or more game servers 602 (e.g., rendering (specifically, ray or path tracing) for the graphical output of the game session is performed by one or more GPUs of one or more game servers 602). In other words, the game session is streamed from one or more game servers 602 to one or more client devices 604, thereby reducing the demand on the one or more client devices 604 for graphics processing and rendering.

[0120] For example, with reference to instantiation of a game session, client device 604 can display a frame of the game session on display 624 based on display data received from one or more game servers 602. Client device 604 can receive input to one of one or more input devices and generate input data in response. Client device 604 can send the input data to one or more game servers 602 via one or more communication interfaces 620 and over one or more networks 606 (e.g., the Internet), and one or more game servers 602 can receive the input data via communication interface 618. One or more CPUs can receive the input data, process the input data, and send the data to one or more GPUs, causing the one or more GPUs to generate a rendering of the game session. For example, the input data can represent movement of a user's character in the game, weapon firing, reloading, passing, vehicle turning, etc. Rendering component 612 can render the game session (e.g., representing the results of the input data), and rendering capture component 614 can capture the rendering of the game session as display data (e.g., as image data capturing a rendered frame of the game session). The rendering of the game session may include lighting and / or shadow effects using ray or path tracing calculated using one or more parallel processing units (e.g., GPUs), which may further employ one or more dedicated hardware accelerators or processing cores to perform ray or path tracing techniques on one or more game servers 602. The encoder 616 may then encode the display data to generate encoded display data, and the encoded display data may be sent to the client device 604 via the communication interface 618 over one or more networks 606. The client device 604 may receive the encoded display data via the communication interface 620, and the decoder 622 may decode the encoded display data to generate display data. The client device 604 may then display the display data via a display 624.

[0121] Example computing device

[0122] Figure 7 is a block diagram of one or more example computing devices 700 suitable for implementing some embodiments of the present disclosure. The computing device 700 may include an interconnect system 702 that directly or indirectly couples the following devices: memory 704, one or more central processing units (CPUs) 706, one or more graphics processing units (GPUs) 708, a communication interface 710, input / output (I / O) ports 712, input / output components 714, a power supply 716, one or more presentation components 718 (e.g., one or more displays), and one or more logic units 720.

[0123] although Figure 7The various blocks of are shown as being connected with lines via interconnect system 702, but this is not intended to be limiting and is merely for clarity. For example, in some embodiments, presentation component 718 (such as a display device) can be considered to be I / O component 714 (e.g., if the display is a touch screen). As another example, CPU 706 and / or GPU 708 can include memory (e.g., memory 704 can also represent a storage device in addition to the memory of GPU 708, CPU 706, and / or other components). In other words, Figure 7 The term computing device is illustrative only. No distinction is made between categories such as "workstation," "server," "laptop," "desktop," "tablet," "client device," "mobile device," "handheld device," "gaming console," "electronic control unit (ECU)," "virtual reality system," and / or other device or system types, all of which are considered Figure 7 within the range of computing devices.

[0124] The interconnection system 702 may represent one or more links or buses, such as an address bus, a data bus, a control bus, or a combination thereof. The interconnection system 702 may include one or more bus or link types, such as an Industry Standard Architecture (ISA) bus, an Extended Industry Standard Architecture (EISA) bus, a Video Electronics Standards Association (VESA) bus, a Peripheral Component Interconnect (PCI) bus, a Peripheral Component Interconnect Express (PCIe) bus, and / or another type of bus or link. In some embodiments, there is a direct connection between components. As an example, the CPU 706 may be directly connected to the memory 704. Further, the CPU 706 may be directly connected to the GPU 708. In the case where there is a direct connection or a point-to-point connection between components, the interconnection system 702 may include a PCIe link to perform the connection. In these examples, the PCI bus does not need to be included in the computing device 700.

[0125] Memory 704 may include any of a variety of computer-readable media. Computer-readable media can be any available media that can be accessed by computing device 700. Computer-readable media can include volatile and non-volatile media and removable and non-removable media. By way of example and not limitation, computer-readable media can include computer storage media and communication media.

[0126] Computer storage media may include volatile and nonvolatile media and / or removable and non-removable media implemented in any method or technology for storage of information (e.g., computer-readable instructions, data structures, program modules, and / or other data types). For example, memory 704 may store computer-readable instructions (e.g., representing one or more programs and / or one or more program elements, such as an operating system). Computer storage media may include, but is not limited to, RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, digital versatile disks (DVDs) or other optical disk storage devices, cassettes, magnetic tape, magnetic disk storage devices or other magnetic storage devices, or any other medium that can be used to store the desired information and that can be accessed by the computing device 700. As used herein, computer storage media itself does not include signals.

[0127] Computer storage media may include computer-readable instructions, data structures, program modules, and / or other data types in a modulated data signal (such as a carrier wave or other transport mechanism), and include any information delivery media. The term "modulated data signal" may refer to a signal that has one or more characteristics set or changed in such a way as to encode information into the signal. By way of example, and not limitation, computer storage media may include wired media (such as a wired network or a direct-wired connection), and wireless media (such as acoustic, RF, infrared, and other wireless media). Combinations of any of the above should also be included within the scope of computer-readable media.

[0128] The one or more CPUs 706 may be configured to execute at least some computer-readable instructions to control one or more components of the computing device 700 to perform one or more of the methods and / or processes described herein. Each of the one or more CPUs 706 may include one or more cores (e.g., one, two, four, eight, twenty-eight, seventy-two, etc.) capable of processing numerous software threads simultaneously. The one or more CPUs 706 may include any type of processor and, depending on the type of computing device 700 implemented, may include different types of processors (e.g., a processor with fewer cores for mobile devices and a processor with more cores for servers). For example, depending on the type of computing device 700, the processor may be an Advanced RISC Machine (ARM) processor implemented using Reduced Instruction Set Computing (RISC) or an x86 processor implemented using Complex Instruction Set Computing (CISC). The computing device 700 may also include one or more CPUs 706 in addition to one or more microprocessors or auxiliary coprocessors (e.g., math coprocessors).

[0129] In addition to or in lieu of one or more CPUs 706, one or more GPUs 708 may be configured to execute at least some of the computer-readable instructions to control one or more components of the computing device 700 to perform one or more of the methods and / or processes described herein. One or more of the GPUs 708 may be an integrated GPU (e.g., integrated with one or more of the CPUs 706 and / or one or more of the GPUs 708 may be a discrete GPU). In embodiments, one or more of the GPUs 708 may be a coprocessor for one or more of the CPUs 706. The GPUs 708 may be used by the computing device 700 to render graphics (e.g., 3D graphics) or perform general-purpose computations. For example, the GPUs 708 may be used for general-purpose computing on a GPU (GPGPU). The one or more GPUs 708 may include hundreds or thousands of cores capable of processing hundreds or thousands of software threads simultaneously. The one or more GPUs 708 may generate pixel data for output images in response to rendering commands (e.g., rendering commands received from one or more CPUs 706 via a host interface). One or more GPUs 708 may include graphics memory, such as display memory, for storing pixel data or any other suitable data (such as GPGPU data). Display memory may be included as part of memory 704. One or more GPUs 708 may include two or more GPUs operating in parallel (e.g., via a link). The link may connect the GPUs directly (e.g., using NVLINK) or may connect the GPUs through a switch (e.g., using NVSwitch). When combined, each GPU 708 may generate pixel data or GPGPU data for different parts of the output or for different outputs (e.g., a first GPU for a first image and a second GPU for a second image). Each GPU may include its own memory, or may share memory with other GPUs.

[0130] In addition to or in lieu of the CPU 706 and / or GPU 708, the logic unit 720 may be configured to execute at least some of the computer-readable instructions to control one or more components of the computing device 700 to perform one or more of the methods and / or processes described herein. In embodiments, the CPU 706, GPU 708, and / or the logic unit 720 may independently or jointly perform any combination of methods, processes, and / or portions thereof. One or more of the logic units 720 may be a part of and / or integrated with one or more of the CPU 706 and / or GPU 708 and / or one or more of the logic units 720 may be a discrete component or otherwise external to the CPU 706 and / or GPU 708. In embodiments, one or more of the logic units 720 may be a coprocessor for one or more of the CPU 706 and / or GPU 708.

[0131] Examples of the logic unit 720 include one or more processing cores and / or components thereof, such as a tensor core (TC), a tensor processing unit (TPU), a pixel vision core (PVC), a vision processing unit (VPU), a graphics processing cluster (GPC), a texture processing cluster (TPC), a streaming multiprocessor (SM), a tree traversal unit (TTU), an artificial intelligence accelerator (AIA), a deep learning accelerator (DLA), an arithmetic logic unit (ALU), an application-specific integrated circuit (ASIC), a floating point unit (FPU), an input / output (I / O) element, a peripheral component interconnect (PCI) or a peripheral component interconnect express (PCIe) element, etc.

[0132] The communication interface 710 may include one or more receivers, transmitters, and / or transceivers that enable the computing device 700 to communicate with other computing devices via an electronic communication network (including wired and / or wireless communications). The communication interface 710 may include components and functionality to enable communication over any of a number of different networks, such as wireless networks (e.g., Wi-Fi, Z-Wave, Bluetooth, Bluetooth LE, ZigBee, etc.), wired networks (e.g., via Ethernet or wireless broadband communications), low-power wide-area networks (e.g., LoRaWAN, SigFox, etc.), and / or the Internet.

[0133] I / O ports 712 can enable computing device 700 to be logically coupled to other devices including I / O components 714, one or more presentation components 718, and / or other components, some of which may be built into (e.g., integrated into) computing device 700. Exemplary I / O components 714 include microphones, mice, keyboards, joysticks, game pads, game controllers, satellite dishes, scanners, printers, wireless devices, and the like. I / O components 714 can provide a natural user interface (NUI) that processes in-air gestures, voice, or other physiological input generated by a user. In some cases, the input can be sent to an appropriate network element for further processing. The NUI can implement any combination of the following: voice recognition, stylus recognition, facial recognition, biometric recognition, gesture recognition on and adjacent to the screen, in-air gestures, head and eye tracking, and touch recognition associated with the display of computing device 700 (described in more detail below). Computing device 700 can include a depth camera (e.g., stereo camera) system, an infrared camera system, an RGB camera system, touch screen technology, and combinations thereof for gesture detection and recognition. Additionally, computing device 700 may include an accelerometer or gyroscope that enables detection of motion (e.g., as part of an inertial measurement unit (IMU)). In some examples, computing device 700 may use the output of the accelerometer or gyroscope to render immersive augmented or virtual reality.

[0134] The power supply 716 may include a hard-wired power supply, a battery power supply, or a combination thereof. The power supply 716 may provide power to the computing device 700 to enable the components of the computing device 700 to operate.

[0135] The one or more presentation components 718 may include a display (e.g., a monitor, a touch screen, a television screen, a head-up display (HUD), other display types, or a combination thereof), speakers, and / or other presentation components. The one or more presentation components 718 may receive data from other components (e.g., one or more GPUs 708, one or more CPUs 706, etc.) and output data (e.g., as images, video, sound, etc.).

[0136] Sample network environment

[0137] A network environment suitable for implementing embodiments of the present disclosure may include one or more client devices, servers, network attached storage (NAS), other backend devices, and / or other device types. The client devices, servers, and / or other device types (e.g., each device) may be configured to: Figure 7 The computing devices 700 may be implemented on one or more instances of the computing devices 700 of the system—eg, each device may include similar components, features, and / or functionality of the computing devices 700 of the system.

[0138] The components of the network environment can communicate with each other via one or more networks, which can be wired, wireless, or both. The network can include multiple networks or one of multiple networks. For example, the network can include one or more wide area networks (WANs), one or more local area networks (LANs), one or more public networks (such as the Internet and / or the Public Switched Telephone Network (PSTN)), and / or one or more private networks. In the case where the network includes a wireless telecommunications network, components such as base stations, communication towers, or even access points (and other components) can provide wireless connections.

[0139] Compatible network environments may include one or more peer-to-peer network environments (in which case the server may not be included in the network environment) and one or more client-server network environments (in which case one or more servers may be included in the network environment). In a peer-to-peer network environment, the functionality described herein with respect to the server may be implemented on any number of client devices.

[0140] In at least one embodiment, the network environment may include one or more cloud-based network environments, distributed computing environments, combinations thereof, and the like. The cloud-based network environment may include a framework layer, a job scheduler, a resource manager, and a distributed file system implemented on one or more servers, which may include one or more core network servers and / or edge servers. The framework layer may include software supporting the software layer and / or a framework for one or more applications at the application layer. The software or applications may include network-based service software or applications, respectively. In an embodiment, one or more client devices may use web-based service software or applications (e.g., by accessing the service software and / or applications via one or more application programming interfaces (APIs)). The framework layer may be, but is not limited to, a type of free and open source software web application framework, such as one that may use a distributed file system for large-scale data processing (e.g., "big data").

[0141] A cloud-based network environment can provide cloud computing and / or cloud storage that performs any combination of the computing and / or data storage functions described herein (or one or more portions thereof). Any of these different functions can be distributed across multiple locations from a central or core server (e.g., one or more data centers that can be distributed across a state, region, country, globe, etc.). If the connection to the user (e.g., client device) is relatively close to an edge server, the core server can assign at least a portion of the functionality to the edge server. A cloud-based network environment can be private (e.g., limited to a single organization), public (e.g., available to many organizations), and / or a combination thereof (e.g., a hybrid cloud environment).

[0142] Client devices may include Figure 7 At least some of the components, features, and functionality of the described example computing device 700. By way of example and not limitation, the client device may be implemented as a personal computer (PC), a laptop, a mobile device, a smartphone, a tablet computer, a smartwatch, a wearable computer, a personal digital assistant (PDA), an MP3 player, a virtual reality head-mounted display, a global positioning system (GPS) or device, a video player, a camera, a surveillance device or system, a vehicle, a boat, a spacecraft, a virtual machine, a drone, a robot, a handheld communication device, a hospital device, a gaming device or system, an entertainment system, a vehicle computer system, an embedded system controller, a remote control, a home appliance, a consumer electronic device, a workstation, an edge device, any combination of the depicted devices, or any other suitable device.

[0143] Example implementation using lossy compression

[0144] Figure 8 A flow chart of a method 800 for streaming a compressed light field compressed using lossy compression is shown, according to one embodiment. Although method 800 is described in the context of a processing unit, method 800 may also be performed by a program, custom circuitry, or a combination of custom circuitry and program. For example, method 800 may be performed by a GPU (graphics processing unit), a CPU (central processing unit), or any processor capable of performing parallel path spatial filtering via hashing. Furthermore, one of ordinary skill in the art will appreciate that any system that performs method 800 is within the scope and spirit of embodiments of the present invention.

[0145] Should be understood that this arrangement described herein and other arrangements are only set forth as examples.Except those arrangements and elements shown or instead of those arrangements and elements shown, other arrangements and elements (such as, machines, interfaces, functions, sequences, functional groups etc.) can also be used, and some elements can be omitted together. Further, many elements described herein are functional entities that can be implemented as discrete or distributed components or in combination with other components and in any suitable combination and position. The various functions described herein as being performed by an entity can be performed by hardware, firmware and / or software. For example, the various functions can be performed by a processor that performs the instructions stored in a memory.

[0146] As shown in operation 802, a scene to be rendered is identified. In one embodiment, the scene may include a still image, a frame within a video, a frame within a video game, etc. In another embodiment, the remote device may include a computing node within a distributed computing system, one or more nodes within a cloud-based computing environment, a server computing device, etc. In yet another embodiment, the remote device may be physically separate from the client device.

[0147] Furthermore, as shown in operation 804, the light field of the scene is calculated using ray tracing. In one embodiment, the remote device may perform ray tracing within the scene to determine the light field of the scene. For example, ray tracing may include tracing the path of light as pixels within an image plane and simulating the effects of its encounters with objects within the scene. In another embodiment, ray tracing may be performed in real time.

[0148] Furthermore, in one embodiment, a light field may include multiple light probes. In another embodiment, each light probe may store information about light that passes through empty space within the scene.

[0149] Furthermore, as shown in operation 806, the light field is compressed using lossy compression to create compressed light field data of the scene. In one embodiment, the remote device may compress the light field using a video encoding and / or video compression unit. For example, the unit may be located within a GPU.

[0150] Likewise, in one embodiment, the light field may be compressed using one or more video compression algorithms / techniques. In another embodiment, the compression may take into account the current user view within the scene.

[0151] Furthermore, in one embodiment, compressing a light field using lossy compression may include performing a color conversion of the light field. For example, the light field may include color data (e.g., red, green, and blue (RGB) pixels, etc.). In another example, the light field may be converted from RGB pixels to YUV pixels, where Y represents a luma value and U and V represent chroma values. For example, a block of pixels of a predetermined size (e.g., four RGB pixels, etc.) may be re-encoded into YUV pixels of the same size (e.g., four pixels). In another embodiment, the re-encoded YUV or RGB pixels may then be compressed.

[0152] Furthermore, in one embodiment, compressing a light field using lossless or lossy compression may include performing downsampling of chroma planes within the light field. For example, the light field may include color values ​​that are normalized and stored in a predetermined texture (e.g., an A2RGB10 color texture, etc.). In another example, these normalized values ​​may be quantized and shifted into unsigned integer YUV values ​​(e.g., YUV tuples, etc.). In yet another example, these YUV tuples may be reordered into YUV planes. In yet another example, downscaling (e.g., downsampling) may be performed on the YUV plane to create a reduced YUV plane. Furthermore, encoding may then be applied to the reduced YUV plane.

[0153] In this way, compressing light fields using lossless or lossy compression can enable higher levels of compression, which can reduce the size of the compressed data and the amount of bandwidth required to send the compressed data to the client device. This can improve the performance of the computing hardware involved in sending and receiving the compressed data. While some chrominance information may be lost during lossy compression, this loss is acceptable because human visual perception is less sensitive to color than to luminance / brightness. As a result, the human eye may not perceive the loss of color.

[0154] Furthermore, the compressed light field data is transmitted to the client device, as shown in operation 808. In one embodiment, the compressed light field may be transmitted from the remote device to the client device via one or more wired and / or wireless communication networks (e.g., a wireless Internet connection, a cellular communication network, etc.).

[0155] In this way, a compressed light field can be remotely computed, compressed using lossless or lossy compression, and provided to a client device, where the client device can decompress the compressed light field and use the decompressed light field to perform global illumination on the scene at the client device. Implementing temporal reuse during the compression of the light field can result in reduced latency in both the creation and transmission of the compressed light field data. Furthermore, when computing the light field for a scene, the lighting calculations can be computed dynamically and may not rely on a static scene or static lighting. This can effectively account for dynamic changes in the scene / lighting and improve the quality of the rendered results.

[0156] Figure 9 A flow chart of method 900 for receiving and decompressing a light field compressed using color conversion is shown, according to one embodiment. Although method 900 is described in the context of a processing unit, method 900 may also be performed by a program, custom circuitry, or a combination of custom circuitry and program. For example, method 900 may be performed by a GPU (graphics processing unit), a CPU (central processing unit), or any processor capable of performing parallel path spatial filtering via hashing. Furthermore, one of ordinary skill in the art will appreciate that any system that performs method 900 is within the scope and spirit of embodiments of the present invention.

[0157] Should be understood that this arrangement described herein and other arrangements are only set forth as examples.Except those arrangements and elements shown or instead of those arrangements and elements shown, other arrangements and elements (such as, machines, interfaces, functions, sequences, functional groups etc.) can also be used, and some elements can be omitted together. Further, many elements described herein are functional entities that can be implemented as discrete or distributed components or in combination with other components and in any suitable combination and position. The various functions described herein as being performed by an entity can be performed by hardware, firmware and / or software. For example, the various functions can be performed by a processor that performs the instructions stored in a memory.

[0158] As shown in operation 902, compressed light field data of a scene is received. In one embodiment, the compressed light field may be received at a client device from a remote device via one or more wired and / or wireless communication networks (e.g., a wireless Internet connection, a cellular communication network, etc.). In another embodiment, the client device may include a device for at least partially rendering the scene and displaying the rendered scene. In yet another embodiment, the client device may include a mobile device (e.g., a smartphone, a tablet device, etc.), a portable gaming platform, a VR head-mounted display device, etc.

[0159] Furthermore, as shown in operation 904, the compressed light field data is decompressed, and a color conversion is performed on the decompressed light field data to obtain a light field of the scene. In one embodiment, the client device may use a video decoder and / or a video decompression unit to decompress the compressed light field. In another embodiment, the video decoder and / or the video decompression unit may comprise a dedicated hardware unit within the client device. In yet another embodiment, the video decoder and / or the video decompression unit may be located within a GPU of the client device.

[0160] Further, in one embodiment, the light field may have been previously converted from RGB pixels to YUV pixels. In another embodiment, the decompressed light field may therefore be in YUV format. In yet another embodiment, the decompressed light field may be converted from YUV pixels back to RGB pixels.

[0161] Furthermore, as shown in operation 906, global illumination can be performed for the scene using the scene's light field. In one embodiment, when calculating global illumination, the client device can perform a lookup within the light field (e.g., instead of performing ray tracing or other resource-intensive actions). In another embodiment, performing global illumination can include modeling indirect light (e.g., how light reflects from surfaces onto other surfaces within the scene) and direct light (e.g., light that strikes a surface directly from a light source). In yet another embodiment, the calculated global illumination can be used to render the scene at the client device. In yet another embodiment, the rendered scene can be displayed by the client device.

[0162] In this way, the performance of global illumination of a scene can improve the realism of the scene as displayed by a client device. Additionally, global illumination can be performed more efficiently using light field lookups, which can reduce the amount of processing required at the client device. This can improve the performance of the client device when rendering the scene and can result in high-quality, dynamically rendered images with low bandwidth usage.

[0163] Exemplary Distributed Computing Environment

[0164] Figure 10 An exemplary distributed rendering system 1000 is shown according to an exemplary embodiment. As shown, a client device 1002 communicates with a remote device 1004 via a communication network 1006. In one embodiment, the communication network 1006 may include one or more wired and / or wireless communication networks.

[0165] Furthermore, in one embodiment, a scene to be rendered may be received by the remote device 1004. For example, the scene may include a scene within a video game played by a user at the client device. In another example, game information (e.g., the user's position within the game, the user's view within the game, etc.) may be transmitted from the client device 1002 to the remote device 1004 via the communication network 1006.

[0166] Furthermore, in one embodiment, remote device 1004 may utilize ray tracing to calculate the light field of the scene. Additionally, remote device 1004 may compress the light field of the scene using lossless or lossy compression to create compressed light field data for the scene. For example, remote device 1004 may perform color conversion of the light field. In another example, remote device 1004 may perform downsampling of chroma planes within the light field.

[0167] In one embodiment, the light field may be compressed by the remote device 1004 using one or more video compression algorithms / techniques that implement time multiplexing. In another example, the calculation and compression of the light field may take into account the currently visible user view within the scene provided from the client device 1002 to the remote device 1004.

[0168] Furthermore, in one embodiment, remote device 1004 may transmit compressed light field data to client device 1002 via communication network 1006. Client device 1002 may then decompress the compressed light field data and use the decompressed data to obtain a light field for the scene. Client device 1002 may then calculate global illumination values ​​for the scene using one or more lookups within the light field. Client device 1002 may also perform color conversion on the decompressed light field data in response to determining that color conversion was performed on the light field data by remote device 1004 prior to compression. The client device may then render the scene using the calculated global illumination values ​​for the scene.

[0169] In this way, rendering responsibilities can be divided between the client device 1002 and the remote device 1004, such that the remote device 1004 performs more computationally intensive rendering or ray tracing hardware-dependent responsibilities (e.g., ray tracing, light field calculations) in order to simplify / reduce rendering responsibilities performed at the client device (e.g., global illumination with light field lookups), or to simplify / reduce ray tracing hardware dependencies at the client device. Furthermore, by utilizing one or more video compression algorithms / techniques and considering the current user view within the scene during compression, the resulting compressed data can be minimized, which can reduce the amount of data sent over the communication network 1006 and can reduce the overall latency of the system 1000.

[0170] A distributed decoupled system for streaming dynamic light probes to thin clients

[0171] Today's high-performance graphics systems with multiple GPUs for gaming, hardware-accelerated ray tracing, and efficient algorithms can use ray-traced global illumination in real time to approach the rendering quality of film effects and provide interactive latency measured in milliseconds. These high-end systems set expectations that cannot be met on lower-end consumer platforms constrained by thermal limitations, battery power, and limited GPU feature sets.

[0172] Screen resolutions, refresh rates, and high dynamic range (HDR) color depth are increasing, and XR applications need to render multiple views. These advancing requirements may overwhelm current mobile processors used in unpowered devices.

[0173] In one embodiment, global illumination (GI) data can be streamed from a cloud rendering server to thin clients. The streamed data enables dynamic, high-quality, ray-traced diffuse GI at low computational cost on thin clients without ray tracing capabilities. Efficient encoding and decoding hardware features can be used to achieve high compression ratios with low latency. Further, heuristic light probe update schemes can be used to reduce the amount of data transmitted within a GI update. Thus, in one embodiment, a single server can update thousands (tens of thousands) of light probes per second across multiple connected thin clients, thereby amortizing the rendering cost.

[0174] An exemplary optical probe streaming pipeline may include the following 4 steps:

[0175] 1. Server-side ray tracing of light fields (light probe volumes)

[0176] 2. Encode light probe data optimized for each client

[0177] 3. Transmit the encoded data to the client via the network

[0178] 4. Decode the light probe data and render it on the client

[0179] Figure 11 An exemplary data flow 1100 according to one embodiment is shown. Instead of streaming entire video frames, a client-server architecture can distribute the graphics pipeline across one or more servers and one or more clients. In one embodiment, a server, assumed to be in a data center, conceptually consists of two nodes interconnected by a high-bandwidth link. The first node acts as a game server 1102 to reliably receive user input from clients 1106 and update the game state 1108 accordingly. This is similar to a standard multi-player game server. The second node acts as a GI server 1104, which performs ray tracing of light probe data and encodes it for client delivery.

[0180] Assume that any scene changes are reported to the GI server 1104 by the game server 1102 with negligible latency. Using the current scene state, the GI server 1104 computes view-independent diffuse light probe texture data. As a result, this GI data can be used by multiple clients in a multiplayer game scene, amortizing the rendering cost. Hardware-accelerated video encoding is used on the server to compress the light probe data. The encoded data is transmitted to the client 1106 using a reliable low-latency network protocol over a combination of WiFi, 5G, and / or wired connections.

[0181] On the client 1106 side, hardware video decoding is used (when available) to decompress the light probe data with low latency. The client rendering loop 1110 then uses the uncompressed light probe textures to add dynamic and high-quality diffuse GI at low computational cost. The GI texture decoding process is decoupled from the rendering loop and can occur at a completely different rate. This allows the client 1106 to run at full frame rate, resulting in minimal perceived delay to user input or movement. Thus, input lag, as occurs in traditional cloud rendering solutions, is avoided.

[0182] In one embodiment, when rendering GI data in the cloud, the GI data provides certain benefits: (1) diffuse GI is view-independent and is used across multiple users and potentially multiple frames; (2) rendering GI data is computationally expensive and thin client hardware is insufficient for rendering GI data; and (3) users are less sensitive to delayed diffuse lighting than view-dependent effects.

[0183] Uncompressed light probe streaming

[0184] Streaming probe color

[0185] In one embodiment, a dynamic diffuse global illumination (DDGI) irradiance description can be used. Each probe can encode the color data for each direction in the form of an octahedral map using a 10x10 texel array (each texel has 32 bits of color). Therefore, the required network throughput Tcolor for updating any given N probe color probes at rate Rcolor (in Hz) can be as follows:

[0186] Tcolor=Rcolor×Nprobes×3.125kb (1)

[0187] In one example, a 16x8x16 (2048) probe body with an uncompressed DDGI color texture size of 6.25Mb (0.78MB) requires 62.5Mbps of bandwidth to update 2048 uncompressed color probes at 10Hz.

[0188] Streaming Probe Visibility

[0189] In one embodiment, DDGI stores the average distance and squared distance for each probe. During shading, these values ​​are used to determine the visibility weight between the probe and the shading point using a Chebyshev statistical test. The average distance / squared distance data is called "visibility data" and the texture itself is called a "visibility texture." The visibility texture contains 18x18 texture pixels per probe encoded as a pair of half-precision (16-bit) floating point values ​​(32 bits / texture pixel). Thus, R visibilityThe required throughput of the visibility update rate (T visibility ) can be shown as follows:

[0190] Tvisibility=Rvisibility×Npriobes×10.125kb (2)

[0191] For the same probe body as above for color analysis (2048 probes), the size of the visibility texture is 3.24 times higher than the size of the color texture, or 20.3 Mb (2.5 MB). Streaming the original visibility texture at 10 Hz requires a throughput of 202.5 Mbps. The minimum required throughput for uncompressed color and visibility textures updated at 10 Hz for 2048 probes is 265 Mbps. These are exemplary bandwidth limitations used when considering multiple thin client devices in a public current wireless network. Although this naive, uncompressed approach is lossless and maintains perfect probe "consistency", it requires an unreasonably high network throughput because it sends all probe data, regardless of whether the data is unchanged or may be used on the client. However, this throughput can be reduced.

[0192] Low-latency optical probe compression

[0193] In one embodiment, a predetermined video decoding / compression standard is used to implement GPU-accelerated high dynamic range (HDR) video compression. In another embodiment, hardware-accelerated HDR10 encoding and decoding can be implemented.

[0194] Color coded

[0195] In one embodiment, a method for compressing color data is as follows. Normalized color values ​​are stored in an A2RGB10F texture (10-bit unsigned small floating point for each color, with 2 unused bits for alpha). Since the unsigned small floating point is normalized, these values ​​can be quantized to 10-bit unsigned integer values ​​in the range [0, 1023] without loss. An exemplary input format for a hardware encoder is a 16-bit unsigned integer organized in three Y, U, and V planes (linear Y - luminance, U, V chrominance). The YUV420 surface format can be used for lossy compression, while the YUV444 surface format can be used for lossless compression, which avoids chroma subsampling. In one embodiment, the hardware encoder uses only 10 of the 16 bits for color encoding. The remaining bits are reserved for future versions of the codec that support 12-bit and 16-bit color encoding. Therefore, these bits can be set to zero. The YUV tuples are then reordered into the Y, U, and V planes, and hardware encoding is applied thereon. For decoding, the process is reversed.

[0196] Coding Visibility

[0197] One solution for visibility relies on lossless and low bit depth video encoding. The idea is to distribute a single 16-bit floating point value over two adjacent 8-bit integer values. Since the visibility texture for light probes holds two channels (RG16F), the bits are distributed over four 8-bit integer values. If the hardware encoder used does not support four-channel images, a sequence of three RG16F values ​​can be packed into a sequence of four YUV values ​​that effectively distributes the bits into a YUVY, UVYU, VYUV sequence. The YUV texture can be widened to allocate enough memory for the visibility texture. For the original light probe visibility texture width x, the YUV texture width can be increased to

[0198] Threaded client-side decoding

[0199] To maximize throughput and avoid CPU blocking between rendering updates and decoding incoming texture updates, texture decoding can occur in a separate thread on the client. This frees up the game loop from blocking during CPU-based texture decoding or GPU decode dispatch.

[0200] Selective Light Probe Update

[0201] As a next step to optimize the network throughput consumption of each GI update, you can avoid sending probes that do not contain information relevant to client-side shading. Relevant probes meet the following three conditions:

[0202] 1. It is updated and used for shading in the scene

[0203] 2. Its attached texture pixels have changed significantly since the last update sent to the client

[0204] 3. It is a potentially visible coloring point for users

[0205] On the server, all probes that hold their previous three conditions are collected and sent to the client. At each send, the sent probe data is recorded on the server and compared with the rendered probe data to determine which probes should be sent in the next send.

[0206] Light Probe Repackaging

[0207] Instead of transmitting the entire texture (global probe volume update), inactive probes can be removed from the original texture by moving to an index-based probe update system (selective, iterative). Additional reductions are possible for each core texel array, in both color and visibility textures. The original DDGI texture includes a one-pixel guard band (per probe) for texture filtering, which contains redundant information from the core texel array. This data can be removed on the server during the packing step and reconstructed on the client without loss during unpacking. This strategy reduces the size of the transmitted texture, resulting in the benefit of being independent of the compression scheme (or uncompressed transmission). However, for time multiplexing, the probe positions in the texture can be maintained over multiple frames if possible.

[0208] The index-based update scheme allows the total packet size to be adjusted in a flexible manner, as the number of probes to update iteratively can be specified. This effectively introduces a 'knob' to adjust throughput on a per-client basis.

[0209] Light Probe Update Texture

[0210] The following describes selective probe updates on the server and client.

[0211] The server stores a permanent client probe texture and a temporary selective probe update texture for each client. The permanent texture represents the state of the probes as they exist on each client. Each update step creates a temporary probe update texture and stores the information of the probe being updated. Together with this texture, the client can be informed which probes in the volume are being updated. This information is provided by the probe index buffer, which maps the probe update texture coordinates into the client's probe texture. The size of the probe index buffer (encoded as a single two-byte unsigned integer per probe) is very small compared to the probe update texture, and is further reduced by delta encoding.

[0212] For the client, this process happens in reverse order. Only those Light Probes that have been sent to the client are updated in the persistent probe texture. The mapping from the update texture to the persistent light probe texture uses the received and decompressed probe index buffer. The remaining texture pixels are assumed to be valid from the previous state. Since the texture decompression and update steps occur asynchronously in parallel threads, the rendering loop is not blocked and can run continuously at full frame rate.

[0213] Light probe change threshold

[0214] To conservatively estimate all probes that may affect the primary client's view, all probes whose values ​​have changed since they were last sent can be considered. Two additional textures can be maintained on the server for each connected client, one for color and one for visibility, which store the state of each probe when it was last sent. At send time, each probe can be compared to its last sent state and only considered for transmission if its value has changed by more than a specified threshold. Although a higher threshold may be perceptually tolerable, a probe may be marked as changed if it is not the same texel-for-texel as its last sent state. Fully converged color on the server making the scene change can be assumed to be the only source of change in the probe results. Before each probe is sent, its data can be written to the corresponding last sent texture.

[0215] Server-side client view probe culling

[0216] Given a viewpoint within a scene, the subset of probe texels that may contribute to a rendered frame are only those texels close to points with primary visibility. This can represent a substantial reduction in the overall number of active / updated probes from the total volume.

[0217] By limiting the updated probes to those that contribute to shading the primary visibility frustum, the number of probes that need to be updated can be reduced. The number of updated probes can be adjusted based on the game's client-side view prediction scheme. Such predictions can be coupled to network latency and game content. An application-independent strategy can be implemented by estimating the Potentially Visible Set (PVS) of probes on the server, which contribute to client-side shading by rendering a spherical view from the client's position. Gathering probes from this PVS estimate ensures the correctness of the client's view under camera rotation.

[0218] Although various embodiments have been described above, it should be understood that these embodiments are presented by way of example only and not limitation. Thus, the breadth and scope of the preferred embodiment should not be limited by any of the above exemplary embodiments, but should be defined only in accordance with the appended claims and their equivalents.

[0219] The present disclosure may be described in the general context of computer code or machine-usable instructions, including computer-executable instructions (e.g., program modules) executed by a computer or other machine (e.g., a personal data assistant or other handheld device). In general, program modules (including routines, programs, objects, components, data structures, etc.) refer to code that performs a specific task or implements a specific abstract data type. The present disclosure may be practiced in a variety of system configurations, including handheld devices, consumer electronics, general-purpose computers, more specialized computing devices, and the like. The present disclosure may also be practiced in distributed computing environments where tasks are performed by remote processing devices linked via a communications network.

[0220] As used herein, the phrase "and / or" for two or more elements should be interpreted as meaning only one element or combination of elements. For example, "element A, element B, and / or element C" may include only element A, only element B, only element C, element A and element B, element A and element C, element B and element C, or elements A, B, and C. In addition, "at least one of element A or element B" may include at least one of element A, at least one of element B, or at least one of element A and at least one of element B.

[0221] The subject matter of the present disclosure is described herein in detail to satisfy statutory requirements. However, the description itself is not intended to limit the scope of the present disclosure. On the contrary, the inventors have anticipated that the claimed subject matter may also be embodied in other ways in conjunction with other current or future technologies, to include steps or combinations of steps different from those described in this document. Furthermore, although the terms "step" and / or "box" may be used herein to refer to different elements of the method employed, these terms should not be interpreted as implying any particular order among or between the various steps disclosed herein unless the order of the various steps is explicitly described.

Claims

1. A method comprising: At the device, Identify the scene to render; computing a light field of the scene using ray tracing, wherein the light field comprises color values ​​that are normalized and stored in a predetermined texture, and wherein the normalized color values ​​are quantized and shifted into unsigned integer YUV tuples; compressing the light field using lossless or lossy compression to create compressed light field data of the scene; as well as The compressed light field data is sent to a client device.

2. The method of claim 1 , wherein compressing the light field using lossless or lossy compression comprises: A color conversion is performed on the light field. 3 . The method of claim 1 , wherein the color value comprises red, green, and blue RGB pixels converted into the YUV tuple, where Y represents a luminance value and U, V represent chrominance values.

4. The method of claim 1 , wherein compressing the light field using lossless or lossy compression comprises: Downsampling is performed on chroma planes within the light field.

5. The method according to claim 1, further comprising: The YUV tuples are reordered into YUV planes, and downscaling is performed on the YUV planes to create downscaled YUV planes.

6. The method according to claim 5, further comprising: Encoding is applied to the downscaled YUV planes.

7. The method according to claim 1, wherein: The light field comprises an array of blocks, Each block includes color texture information and visibility texture information. The color texture information includes lighting color information within the block, and The visibility texture information includes a distance to a closest surface within the block.

8. The method of claim 1, wherein the light field is compressed using one or more video compression techniques that implement time multiplexing during compression.

9. The method of claim 8, wherein only the differences between the light field of the scene and the light field of a previous scene are compressed.

10. The method of claim 1, wherein the compressing takes into account a current user view within the scene, such that only a portion of the computed light field that is within the current user view is compressed.

11. The method of claim 1 , wherein at the client device: receiving compressed light field data of a scene; decompressing the compressed light field data and performing a color conversion on the decompressed light field data to obtain the light field of the scene; and Global illumination of the scene is calculated using the light field of the scene.

12. The method of claim 11, wherein the decompressed light field data is converted from YUV pixels to RGB pixels.

13. A non-transitory computer-readable storage medium storing instructions that, when executed by a processor of a device, cause the processor to cause the device to: Identify the scene to render; computing a light field of the scene using ray tracing, wherein the light field comprises color values ​​that are normalized and stored in a predetermined texture, and wherein the normalized color values ​​are quantized and shifted into unsigned integer YUV tuples; compressing the light field using lossless or lossy compression to create compressed light field data of the scene; as well as The compressed light field data is sent to a client device.

14. The computer-readable storage medium of claim 13, wherein compressing the light field using lossless or lossy compression comprises: A color conversion is performed on the light field.

15. The computer-readable storage medium of claim 13, wherein the color value comprises red, green, and blue RGB pixels converted to the YUV tuple, where Y represents a luma value and U, V represent chroma values.

16. The computer-readable storage medium of claim 13, wherein compressing the light field using lossless or lossy compression comprises: Downsampling is performed on chroma planes within the light field.

Images