USE OF RESIDUAL VIDEO DATA RESULTING FROM COMPRESSION OF ORIGINAL VIDEO DATA TO IMPROVE DECOMPRESSION OF THE ORIGINAL VIDEO DATA

Abstract

Procedure, comprehensive: Identifying residual video data resulting from compression of original video data; Encoding the remaining video data to generate encoded remaining video data; Binarize the encoded residual video data to generate binarized residual video data; Compressing the binarized residual video data to produce compressed residual video data; and Transferring the compressed remaining video data; wherein the residual video data is determined by calculating a difference between decompressed original video data and the original video data, wherein the residual video data is encoded using a binary residual autoencoder.

Description

AREA OF INVENTION

[0001] The present invention relates to video streaming and in particular to the use of residual video data resulting from compression of original video data to improve decompression of the original video data. BACKGROUND

[0002] Video streaming is a popular method for delivering content to users. Due to large video sizes and limited network bandwidth, video compression may be necessary to stream video data. However, current methods for implementing video compression result in undesirable artifacts and effects within the decompressed video data. Therefore, it is desirable to improve the quality of streamed video data. Consequently, there is a need to address these and / or other issues related to the state of the art. KIN, C. et al. describe this in: Video compression using recurrent convolutional neural networks. July 2, 2017. URL: https: / / cs231n.stanford.edu / reports / 2017 / pdfs / 423.pdf describes a method for video compression using convolutional neural networks. US 2010 / 0 054 615 A1 describes a method for encoding and decoding an image using a binarization table. An object of the invention is to improve the decompression of compressed video data. SUMMARY

[0003] This problem is solved by the features of the independent claims. Further preferred embodiments are described in the dependent claims. BRIEF DESCRIPTION OF THE DRAWINGS Fig. Figure 1 illustrates a flowchart of a method for generating and transmitting binary remainder representations according to one embodiment. Fig. Figure 2 illustrates a parallel processing unit according to one embodiment. Fig. Figure 3A illustrates a general-purpose processing cluster within the parallel processing unit of Fig. 2 according to one embodiment. Fig. Figure 3B illustrates a storage partition unit of the parallel processing unit of Fig. 2 according to one embodiment. Fig. Figure 4A illustrates the streaming multiprocessor of Fig. 3A according to one embodiment. Fig. 4B is a conceptual diagram of a [project] using the PPU from Fig. 2 implemented processing systems according to one embodiment. Fig. 4C illustrates an example system in which the different architecture and / or functionality of the various previous embodiments can be implemented. Fig. Figure 5 is a conceptual diagram of a PPU by Fig. 2 implemented graphics processing pipeline according to one embodiment. Fig. Figure 6 is an example overview of a video streaming pipeline according to one embodiment. Fig. Figure 7 is an example binary rest autoencoder architecture according to one embodiment. DETAILED DESCRIPTION

[0004] Video streaming is a common method for sending video content (e.g., movies, TV shows, recorded events, etc.) from one location to another. For example, video can be streamed from a server to many home computers, media streaming devices, mobile phones, and so on. Videos are often compressed (reduced in size) to stream more efficiently. However, the methods used to compress a video can negatively affect the video quality when the video is later decompressed (restored to its original size). For example, the compression / decompression process can introduce artifacts (glitches / distortion) into the decompressed video.

[0005] To address this issue, a video is first compressed and then decompressed. The results of this process are compared to the original video to determine the difference between the two, which is referred to as residual video data. This residual video data is then converted into a smaller format through processes known as encoding, binarizing, and compressing, and the converted residual data is sent to a destination along with the compressed video. At the destination, the residual video data is converted back into its original format and used to improve the quality of the decompressed video.

[0006] Fig. Figure 1 illustrates a flowchart of a method 100 for generating and transmitting binary remainder representations according to one embodiment. Although the method 100 is described in connection with a processing unit, it can also be performed by a program, a user-defined circuit, or a combination of both. For example, the method 100 can be performed by a GPU (graphics processing unit), a CPU (central processing unit), or any processor capable of performing parallel path space filtering by hashing. Furthermore, it is understood that any system performing the method 100 falls within the scope and spirit of the embodiments of the present invention.

[0007] As shown in Operation 102, residual video data resulting from compression of the original video data is identified. In one embodiment, the original video data can include any type of video data (e.g., recorded video data, generated video data, game video data, etc.). For example, the original video data can be generated by one or more cameras. In another example, the one or more cameras can be located on or inside a vehicle.

[0008] Additionally, in one embodiment, the original video data can be compressed using one or more compression algorithms. In another embodiment, the original video data can be compressed using one or more encoders (e.g., an H.264 encoder, etc.). In yet another embodiment, the results of the compression of the original video data can include compressed original video data.

[0009] Furthermore, in one embodiment, the compressed original video data can be compared with the original video data. For example, the compressed video data can first be decompressed / decoded (e.g., using one or more decoders such as an H.264 decoder, etc.) to generate decompressed / decoded original video data. According to the invention, the remaining video data is determined by calculating the difference between the decompressed / decoded original video data and the original video data.

[0010] Furthermore, as shown in Operation 104, the residual video data is encoded to generate encoded residual video data. In one embodiment, the encoding of the residual video data may include compressing the residual video data. According to the invention, the residual video data is encoded using a binary residual autoencoder. For example, the binary residual autoencoder may be separate from an encoder module that has compressed the original video data.

[0011] In one embodiment, the binary remainder autoencoder can also include a remainder encoder. For example, the remainder data can be encoded using the remainder encoder. In another embodiment, the binary remainder autoencoder can include a neural network.

[0012] Additionally, as shown in Operation 106, the encoded residual video data is binarized to produce binarized residual video data. In one embodiment, the encoded residual video data can be binarized using the Binary Residual Autoencoder. In another embodiment, the Binary Residual Autoencoder can include a binarizer. For example, the binarizer can convert frame-by-frame data from the encoded residual video data into a compressible binary bitstream, so that the binarized residual video data includes a binary bitstream. In yet another example, the binarizer can be trained and implemented using Tanh activation. In a still further example, the binarizer can be trained and implemented using HardTanh activation.In yet another example, the binarizer can be trained and implemented using one or more of the following methods: sigmoid activation, stochastic regularization, and Gumbel noise.

[0013] Furthermore, in one embodiment, the binarized residual video data is compressed to generate compressed residual video data. In one embodiment, the binarized residual video data can be compressed using the binary residual autoencoder. In another embodiment, the binary residual autoencoder can include a Huffman encoder. For example, the Huffman encoder can compress the binary bitstream generated by the binarizer to reduce the amount of data to be transmitted.

[0014] In another embodiment, the encoding, binarizing, and compression can be domain-specific. For example, the binary rest autoencoder can perform the encoding, binarizing, and compression of the residual video data, and such a binary rest autoencoder can be trained in a domain-specific manner. In particular, a limited set of possible outputs (e.g., frames, scenes, situations, etc.) can be used during the training of the binary rest autoencoder. In this way, video compression performance can be improved using a binary rest autoencoder restricted to a specific domain. In yet another embodiment, the encoding, binarizing, and compression can be performed using one or more parallel processing units (PPUs), as described in Fig. 2 shown below.

[0015] Furthermore, as shown in Operation 108, the compressed residual video data is transmitted. In one embodiment, the compressed residual video data can be transmitted together with the compressed original video data. In another embodiment, the compressed residual video data and the compressed original video data can be sent simultaneously over a single channel. In yet another embodiment, the compressed residual video data and the compressed original video data can be generated in a server device and sent to a client device for reception, decompression, and the creation of an output video. In yet another embodiment, the compressed residual video data and the compressed original video data can be streamed to a client.

[0016] Additionally, in one embodiment, the compressed original video data and the compressed residual video data can be received in a client device. In another embodiment, the compressed original video data can be decompressed in the client device to generate decompressed original video data. For example, the compressed original video data can be decompressed by a decoder module. In yet another example, the compressed original video data can be decompressed using one or more decompression algorithms. In still another example, the compressed original video data can be decompressed using one or more decoders (e.g., an H.264 decoder, etc.).

[0017] Furthermore, in one embodiment, the compressed residual video data can be reconstructed in the client device to generate reconstructed residual video data. For example, reconstructing the compressed residual video data can involve decompressing the compressed residual video data. In another example, the compressed residual video data can be decompressed using a residual module separate from the decoder module. In yet another example, the decoder module can include a Huffman decoder. In yet another example, the decoder module can include a residual decoder.

[0018] Furthermore, in one embodiment, the reconstructed residual video data can be added back to the decompressed original video data to generate output video data. In another embodiment, the output video data can be displayed. For example, the output video data can be displayed on a screen, shown in a vehicle, analyzed, viewed on a gaming device, etc.

[0019] In this way, residual video data can be encoded and transmitted with compressed original video data to improve the results of decompressing the compressed original video data. Separate encoders can be used for the original video data and the residual video data. This residual video data can be compressed in a region-specific manner, which can reduce the size of the compressed residual video data and decrease the amount of bandwidth used to transmit the compressed residual video data to a client device. Furthermore, a binary residual autoencoder can be used to encode the residual video data, where the binary residual autoencoder includes a residual encoder and a binarizer.

[0020] Further information is now provided on various optional architectures and features that can be used to implement the aforementioned framework according to the user's wishes. It is crucial to note that the following information is provided for illustrative purposes only and should not be interpreted as restrictive in any way. Any of the following features can be included optionally, with or without excluding other described features. Parallel processing architecture

[0021] Fig. Figure 2 illustrates a parallel processing unit (PPU) 200 according to one embodiment. In one embodiment, the PPU 200 is a multi-threaded processor implemented on one or more integrated circuits. The PPU 200 is a latency-hiding architecture designed to process many threads in parallel. A thread (i.e., a thread of execution) is an instantiation of a set of instructions configured to be executed by the PPU 200. In one embodiment, the PPU 200 is a graphics processing unit (GPU) configured to implement a graphics rendering pipeline for processing three-dimensional (3D) graphics data to produce two-dimensional (2D) image data for display on a display device such as a liquid crystal display (LCD) device.In other embodiments, the PPU 200 can be used to perform general-purpose calculations. Although an example parallel processor is provided herein for illustrative purposes, it should be expressly noted that this processor is specified for illustrative purposes only and that any processor can be used to supplement and / or replace it.

[0022] One or more PPU 200s can be configured to accelerate thousands of high-performance computing (HPC) systems, a data center, and machine learning applications. The PPU 200 can be configured to accelerate numerous deep learning systems and applications, including autonomous vehicle platforms, deep learning, high-precision speech, image, and text recognition systems, intelligent video analytics, molecular simulations, drug discovery, disease diagnosis, weather forecasting, big data analytics, astronomy, molecular dynamics simulation, financial modeling, robotics, factory automation, real-time speech translation, online search optimization, personalized user recommendations, and similar applications.

[0023] As in Fig. As shown in Figure 2, the PPU 200 includes an input / output (I / O) unit 205, a preprocessing unit 215, a scheduler unit 220, a workload distribution unit 225, a hub 230, a crossbar (Xbar) 270, one or more general-purpose processing clusters (GPCs) 250, and one or more partition units 280. The PPU 200 can be connected to a host processor or other PPUs 200 via one or more high-speed NVLink 210 connections. The PPU 200 can be connected to a host processor or other peripheral devices via a connection 202. The PPU 200 can also be connected to local memory comprising a number of memory devices 204. In one embodiment, the local memory can comprise a number of DRAM (Dynamic Random Access Memory) devices.The DRAM devices can be configured as an HBM (High-Bandwidth Memory) subsystem, with multiple DRAM dies (wafer pieces) stacked in each device.

[0024] The NVLink 210 connection allows systems to scale and integrate one or more PPUs 200 in combination with one or more CPUs, supports cache coherence between the PPUs 200 and CPUs, and enables CPU mastering. Data and / or instructions can be transferred from the NVLink 210, via the Hub 230, to / from other PPU 200 units, such as one or more copy engines, a video encoder, a video decoder, a power management unit, etc. (not explicitly shown). The NVLink 210 is used in conjunction with Fig. 4B is described in more detail.

[0025] The I / O unit 205 is configured to send and receive communication (i.e., commands, data, etc.) from a host processor (not shown) via connection 202. The I / O unit 205 can communicate directly with the host processor via connection 202 or via one or more intermediate devices, such as a memory bridge. In one embodiment, the I / O unit 205 can communicate with one or more other processors, such as one or more of the PPUs 200, via connection 202. In one embodiment, the I / O unit 205 implements a PCIe (Peripheral Component Interconnect Express) interface for communication over a PCIe bus, and connection 202 is a PCIe bus. In alternative embodiments, the I / O unit 205 can implement other types of known interfaces for communication with external devices.

[0026] The I / O unit 205 decodes packets received over connection 202. In one embodiment, the packets represent commands configured to cause the PPU 200 to perform various operations. The I / O unit 205 transmits the decoded commands to various other units of the PPU 200, as specified by the commands. For example, some commands may be transmitted to the preprocessing unit 215. Other commands may be transmitted to the hub 230 or other units of the PPU 200, such as one or more copy engines, a video encoder, a video decoder, a power management unit, etc. (not explicitly shown). In other words, the I / O unit 205 is configured to forward communication between and among the various logical units of the PPU 200.

[0027] In one embodiment, a program executed by the host processor encodes an instruction stream in a buffer, which provides the PPU 200 with workload for processing. A workload can include multiple instructions as well as data to be processed by those instructions. The buffer is a region in memory that both the host processor and the PPU 200 can access (i.e., read / write). For example, the I / O unit 205 can be configured to access the buffer in system memory connected to link 202 via memory requests transmitted over link 202. In one embodiment, the host processor writes the instruction stream to the buffer and then sends a pointer to the beginning of the instruction stream to the PPU 200. The preprocessing unit 215 receives pointers to one or more instruction streams.The preprocessing unit 215 manages one or more streams, reads instructions from the streams and forwards instructions to the various units of the PPU 200.

[0028] The preprocessing unit 215 is coupled to a scheduler unit 220, which configures the various GPCs 250 to process tasks defined by the one or more streams. The scheduler unit 220 is configured to track status information for the various tasks it manages. This status can indicate which GPC 250 a task is assigned to, whether the task is active or inactive, the priority level assigned to the task, and so on. The scheduler unit 220 manages the execution of a multitude of tasks on the one or more GPCs 250.

[0029] The scheduler unit 220 is coupled to a work distribution unit 225, which is configured to dispatch tasks for execution on the GPCs 250. The work distribution unit 225 can track a number of scheduled tasks received from the scheduler unit 220. In one embodiment, the work distribution unit 225 manages a pool of pending tasks and a pool of active tasks for each of the GPCs 250. The pool of pending tasks can contain a number of slots (e.g., 32 slots) for tasks to be processed by a particular GPC 250. The pool of active tasks can contain a number of slots (e.g., 4 slots) for tasks currently being actively processed by the GPCs 250.When a GPC 250 finishes executing a task, that task is removed from the pool of active tasks for the GPC 250, and one of the other tasks from the pool of pending tasks is selected and scheduled for execution on the GPC 250. If an active task on the GPC 250 has been idle, for example, while waiting for a data dependency to be resolved, the active task can be removed from the GPC 250 and returned to the pool of pending tasks, while another task from the pool of pending tasks is selected and scheduled for execution on the GPC 250.

[0030] The work distribution unit 225 communicates with one or more GPCs 250 via the XBar 270. The XBar 270 is a network interface that connects many PPU 200 units to other PPU 200 units. For example, the XBar 270 can be configured to connect the work distribution unit 225 to a specific GPC 250. Although not explicitly shown, one or more other PPU 200 units can also be connected to the XBar 270 via the hub.

[0031] The tasks are managed by the scheduler unit 220 and dispatched by the workload distribution unit 225 to a GPC 250. The GPC 250 is configured to process the task and generate results. The results can be taken by other tasks within the GPC 250, forwarded to another GPC 250 via the XBar 270, or stored in memory 204. The results can be written to memory 204 via the partition units 280, which implement a memory interface for reading and writing data to and from memory 204. The results can be transferred to another PPU 204 or CPU via the NVLink 210. In one embodiment, the PPU 200 contains a number U of partition units 280, corresponding to the number of separate and isolated memory devices 204 coupled to the PPU 200. A partition unit 280 is referred to below in conjunction with Fig. 3B is described in more detail.

[0032] In one embodiment, a host processor executes a driver kernel that implements an application programming interface (API) enabling one or more applications running on the host processor to schedule operations for execution on the PPU 200. In this embodiment, multiple compute applications are executed concurrently by the PPU 200, which provides isolation, quality of service (QoS), and independent address spaces for these multiple compute applications. An application can generate instructions (i.e., API calls) that cause the driver kernel to generate one or more tasks for execution by the PPU 200. The driver kernel outputs tasks to one or more streams that are processed by the PPU 200. Each task can comprise one or more groups of related threads, referred to here as a warp.In one embodiment, a Warp 32 comprises related threads that can be executed in parallel. Cooperating threads can refer to a variety of threads, including instructions for performing the task and the ability to exchange data over shared memory. Threads and cooperating threads are discussed in conjunction with... Fig. 4A described in more detail.

[0033] Fig. 3A illustrates a GPC 250 of the PPU 200 from Fig. 2 according to one embodiment. As in Fig. As shown in Figure 3A, each GPC 250 contains a number of hardware units for handling tasks. In one embodiment, each GPC 250 contains a pipeline manager 310, a pre-raster operation unit (PROP) 315, a raster engine 325, a work distribution coupling field (WDX) 380, a memory management unit (MMU) 390, and one or more data processing clusters (DPCs) 320. Note that the GPC 250 is Fig. 3A instead of or in addition to the ones in Fig. The units shown in 3A may contain other hardware units.

[0034] In one embodiment, the operation of the GPC 250 is controlled by the pipeline manager 310. The pipeline manager 310 manages the configuration of the one or more DPCs 320 for processing tasks assigned to the GPC 250. In one embodiment, the pipeline manager 310 can configure at least one of the one or more DPCs 320 to implement at least part of a graphics rendering pipeline. For example, a DPC 320 can be configured to execute a vertex shader program on the programmable streaming multiprocessor (SM) 340. The pipeline manager 310 can also be configured to forward packets received by the workload distribution unit 225 to the appropriate logical units within the GPC 250.For example, some packets can be forwarded to fixed-function hardware units in the PROP 315 and / or the Raster Engine 325, while other packets can be forwarded to the DPCs 320 for processing by the Primitive Engine 335 or the SM 340. In one embodiment, the Pipeline Manager 310 can configure at least one of the one or more DPCs 320 to implement a model for a neural network and / or a computation pipeline.

[0035] The PROP unit 315 is configured to forward data generated by the raster engine 325 and the DPCs 320 to a raster operation (ROP) unit, which, in conjunction with Fig. 3B is described in more detail. The PROP unit 315 can also be configured to perform color mixing optimizations, organize pixel data, perform address translations, and the like.

[0036] The Raster Engine 325 contains a set of fixed-function hardware units configured to perform various raster operations. In one embodiment, the Raster Engine 325 includes a setup engine, a coarse raster engine, a culling engine, a clipping engine, a fine raster engine, and a tile coalescing engine. The setup engine receives transformed vertices and generates plane equations associated with the geometric primitive defined by the vertices. The plane equations are passed to the coarse raster engine to generate coverage information (e.g., an x,y coverage mask for a tile) for the primitive.The output of the coarse raster engine is passed to the culling engine, where fragments associated with the primitive that fails a Z-test are culled (read out), and to a clipping engine, where fragments lying outside a viewing stump are clipped (cut off). Those fragments that survive clipping and culling can be passed to the fine raster engine to generate attributes for the pixel fragments based on the plane equations generated by the setup engine. The output of Raster Engine 325 includes fragments that, for example, are to be processed by a fragment shader implemented within a DPC 320.

[0037] Each DPC 320 contained within the GPC 250 includes an M-Pipe Controller (MPC) 330, a Primitive Engine 335, and one or more SMs 340. The MPC 330 controls the operation of the DPC 320 and forwards packets received by the Pipeline Manager 310 to the appropriate units within the DPC 320. For example, packets associated with a vertex can be forwarded to the Primitive Engine 335, which is configured to retrieve vertex attributes associated with that vertex from memory 204. Conversely, packets associated with a shader program can be forwarded to the SM 340.

[0038] The SM 340 comprises a programmable streaming processor configured to handle tasks represented by a set of threads. Each SM 340 is multi-threaded and configured to execute a large number of threads (e.g., 32 threads) from a given group of threads concurrently. In one embodiment, the SM 340 implements a SIMD (Single Instruction, Multiple Data) architecture, where each thread in a group of threads (e.g., a warp) is configured to process a different data set based on the same instruction set. All threads in the group of threads execute the same instructions.In another embodiment, the SM 340 implements a SIMT (Single-Instruction, Multiple-Thread) architecture, in which each thread in a group of threads is configured to process a different data set based on the same instruction set, although individual threads within the group are allowed to diverge during execution. In one embodiment, a program counter, a call stack, and an execution state are maintained for each warp, enabling concurrency between warps and serial execution within warps if threads within the warp diverge. In another embodiment, a program counter, a call stack, and an execution state are maintained for each individual thread, enabling equal concurrency between all threads, both within and between warps.If the execution state is maintained for each individual thread, threads executing the same instructions can be converged and run in parallel for maximum efficiency. The SM 340 is discussed below in conjunction with... Fig. 4A described in more detail.

[0039] The MMU 390 provides an interface between the GPC 250 and the Partition Unit 280. The MMU 390 can perform virtual address to physical address translation, memory protection, and memory request arbitration. In one embodiment, the MMU 390 provides one or more translation lookaside buffers (TLBs) to perform virtual address to physical address translation in memory 204.

[0040] Fig. Figure 3B illustrates a storage partition unit 280 of the PPU 200. Fig. 2 according to one embodiment. As in Fig. As shown in Figure 3B, the memory partition unit 280 includes a raster operations (ROP) unit 350, a level 2 cache (L2 cache) 360, and a memory interface 370. The memory interface 370 is coupled to the memory 204. The memory interface 370 can implement 32, 64, 128, 1024-bit data buses, or the like, for high-speed data transmission. In one embodiment, the PPU 200 includes U memory interfaces 370, one memory interface 370 per pair of partition units 280, with each pair of partition units 280 connected to a corresponding memory device 204. For example, the PPU 200 can be connected to up to Y memory devices 204, such as high-bandwidth memory stacks or double-data-rate graphics, version 5, synchronous dynamic direct access memory, or other types of persistent memory.

[0041] In one embodiment, the memory interface 370 implements an HBM2 memory interface, and Y corresponds to half a U. In one embodiment, the HBM2 memory stacks are located on the same physical package as the PPU 200, resulting in significant performance and footprint savings compared to conventional GDDR5 SDRAM systems. In one embodiment, each HBM2 stack contains four memory dies, and Y equals 4, with the HBM2 stack having two 128-bit channels per die for a total of 8 channels and a data bus width of 1024 bits.

[0042] In one embodiment, the memory supports single-error correcting and double-error detecting (SECDED) error correction code (ECC) to protect data. ECC offers higher reliability for computing applications that are sensitive to data corruption. Reliability is especially important in large cluster computing environments where PPUs process very large datasets and / or applications run for extended periods.

[0043] In one embodiment, the PPU 200 implements a multi-level memory hierarchy. In another embodiment, the memory partition unit 280 supports unified memory to provide a single unified virtual address space for memory of the CPU and PPU 200, enabling data sharing between virtual memory systems. In another embodiment, the frequency of accesses by a PPU 200 to memory located on other processors is tracked to ensure that memory pages are moved to the physical memory of the PPU 200 that accesses the pages more frequently. In another embodiment, the NVLink 210 supports address translation services that allow the PPU 200 to directly access page tables of a CPU and provide full access to CPU memory.

[0044] In one embodiment, copy engines transfer data between multiple PPUs 200 or between PPUs 200 and CPUs. The copy engines can generate page messes for addresses that are not entered in the page tables. The memory partition unit 280 can then handle the page messes and enter the addresses into the page table, after which the copy engine can perform the transfer. In a conventional system, memory for multiple copy engine operations is staked (i.e., not swappable) between multiple processors, which significantly reduces the available memory. With hardware page messing, addresses can be passed to the copy engines without concern about whether the memory pages are resident, and the copy operation is transparent.

[0045] Data from memory 204 or other system memory can be retrieved by the memory partition unit 280 and stored in the L2 cache 360, which is located on-chip and shared among the various GPCs 250. As shown, each memory partition unit 280 contains a portion of the L2 cache 360 ​​allocated to a corresponding memory device 204. Subordinate caches can then be implemented in different units within the GPCs 250. For example, each of the SMs 340 can implement a level-one cache (L1 cache). The L1 cache is private memory allocated to a specific SM 340. Data from the L2 cache 360 ​​can be retrieved and stored in any of the L1 caches for processing in the functional units of the SMs 340. The L2 cache 360 ​​is coupled with the memory interface 370 and the XBar 270.

[0046] The ROP unit 350 performs graphic raster operations related to pixel color, such as color compression, pixel blending, and the like. The ROP unit 350 also performs depth checks in conjunction with the Raster Engine 325, receiving a depth for a sample position associated with a pixel fragment from the Raster Engine 325's culling engine. This depth is tested against a corresponding depth in a depth buffer for the sample position associated with the fragment. If the fragment passes the depth test for the sample position, the ROP unit 350 updates the depth buffer and sends the depth test result to the Raster Engine 325. Note that the number of partition units 280 can differ from the number of GPCs 250, and therefore each ROP unit 350 can be coupled to any of the GPCs 250.The ROP unit 350 tracks the packets received by the various GPCs 250 and determines to which GPC 250 a result generated by the ROP unit 350 is routed by the Xbar 270. Although the ROP unit 350 is located in... Fig. While ROP unit 350 is contained within storage partition unit 280 in one embodiment, it may be located outside of storage partition unit 280 in another embodiment. For example, ROP unit 350 may be located in GPC 250 or another unit.

[0047] Fig. Figure 4A illustrates the 340 streaming multiprocessor from Fig. 3A according to one embodiment. As in Fig. As shown in Figure 4A, the SM 340 contains an instruction cache 405, one or more scheduler units 410(K), a register file 420, one or more processing cores 450, one or more special function units (SFUs) 452, one or more load / store units (LSUs) 454, a link network 480 and a shared memory / L1 cache 470.

[0048] As described above, the workload distribution unit 225 prepares tasks for execution on the GPCs 250 of the PPU 200. The tasks are assigned to a specific DPC 320 within a GPC 250, and if the task is associated with a shader program, it can be assigned to an SM 340. The scheduler unit 410(K) receives the tasks from the workload distribution unit 225 and manages instruction scheduling for one or more thread blocks assigned to the SM 340. The scheduler unit 410(K) schedules thread blocks for execution as warps of parallel threads, with each thread block assigned at least one warp. In one embodiment, each warp executes 32 threads.The Scheduler Unit 410(K) can manage a variety of different thread blocks, assign warps to the different thread blocks, and then, during each clock cycle, dispatch instructions from the variety of different cooperative groups to the various functional units (i.e., compute cores 450, SFUs 452, and LSUs 454).

[0049] Cooperative Groups is a programming model for organizing groups of communicating threads that allows developers to express the granularity at which threads communicate, enabling richer, more efficient parallel decompositions. Cooperative start APIs support synchronization between thread blocks for the execution of parallel algorithms. Traditional programming models offer a single, simple construct for synchronizing cooperating threads: a barrier across all threads in a thread block (i.e., the `syncthreads()` function). However, programmers often want to define groups of threads with a smaller granularity than thread blocks and synchronize within those defined groups to enable greater performance, design flexibility, and software reuse in the form of common group-wide function interfaces.

[0050] Cooperative Groups allows programmers to explicitly define groups of threads at the subblock level (i.e., as small as a single thread) and multiblock granularities, and to perform collective operations such as synchronizing threads within a cooperative group. The programming model supports clean composition across software boundaries, enabling libraries and utility functions to safely synchronize within their local context without making assumptions about convergence. Cooperative Groups primitives enable new patterns of cooperative concurrency, including producer-consumer parallelism, opportunistic parallelism, and global synchronization across an entire network of thread blocks.

[0051] A dispatch unit 415 is configured to send instructions to one or more of the functional units. In this embodiment, the scheduler unit 410(K) contains two dispatch units 415, enabling it to send two different instructions from the same warp during each clock cycle. In alternative embodiments, each scheduler unit 410(K) can contain a single dispatch unit 415 or additional dispatch units 415.

[0052] Each SM 340 contains a register file 420, which provides a set of registers for the functional units of the SM 340. In one embodiment, the register file 420 is partitioned among the individual functional units, so that each functional unit is allocated a specific portion of the register file 420. In another embodiment, the register file 420 is partitioned among the various warps executed by the SM 340. The register file 420 provides temporary storage for operands associated with the data paths of the functional units.

[0053] Each SM 340 comprises L processing cores 450. In one embodiment, the SM 340 contains a large number (e.g., 128, etc.) of separate processing cores 450. Each processing core 450 can contain a full-pipeline, single-precision, single-precision, double-precision, and / or mixed-precision processing unit, which includes a floating-point arithmetic logic unit and an integer arithmetic logic unit. In one embodiment, the floating-point arithmetic logic units implement the IEEE 744-2008 standard for floating-point arithmetic. In one embodiment, the processing cores contain 450 64 single-precision (32-bit) floating-point cores, 64 integer cores, 32 double-precision (64-bit) floating-point cores, and 8 tensor cores.

[0054] Tensor cores are configured to perform matrix operations, and in one embodiment, one or more tensor cores are contained within the 450 cores. In particular, the tensor cores are configured to perform deep learning matrix arithmetic, such as convolution operations for training and inferencing neural networks. In one embodiment, each tensor core operates on a 4x4 matrix and performs a matrix multiplication and accumulation operation D = A × B + C, where A, B, C, and D are 4x4 matrices.

[0055] In one embodiment, the matrix multiplication inputs A and B are 16-bit floating-point matrices, while the accumulation matrices C and D can be either 16-bit or 32-bit floating-point matrices. Tensor cores operate on 16-bit floating-point input data with 32-bit floating-point accumulation. The 16-bit floating-point multiplication requires 64 operations and results in a highly precise product, which is then accumulated with the other intermediate products using 32-bit floating-point addition for a 4x4x4 matrix multiplication. In practice, tensor cores are used to perform much larger two-dimensional or higher-dimensional matrix operations composed of these smaller elements. An API, such as... For example, the CUDA 9 C++ API provides special matrix load, matrix multiplication and accumulation, and matrix memory operations to efficiently utilize tensor processing cores from a CUDA C++ program.At the CUDA level, the interface at the warp level requires matrices of size 16x16, spanning all 32 threads of the warp.

[0056] Each SM 340 also contains M SFUs 452 that perform special functions (e.g., attribute evaluation, reciprocal square root, and the like). In one embodiment, the SFUs 452 can include a tree traverse unit configured to traverse a hierarchical tree data structure. In another embodiment, the SFUs 452 can include a texture unit configured to perform texture mapping filtering operations. In one embodiment, the texture units are configured to load texture mappings (e.g., a two-dimensional array of texels) from memory 204 and sample the texture mappings to produce sampled texture values ​​for use in shader programs executed by the SM 340. In one embodiment, the texture mappings are stored in shared memory / L1 cache 370. The texture units implement texture operations such as filtering operations using mipmaps (i.e.,Texture images with varying levels of detail). In one embodiment, each SM 240 contains two texture units.

[0057] Each SM 340 also includes N LSUs 454, which implement load and store operations between the shared memory / L1 cache 470 and the register file 420. Each SM 340 contains a linking network 480 that connects each of the functional units to the register file 420 and the LSUs 454 to the register file 420 and the shared memory / L1 cache 470. In one embodiment, the linking network 480 is a coupling array that can be configured to connect any of the functional units to any of the registers in the register file 420 and to connect the LSUs 454 to the register file and memory locations in the shared memory / L1 cache 470.

[0058] The shared memory / L1 cache 470 is an array of on-chip memory that enables data storage and communication between the SM 340 and the primitive engine 335, as well as between threads in the SM 340. In one embodiment, the shared memory / L1 cache 470 comprises 128 KB of memory capacity and is located in the path from the SM 340 to the partition unit 280. The shared memory / L1 cache 470 allows read and write accesses to be cached. One or more of the shared memory / L1 cache 470, L2 cache 360, and memory 204 serve as backup memory.

[0059] Combining data caching and shared memory functionality in a single memory block offers the best overall performance for both types of memory access. The capacity can be used as a cache by programs that do not use shared memory. For example, if shared memory is configured to use half of its capacity, texture and load / save operations can utilize the remaining capacity. Integration with Shared Memory / L1 Cache 470 allows Shared Memory / L1 Cache 470 to act as a high-throughput lane for streaming data while also providing high-bandwidth, low-latency access to frequently reused data.

[0060] When configured for universal parallel processing, a simpler configuration can be used compared to graphics processing. In particular, the following are used: Fig. The two fixed-function graphics processing units (GPUs) shown are bypassed, resulting in a much simpler programming model. In the universal parallel computing configuration, the workload distribution unit (WDU) 225 directly assigns and distributes thread blocks to the DPCs 320. Threads within a block execute the same program, using a unique thread ID during computation to ensure each thread produces unique results. The SM 340 is used to execute the program and perform computations, the shared memory / L1 cache 470 to communicate between threads, and the LSU 454 to read from and write to global memory via the shared memory / L1 cache 470 and the memory partition unit 280. When configured for universal parallel computing, the SM 340 can also write instructions that allow the scheduler unit 220 to start new work on the DPCs 320.

[0061] The PPU 200 can be included in a desktop computer, laptop computer, tablet computer, server, supercomputer, smartphone (e.g., a wireless wearable device), personal digital assistant (PDA), digital camera, vehicle, head-mounted display, portable electronic device, and the like. In one embodiment, the PPU 200 is implemented on a single semiconductor substrate. In another embodiment, the PPU 200 is included in a system-on-a-chip (SoC) together with one or more other devices, such as additional PPUs 200, memory 204, a reduced instruction set computing (RISC) CPU, a memory management unit (MMU), a digital-to-analog converter (DAC), and the like.

[0062] In one embodiment, the PPU 200 can be included on a graphics card that contains one or more memory devices 204. The graphics card can be configured to provide an interface connection to a PCIe slot on a desktop computer motherboard. In another embodiment, the PPU 200 can be an integrated graphics processing unit (iGPU) or a parallel processor included in the motherboard chipset. Example computing system

[0063] Systems with multiple GPUs and CPUs are deployed across numerous industries as developers uncover and leverage greater parallelism in applications such as artificial intelligence. High-performance GPU-accelerated systems with tens of thousands of compute nodes are used in data centers, research facilities, and supercomputers to tackle increasingly complex problems. As the number of processing units within these high-performance systems grows, communication and data transmission mechanisms must scale to support the increased bandwidth.

[0064] Fig. 4B is a conceptual diagram of a [project] using the PPU 200 from [company name]. Fig. 2 implemented processing system 400 according to one embodiment. The example system 465 can be configured to perform the processing system described in Fig. The method shown in 100 is implemented. The processing system 400 contains a CPU 430, a switch 410, and several PPUs 200 and corresponding memory 204. The NVLink 210 provides high-speed communication links between each of the PPUs 200. Although in Fig. Figure 4B shows a specific number of connections via NVLink 210 and Link 202; the number of connections to each PPU 200 and CPU 430 can vary. Switch 410 provides an interface between Link 202 and CPU 430. The PPUs 200, memory 204, and NVLinks 210 can be located on a single semiconductor platform to form a parallel processing module 425. In one embodiment, Switch 410 supports two or more protocols for interface between different links.

[0065] In another embodiment (not shown), the NVLink 210 provides one or more high-speed communication links between each of the PPUs 200 and the CPU 430, and the Switch 410 provides interfaces between the Link 202 and each of the PPUs 200. The PPUs 200, the Memory 204, and the Link 202 can be located on a single semiconductor platform to form a Parallel Processing Module 425. In yet another embodiment (not shown), the Link 202 provides one or more communication links between each of the PPUs 200 and the CPU 430, and the Switch 410 provides interfaces between each of the PPUs 200 using the NVLink 210 to provide one or more high-speed communication links between the PPUs 200.In another embodiment (not shown), the NVLink 210 provides one or more high-speed communication links between the PPUs 200 and the CPU 430 via the switch 410. In yet another embodiment (not shown), the link 202 provides one or more communication links directly between each of the PPUs 200. One or more of the high-speed communication links NVLink 210 can be implemented as a physical NVLink link or as an on-chip or on-die link using the same protocol as the NVLink 210.

[0066] In the context of this description, a single semiconductor platform can refer to a single, unified semiconductor-based integrated circuit fabricated on a die or chip. Note that the term single semiconductor platform can also refer to multi-chip modules with enhanced connectivity that simulate on-chip operation and offer significant improvements over conventional bus implementations. Naturally, the various circuits or devices can also be arranged separately or in various combinations of semiconductor platforms, according to the user's requirements. Alternatively, the parallel processing module 425 can be implemented as a printed circuit board substrate, and each of the PPUs 200 and / or memory 204 can be implemented as a packaged device. In one embodiment, the CPU 430, the switch 410, and the parallel processing module 425 are located on a single semiconductor platform.

[0067] In one embodiment, the signal transmission rate of each NVLink 210 is 20 to 25 gigabits per second, and each PPU 200 contains six NVLink 210 interfaces (as shown in Fig. As shown in Figure 4B, five NVLink 210 interfaces are included for each PPU 200. Each NVLink 210 provides a data transfer rate of 25 gigabytes per second in each direction, with six links delivering 300 gigabytes per second. The NVLinks 210 may be used exclusively for PPU-to-PPU communication, as shown in Figure 4B. Fig. 4B shown, or for any combination of PPU-to-PPU and PPU-to-CPU if the CPU 430 also includes one or more NVLink 210 interfaces.

[0068] In one embodiment, the NVLink 210 enables direct load / store / atomic access from the CPU 430 to the memory 204 of each PPU 200. In another embodiment, the NVLink 210 supports coherence operations, allowing data read from the memory 204 to be stored in the cache hierarchy of the CPU 430, thus reducing cache access latency for the CPU 430. In another embodiment, the NVLink 210 includes support for Address Translation Services (ATS), allowing the PPU 200 to directly access page tables within the CPU 430. One or more of the NVLinks 210 can also be configured to operate in a low-power mode.

[0069] Fig. Figure 4C illustrates an example system 465 in which the various architectures and / or functionalities of the different earlier embodiments may be implemented. The example system 465 can be configured to implement the features shown in Figure 4C. Fig. 1 of the methods shown are implemented in 100.

[0070] As shown, a System 465 is provided, which includes at least one central processing unit 430 connected to a communication bus 475. The communication bus 475 can be implemented using any suitable protocol, such as PCI (Peripheral Component Interconnect), PCI Express, AGP (Accelerated Graphics Port), HyperTransport, or any other bus or point-to-point communication protocol(s). The System 465 also includes main memory 440. Control logic (software) and data are stored in main memory 440, which can be implemented as random-access memory (RAM).

[0071] The System 465 also includes input devices 460, the parallel processing system 425, and display devices 445, such as a conventional CRT (cathode ray tube), LCD (liquid crystal display), LED (light-emitting diode), plasma display, or the like. User input can be received by the input devices 460, such as a keyboard, mouse, touchpad, microphone, and the like. Each of the aforementioned modules and / or devices can even be located on a single semiconductor platform to form the System 465. Alternatively, the various modules can also be arranged separately or in various combinations of semiconductor platforms according to the user's requirements.

[0072] Furthermore, the system 465 can be coupled to a network (e.g., a telecommunications network, local area network (LAN), wireless network, wide area network (WAN) such as the Internet, peer-to-peer network, cable network, or the like) via a network interface 435 for communication purposes.

[0073] The System 465 may also include secondary storage (not shown). The secondary storage may include, for example, a hard disk drive and / or a removable storage drive, which could be a floppy disk drive, a magnetic tape drive, a CD drive, a DVD drive, a recording device, or a USB flash drive. The removable storage drive reads from and / or writes to a removable storage device in a known manner.

[0074] Computer programs or computer control logic algorithms can be stored in main memory (440) and / or secondary memory. Such computer programs enable the System 465 to perform various functions when executed. Main memory (440), secondary memory, and / or any other storage media are possible examples of computer-readable media.

[0075] The architecture and / or functionality of the various preceding figures can be implemented in the context of a general-purpose computer system, a circuit board system, a game console system for entertainment purposes, an application-specific system, and / or any other desired system. For example, System 465 can take the form of a desktop computer, a laptop computer, a tablet computer, servers, supercomputers, a smartphone (e.g., a wireless portable device), a personal digital assistant (PDA), a digital camera, a vehicle, a head-mounted display, a portable electronic device, a mobile phone, a television, a workstation, a game console, an embedded system, and / or any other type of logic.

[0076] Although various embodiments have been described above, it is understood that they are presented only as examples and not as limitations. Therefore, the scope and extent of protection of a preferred embodiment are not to be limited by any of the embodiments described above, but only in accordance with the following claims and their equivalents. Graphics processing pipeline

[0077] In one embodiment, the PPU 200 includes a graphics processing unit (GPU). The PPU 200 is configured to receive instructions specifying shader programs for processing graphics data. Graphics data can be defined as a set of primitives, such as points, lines, triangles, quadrilaterals, triangular strips, and the like. Typically, a primitive contains data specifying a number of vertices (or corner points) for the primitive (e.g., in a model-space coordinate system), as well as attributes associated with each vertex of the primitive. The PPU 200 can be configured to process the graphics primitives to generate a frame buffer (i.e., pixel data for each pixel of the display).

[0078] An application writes model data for a scene (i.e., a collection of vertices and attributes) to memory, such as system memory or memory 204. The model data defines each of the objects that can be visible on a display. The application then makes an API call to the driver kernel, requesting that the model data be rendered and displayed. The driver kernel reads the model data and writes commands to one or more streams to perform operations on the model data. These commands can refer to various shader programs to be implemented on the PPU 200's SMs 340, including one or more vertex shaders, hull shaders, domain shaders, geometry shaders, and pixel shaders.For example, one or more SMs 340 can be configured to run a vertex shader program that processes a number of vertices defined by the model data. In one embodiment, the different SMs 340 can be configured to run different shader programs concurrently. For example, a first subset of SMs 340 can be configured to run a vertex shader program, while a second subset of SMs 340 can be configured to run a pixel shader program. The first subset of SMs 340 processes vertex data and writes the processed vertex data to the L2 cache 360 ​​and / or memory 204. After the processed vertex data is rasterized (i.e., pixelated), the second subset of SMs 340...(Three-dimensional data transformed into two-dimensional data in screen space) to generate fragment data, the second subset of SMs 340 executes a pixel shader to generate processed fragment data, which is then mixed with other processed fragment data and written to the image buffer in memory 204. The vertex shader program and the pixel shader program can be executed concurrently, processing different data from the same scene according to a pipelined approach, until all model data for the scene has been rendered into the image buffer. The contents of the image buffer are then transferred to a display controller for display on a display device.

[0079] Fig. Figure 5 is a conceptual diagram of a PPU from Fig. 2. Implemented graphics processing pipeline according to one embodiment. The graphics processing pipeline 500 is an abstract flowchart of the processing steps implemented to generate 2D computer images from 3D geometry data. As is known, pipeline architectures can perform long-latency operations more efficiently by dividing the operation into a multitude of stages, with the output of each stage coupled to the input of the next stage. Thus, the graphics processing pipeline 500 receives input data 501, which is transferred from one stage to the next stage of the graphics processing pipeline 500 to generate output data 502. In one embodiment, the graphics processing pipeline 500 can implement an OpenGL ®The graphics processing pipeline is defined by the API. Optionally, the graphics processing pipeline 500 can be implemented in the context of the functionality and architecture of the preceding figures and / or any subsequent figure(s).

[0080] As in Fig. As shown in Figure 5, the graphics processing pipeline 500 has a multi-stage pipeline architecture. The stages include, but are not limited to, a data build-up stage 510, a vertex shading stage 520, a primitive build-up stage 530, a geometry shading stage 540, a viewport scaling, culling, and clipping (VSCC) stage 550, a rasterization stage 560, a fragment shading stage 570, and a raster operations stage 580. In one embodiment, the input data 501 includes instructions that configure the processing units to implement the stages of the graphics processing pipeline 500 and geometric primitives (e.g., points, lines, triangles, squares, triangular strips, or fans, etc.) to be processed by the stages. The output data 502 can include pixel data (i.e., color data) that is copied into an image buffer or some other type of surface data structure in memory.

[0081] The data assembly stage 510 receives the input data 501, which specifies vertex data for higher-order surfaces, primitives, or the like. The data assembly stage 510 collects the vertex data in temporary storage or a queue, for example, by receiving a command from the host processor containing a pointer to a buffer in memory and reading the vertex data from the buffer. The vertex data is then transferred to the vertex shading stage 520 for processing.

[0082] The vertex shading level 520 processes vertex data by executing a series of operations (i.e., a vertex shader or program) once for each of the vertices. Vertices can be represented, for example, as a 4-coordinate vector (i.e.,<x, y, z, w> The Vertex Shading Level 520 can handle individual vertex attributes such as position, color, texture coordinates, and the like. In other words, the Vertex Shading Level 520 performs operations on the vertex coordinates or other vertex attributes associated with a vertex. Such operations typically include illumination operations (i.e., modifying color attributes for a vertex) and transformation operations (i.e., modifying the coordinate space for a vertex).For example, vertices can be specified using coordinates in an object coordinate space, which are transformed by multiplying the coordinates with a matrix that translates the coordinates from the object coordinate space into a world space or an NCD space (space with normalized device coordinates). The vertex shading stage 520 produces transformed vertex data, which is then transferred to the primitive development stage 530.

[0083] The primitive build-up stage 530 collects vertices output by the vertex shading stage 520 and groups them into geometric primitives for processing by the geometry shading stage 540. For example, the primitive build-up stage 530 can be configured to always group three consecutive vertices as one geometric primitive (i.e., a triangle) for transfer to the geometry shading stage 540. In some embodiments, certain vertices can be reused for consecutive geometric primitives (e.g., two consecutive triangles in a triangular strip can share two vertices). The primitive build-up stage 530 transfers geometric primitives (i.e., a collection of associated vertices) to the geometry shading stage 540.

[0084] The geometry shading stage 540 processes geometric primitives by performing a series of operations (i.e., a geometry shader or program) on them. Tessellation operations can create one or more geometric primitives from any geometric primitive. In other words, the geometry shading stage 540 can subdivide any geometric primitive into a finer mesh of two or more geometric primitives for processing by the rest of the graphics processing pipeline 500. The geometry shading stage 540 then passes geometric primitives to the viewport SCC stage 550.

[0085] In one embodiment, the graphics processing pipeline 500 can operate within a streaming multiprocessor, and the vertex shading stage 520, the primitive build-up stage 530, the geometry shading stage 540, the fragment shading stage 570, and / or associated hardware or software can perform sequential processing operations. Once the sequential processing operations are complete, the viewport SCC stage 550 can use the data in one embodiment. In another embodiment, primitive data processed by one or more stages of the graphics processing pipeline 500 can be written to a cache (e.g., an L1 cache, a vertex cache, etc.). In this case, the viewport SCC stage 550 can access the data in the cache in one embodiment. In another embodiment, the viewport SCC stage 550 and the rasterization stage 560 are implemented as fixed-function circuits.

[0086] The Viewport SCC Level 550 performs scaling, culling, and clipping of geometric primitives. Each surface being rendered is associated with an abstract camera position. The camera position represents the location of a viewer looking at the scene and defines a viewing stump that encompasses the objects in the scene. The viewing stump can include a viewing plane, a back plane, and four clipping planes. Any geometric primitive that lies entirely outside the viewing stump can be culled (i.e., discarded) because the geometric primitive does not contribute to the final rendered scene. Any geometric primitive that lies partly inside and partly outside the viewing stump can be clipped (i.e., transformed into a new geometric primitive that is enclosed within the viewing stump).Furthermore, geometric primitives can be scaled based on the depth of the viewing truncation. All potentially visible geometric primitives are then transferred to the 560 pixel resolution.

[0087] The 560 rasterization stage converts the 3D geometric primitives into 2D fragments (for use in displays, etc.). The 560 rasterization stage can be configured to use the vertices of the geometric primitives to create a set of plane equations from which various attributes can be interpolated. The 560 rasterization stage can also compute a cover mask for a variety of pixels, indicating whether one or more sample positions for the pixel will truncate the geometric primitive. In one embodiment, Z-testing can also be performed to determine whether the geometric primitive is truncated by other geometric primitives that have already been rasterized. The 560 rasterization stage produces fragment data (i.e., interpolated vertex attributes associated with a specific sample position for each covered pixel), which is passed to the 570 fragment shading stage.

[0088] Fragment Shading Stage 570 processes fragment data by performing a series of operations (e.g., a fragment shader or program) on each fragment. Fragment Shading Stage 570 can generate pixel data (i.e., color values) for the fragment, for example, by performing illumination operations or sampling texture mappings using interpolated texture coordinates for the fragment. Fragment Shading Stage 570 generates pixel data that is then passed to Raster Operations Stage 580.

[0089] Raster operations stage 580 can perform various operations on the pixel data, such as alpha testing, stencil testing, and blending the pixel data with other pixel data corresponding to other fragments associated with the pixel. Once raster operations stage 580 has completed processing the pixel data (i.e., the output data 502), the pixel data can be written to a rendering destination, such as an image buffer, a color buffer, or the like.

[0090] Note that one or more additional stages may be included in the graphics processing pipeline 500 in addition to or instead of one or more of the stages described above. Different implementations of the abstract graphics processing pipeline may implement different stages. Furthermore, in some embodiments, one or more of the stages described above (such as the geometry shading stage 540) may be excluded from the graphics processing pipeline. Other types of graphics processing pipelines are considered to be within the scope of protection of this disclosure. In addition, each of the stages of the graphics processing pipeline 500 may be implemented by one or more dedicated hardware units within a graphics processor, such as the PPU 200. Other stages of the graphics processing pipeline 500 may be implemented by programmable hardware units, such as...The SM 340 of the PPU 200 will be implemented.

[0091] The Graphics Processing Pipeline 500 can be implemented via an application running on a host processor, such as a CPU. In one embodiment, a device driver can implement an application programming interface (API) that defines various functions that an application can use to generate graphical data for display. The device driver is a software program containing a variety of instructions that control the operation of the PPU 200. The API provides an abstraction for a programmer, allowing them to use specialized graphics hardware, such as the PPU 200, to generate the graphical data without having to use the specific instruction set for the PPU 200. The application can include an API call that is forwarded to the device driver for the PPU 200.The device driver interprets the API call and performs various operations to respond to it. In some cases, the device driver can perform operations by executing instructions on the CPU. In other cases, the device driver can perform at least some operations by initiating operations on the PPU 200 via an input / output interface between the CPU and the PPU 200. In one embodiment, the device driver is configured to implement the graphics processing pipeline 500 using the hardware of the PPU 200.

[0092] Within the PPU 200, various programs can be executed to implement the different stages of the 500 graphics processing pipeline. For example, the device driver can start a kernel on the PPU 200 to perform the 520 vertex shading stage on an SM 340 (or multiple SMs 340). The device driver (or the initial kernel executed by the PPU 300) can also start other kernels on the PPU 300 to execute other stages of the 500 graphics processing pipeline, such as the 540 geometry shading stage and the 570 fragment shading stage. Furthermore, some stages of the 500 graphics processing pipeline can be implemented on fixed-unit hardware, such as a rasterizer or a data assembler implemented within the PPU 200.Note that results from a kernel may be processed by one or more intermediate fixed-function hardware units before being processed by a subsequent kernel on an SM 340. Machine learning

[0093] Deep neural networks (DNNs), developed on processors such as the PPU 200, have been used for a variety of applications, from self-driving cars to accelerated drug development, from automatic image capture in online image databases to intelligent real-time speech translation in video chat applications. Deep learning is a technique that models the neural learning process of the human brain, continuously learning, becoming increasingly intelligent, and delivering faster and more accurate results over time. A child is first taught by an adult to correctly identify and classify different shapes, eventually being able to identify shapes without further instruction. Similarly, a deep learning or neural learning system in object recognition and classification must be trained to become more intelligent and efficient at identifying basic objects, hidden objects, and so on.to identify and assign context to the objects.

[0094] At its simplest level, neurons in the human brain consider various inputs they receive, assign importance levels to each input, and send outputs to other neurons for action. An artificial neuron, or perceptron, is the most basic model of a neural network. For example, a perceptron might receive one or more inputs representing different features of an object. The perceptron is trained to recognize and classify these features, and each is assigned a specific weight based on its importance in defining the object's shape.

[0095] A DNN model contains multiple layers of many interconnected perceptrons (e.g., nodes) that can be trained on enormous amounts of input data to solve complex problems quickly and with high accuracy. In one example, a first layer of the DLL model decomposes an input image of a car into its constituent parts and looks for basic patterns such as lines and angles. The second layer reassembles the lines to search for higher-level patterns such as wheels, windshields, and mirrors. The next layer identifies the vehicle type, and the final few layers generate a label for the input image that identifies the model as belonging to a specific vehicle brand.

[0096] Once the DNN is trained, it can be deployed and used to identify and classify objects or patterns in a process known as inference. Examples of inference (the process by which a DNN extracts useful information from a given input) include identifying handwritten numbers on checks deposited at ATMs, identifying pictures of friends in photos, providing movie recommendations to over fifty million users, identifying and classifying different types of automobiles, pedestrians, and road hazards in autonomous vehicles, or translating human language in real time.

[0097] During training, data flows through the DNN in a forward propagation phase until a prediction is generated that displays a label corresponding to the input. If the neural network does not label the input correctly, errors between the correct label and the predicted label are analyzed, and the weights for each feature are adjusted during a backward propagation phase until the DNN correctly labels the input and other inputs in a training dataset. Training complex neural networks requires enormous amounts of parallel computing power, including floating-point multiplication and addition, which is supported by the PPU 200.Inference is less computationally intensive than training, as it is a latency-sensitive process in which a trained neural network is applied to new inputs that have not existed before, in order to classify images, translate speech, and generally infer new information.

[0098] Neural networks rely heavily on mathematical matrix operations, and complex multilayer networks require enormous amounts of floating-point performance and bandwidth for efficiency and speed. With thousands of processing cores optimized for mathematical matrix operations and delivering tens to hundreds of TFLOPS of performance, the PPU 200 is a computing platform capable of providing the performance required for deep neural network-based artificial intelligence and machine learning applications. Learning binary remainder representations for domain-specific video streaming

[0099] Video streaming services are a popular way to watch entertainment content today. Due to large video sizes and limited network bandwidth, video compression is necessary to stream video content from servers to clients. While video compression can reduce the size of a video, it often comes with undesirable compression artifacts such as frame blocking and blurring.

[0100] In one embodiment, we hypothesize that the residual information can be effectively compressed to improve video compression performance if one is willing to restrict the use of the video compression method to a specific area. In other words, we no longer want a video compression method that works universally well for all videos. We only want a method that performs exceptionally well in a particular area. This setting may be well suited to video streaming applications such as video game streaming and sports streaming. For example, with video game streaming services, the player selects a game title to play, and the video content is rendered on the server and transmitted to the customer's mobile control panel.

[0101] During gameplay, all video content in the stream resides within the same video game territory, making territory-specific video compression entirely appropriate. This approach also suits other use cases, such as streaming sports, which are often limited to a specific discipline, and even compressing dashcam footage, as all videos depict street scenes.

[0102] To verify our hypothesis, we use deep learning models, which have proven to be powerful approximators for nonlinear functions, to encode the highly nonlinear residual information. In our video compression pipeline, we first apply H.264 to compress videos to a specific domain and then train a novel binary autoencoder to encode the resulting residual information frame by frame into a binary representation.

[0103] We then apply Huffman coding to compress the binary representations losslessly. The compressed binary representations can be sent to the client in the metadata field of the H.264 streaming packet. In this way, our method can be integrated into the existing video streaming standard. We show that with our proposed binary residue representation, better video quality (for a given bandwidth) can be achieved by sending the video stream using H.264 with a lower bandwidth and using the saved bandwidth to transmit the learned binary residue representation.

[0104] We are conducting extensive experiments to verify our hypothesis that we can improve state-of-the-art video compression methods by learning to encode the domain-specific residual information in binary form. We are also comparing different methods for training the proposed binary autoencoder to encode the residual information. Region-specific video compression with REST autoencoder

[0105] First, we present our video compression pipeline for streaming region-specific videos. Then, we present the details of our autoencoder for encoding the remaining information into binary forms, along with the training methods. Video streaming pipeline

[0106] An example pipeline consists of two modules: a video compression module and a deep learning-based autoencoder. The video compression module uses the H.264 standard, which has demonstrated good performance in compressing temporally smooth video signals, while our rest autoencoder helps to recover information lost during compression on the client side by leveraging the fact that the videos being compressed originate from the same region. With such a hybrid approach, we can not only improve output quality with minimal effort but also adapt the system to existing compression platforms and train it by utilizing extensive datasets for specific regions.

[0107] Fig. Figure 6 illustrates an example overview of a video streaming pipeline 600 according to one embodiment. The pipeline 600 consists of two modules: a conventional H.264 module 602 and a rest autoencoder module 604. The input to our rest autoencoder module 604 is the difference 606 between the original video 608 and the compressed video 610. The difference 606 is encoded and binarized to produce binary representations. We use a Huffman encoder 612 to further compress the binary representations losslessly into a bitstream. On the client side, we decode the video using a Huffman decoder 614 and a rest decoder 616 and reconstruct the output video 618 by adding the decoded difference 620 back to the compressed video 622.

[0108] Note that while we use H.264 in our pipeline, other video compression standards such as MPEG4 and HEVC can also be used. Given an input video X, we obtain the compressed video Y by applying H.264. The difference between the two videos is called the residual information R = X - Y. The greater the residual information, the lower the quality of the compressed video. Also note that R is not included in Y because it consists of highly nonlinear patterns that cannot be effectively compressed using conventional methods.

[0109] We claim that by limiting the video area, we were able to use a novel autoencoder to effectively compress the remaining information. The autoencoder consists of a pair of functions. (E,D), where the encoder the function R maps to a binary mapping and the decoder The function R reconstructs the binary mapping on the client side. The recovered residual information is denoted as R̂, and the final output video Y+R̂ has better image quality than Y alone. We note that the binary mapping is further reduced to a bitstream using the Huffman coding algorithm, which is asymptotically optimal for reducing bandwidth usage.

[0110] Sending the bitstream containing the remaining information requires additional bandwidth. However, we can train an autoencoder that requires significantly less bandwidth to compress the remaining information than H.264. Therefore, we can run the H.264 standard at a higher compression rate, which uses less bandwidth but results in a larger remaining signal. We then apply our autoencoder to compress the remaining signal into a small bitstream. Considering a scenario where the bandwidth for a video stream is 5 Mbps, we can use the proposed pipeline to compress the video to 4 Mbps using H.264 and use the remaining 1 Mbps to send the remaining signal. Because our autoencoder is more efficient than H.264 at compressing the remaining signal, our system performs better than a basic system that allocates all 5 Mbps to H.264.

[0111] One might ask why the H.264 standard isn't completely replaced by the proposed residual autoencoder. We maintain that our residual autoencoder is only more efficient than H.264 at compressing the residual signal. The carefully engineered H.264 standard is more efficient at compressing the core video. By combining the strengths of H.264 and the proposed autoencoder, our hybrid system achieves superior performance.

[0112] Furthermore, our pipeline integrates easily into the existing H.264 standard, as the residual information can be appended to the meta field of an H.264 streaming packet. Therefore, we can leverage the popularity of the H.264 standard and various hardware accelerators implemented for H.264.

[0113] We note that in the proposed region-specific video streaming pipeline, it may be necessary to send the parameters of D and the Huffman decoder table to the client for the streaming service, which requires additional bandwidth. However, the parameters can be sent before streaming begins. Once the parameters are sent, the user can enjoy the low latency and high video quality features of the proposed video streaming pipeline. Binary Residue Autoencoder

[0114] We are designing an autoencoder consisting of three components: encoder , Binary and decoder For the encoder, a sample goal is to learn how to extract compact feature maps for the subsequent binarizer. We use L convolutional layers in our encoder, where each layer has the same number of channels C and a step size of two, which downclocks feature maps. The binarizer converts the output of the last convolutional layer into a binary map. For the decoder, our goal is to upclock the binary map back to the original input. Our decoder has L convolutional layers.

[0115] At the end of each convolution layer, a subpixel layer is used for upclocking. Since we aim to upclock the feature mapping resolution by two in each spatial dimension, the channel count of each convolution layer in the decoder is set to 4 XC due to the use of the subpixel layer. Batch normalization and ReLU layers are used to facilitate the learning process.

[0116] Fig. Figure 7 illustrates an example binary rest autoencoder architecture 700 according to one embodiment. In one embodiment, the binary rest autoencoder architecture 700 processes the video frame by frame. The autoencoder architecture 700 consists of an encoder 702, a binarizer 704, and a decoder 706. The encoder 702 and the decoder 706 each have L convolution layers, and each layer contains C / 4C channels. For simplicity, Huffman coding modules are not shown.

[0117] We encode and decode the residual signal R frame by frame. Let frig be a set of residual frames computed by applying H.264 with a target bit rate. We train our autoencoder by solving the following optimization problem: minD,E∑i‖ri−D(B(E(ri)))‖22

[0118] We are interested in the bandwidth required for transmitting binary representations, which is determined by two factors: 1) the number of layers L in the encoder and 2) the number of channels C. Let W and H be the width and height of the input image, where the size of the binary representation is given by C×W×H22L Given the given parameters, a large number of L and a smaller number of C would result in a smaller encoder output and thus a smaller binary mapping. However, a smaller encoder output makes training the autoencoder more difficult. Training the binary residual autoencoder

[0119] Binarizing feature mappings in neural networks can be used to reduce storage requirements in mobile devices. They are also used for image compression. We will discuss various binarization methods. formulation

[0120] The output feature mapping of be ei=E(ri). Our Binary Maker aims to produce the binary output {-1; 1} -1 to generate such binary outputs. The process consists of... from two parts: 1) Create each element of the encoder output e i 1) Discretize it to the interval [-1, 1], and 2) Discretize it to {-1, 1}, given by: B(ei)=b(σ(ei)) where σ and b are the activation and discretization functions, respectively. We then discuss various functions for activation (e.g., Tanh, HardTanh, Sigmoid) and various methods for binarization (e.g., stochastic regularization, Gumbel noise, etc.). Tanh / HardTanh Activation

[0121] Tanh is a common activation used to project feature mappings onto the interval [-1, 1], as is its approximation version, the HardTanh function. Here, we define the binarized output as: b(z)={1,if z≥0−1,if z<0, where z = σ (e i ) the function Tanh or HardTanh. However, since binarization is not a differentiable function, we cannot train the proposed autoencoder using backward propagation. To avoid this problem, we use a piecewise function b. bp during reverse propagation: bbp(z)={1,if z>1z,if −1≤z≤1−1if z<−1.

[0122] Using the straight-through estimator, we can calculate the gradient of b bp (z) calculate and pass the gradient through b unchanged as: bbp'(z)={1,if −1≤z≤10,else. Sigmoid activation

[0123] The sigmoid function outputs a value in the interval [0, 1]. We convert the output by applying z = 2(σ(e) i )- 0.5) to the interval of [-1, 1]. We can then use the binarization and training method described in the previous paragraph. Stochastic regularization

[0124] In one embodiment, we integrate a stochastic process in (3) using Tanh activation: b(z)=z+ϵ, where ε is randomized quantization noise, resulting in: b(z)={1, with probability 1+z2−1 with probability 1−z2

[0125] For backward propagation, we can also pass unchanged gradients through b. Gumbel noise

[0126] Gumbel softmax distributions have been used to learn categorical / discrete outputs. We take a similar approach by adding Gumbel noise to the sigmoid function, which we call the Gumbel sigmoid. Since the sigmoid is considered a special two-class case (e i and 0 in our case) can be viewed by softmax, we obtain the Gumbel sigmoid as: σ(ei)=exp((ei+gk) / τ)exp((ei+gk) / τ)+exp(gl / τ) where g is the sampled value from the Gumbel noise and T is the temperature that controls how closely the Gumbel sigmoid distribution approximates the binary distribution. From (7) we can further simplify this to: σ(ei)=11+exp(−(ei+gk−gl) / τ)=sigm((ei+gk−gl) / τ) where sigm is the sigmoid function. Since σ(e i ) is still in the range [0, 1], we use the same approach as previously introduced to find the value via z = 2(σ(e i ) - 0.5). We start with a high temperature and gradually cool it down to a low temperature, which is not zero. Lossless compression

[0127] Once we have generated the binary feature mapping, we continue to use Huffman coding to reduce the size of the binary representation. These coding schemes are asymptotically optimal, meaning that the compression ratio improves as more bits are grouped into a single symbol. It should be noted that other source coding methods, such as arithmetic coding, can also be used.

[0128] Although various embodiments have been described above, it is understood that they are presented only as examples and not as limitations. Therefore, the scope and extent of protection of a preferred embodiment are not to be limited by any of the embodiments described above, but only in accordance with the following claims and their equivalents.

Claims

[1] Method comprising: Identifying residual video data resulting from compression of original video data; encoding the residual video data to generate encoded residual video data; Binarizing the encoded residual video data to generate binarized residual video data; Compressing the binarized residual video data to generate compressed residual video data; and Transferring the compressed residual video data; wherein the residual video data is obtained by calculating a difference between decompressed original video data and the original video data, wherein the residual video data is encoded using a binary residual autoencoder. [2] The method of claim 1, wherein encoding the residual video data comprises compressing the residual video data. [3] A method according to any one of the preceding claims, wherein the encoded residual video data is binarized using a binary residual autoencoder. [4] The method of claim 3, wherein the binary residual autoencoder includes a binarizer that converts frame data of the encoded residual video data into a compressible binary bitstream. [5] The method of claim 4, wherein the binarizer is trained and implemented using tanh activation. [6] The method of claim 4 or 5, wherein the binarizer is trained and implemented using a HardTanh activation. [7] A method according to any one of claims 4 to 6, wherein the binarized residual video data is compressed using a binary residual autoencoder. [8] The method of claim 7, wherein the binary residue autoencoder includes a Huffman encoder that compresses a binary bit stream generated by the binarizer to reduce an amount of data to be transmitted. [9] A method according to any one of the preceding claims, wherein the encoding, binarization and compression are region-specific, such that: a binary residual autoencoder performs encoding, binarization, and compression of the residual video data; the binary residual autoencoder is trained in a domain-specific manner; and a limited set of possible results is used during training of the binary residual autoencoder. [10] A method according to any one of the preceding claims, wherein the compressed residual video data is transmitted with the original video data which has been compressed. [11] A method according to any one of the preceding claims, further comprising: Decompressing the original video data that has been compressed in a client device to generate decompressed original video data; Reconstructing the compressed residual video data in the client device to generate reconstructed residual video data; and Adding the reconstructed residual video data back to the decompressed original video data to generate output video data. [12] A system comprising a processor configured to perform a method according to any one of claims 1 to 11. [13] A computer-readable storage medium storing instructions which, when executed by a processor, cause the processor to perform steps of a method according to any one of claims 1 to 11.

Images