Bi-prediction for video coding by

Abstract

Dual lookahead passes for video encoding are disclosed, specifically apparatuses, systems, and techniques related to dual lookahead passes for video encoding are disclosed. In at least one embodiment, frames of a video are processed during a first lookahead pass to extract features and determine a fixed group of pictures (GOP) structure, which is then optimized based on the extracted features to generate an adaptive GOP structure that is processed during a second lookahead pass.

Description

Background Technology

[0001] To prepare video for compression representation, conventional techniques perform a single lookahead pass. This process analyzes the video to detect scene changes, gather frame complexity information, determine the Group of Pictures (GOP) structure, and generate a quantization parameter (QP) map for each codec unit (flexible-size macroblock). Based on this analysis, macroblock (MB) or codec unit (CU) tree algorithms generate spatial partitions for each frame, which are then used to encode the video into a compressed representation. MB and CU tree algorithms require a fixed GOP structure, meaning that the number and order of P-frames and B-frames (after I-frames) within each frame sequence (e.g., a GOP) are the same. This fixed GOP requirement can limit the quality of the encoded video, especially for videos with high inter-frame content variability and high intra-frame complexity. These issues and / or other problems related to existing techniques need to be addressed. Attached Figure Description

[0002] The present system and method for dual lookahead pass video coding are described in detail below with reference to the accompanying drawings, wherein:

[0003] Figure 1A This is a conceptual diagram illustrating the existing fixed GOP structure.

[0004] Figure 1B This is a conceptual diagram illustrating an adaptable GOP structure according to at least one embodiment.

[0005] Figure 2 This is a block diagram illustrating an example dual-lookahead video coding system according to at least one embodiment.

[0006] Figure 3 A flowchart illustrating a method according to at least one embodiment is shown.

[0007] Figure 4 A parallel processing unit (“PPU”) according to at least one embodiment is shown.

[0008] Figure 5A Use according to at least one embodiment is shown Figure 4 The processing system implemented by the PPU.

[0009] Figure 5B An exemplary system is shown, in which various architectures and / or functions of various prior embodiments can be implemented according to at least one embodiment.

[0010] Figure 5C An exemplary system for training machine learning models according to at least one embodiment is shown.

[0011] Figure 6 An exemplary streaming system according to at least one embodiment is shown. Detailed Implementation

[0012] Systems and methods related to double look-ahead pass for video coding are disclosed. Double look-ahead pass can improve the compression quality of videos with varying content complexity. Examples of complex content can include scenes (a sequence of one or more frames) depicting water, moving leafy plants, and / or highly complex textures. Double look-ahead pass is compatible with existing standards and spatial partitioning techniques, such as codec unit (CU) trees and macroblock (MB) tree algorithms.

[0013] To prepare video for compression, conventional techniques perform a single look-ahead pass to analyze the video, detect scene changes, gather frame complexity information, determine the Group of Pictures (GOP) structure, and generate a quantization parameter (QP) graph for each frame's codec unit (flexible-sized macroblock). Based on the analysis, either the MB tree or CU tree algorithm generates spatial partitions for each frame, which are then used to encode the video. The MB tree algorithm uses a fixed macroblock size (16x16 pixels), while the CU tree uses adapted-size blocks for encoding and decoding, where the size is determined by the content complexity. The MB tree allocates more bits in the QP graph to macroblocks with higher propagation (more frequent references in subsequent frames). The CU tree is a hierarchical quadtree structure where larger CUs are recursively subdivided into smaller CUs based on content complexity. The CU tree attempts to balance compression efficiency and quality. Overall, compared to the MB tree, the CU tree provides improved rate-distortion optimization. MB-tree and CU-tree algorithms require a fixed GOP structure, meaning that for each frame sequence (e.g., GOP), the number and order of prediction (P) frames and bidirectional (B) frames following a single intra-frame (I) frame are the same. A fixed GOP structure can limit the quality of the encoded video, especially for videos with high inter-frame content variability and high intra-frame complexity. Specifically, when inter-frame content complexity varies greatly, a fixed GOP structure can lead to a lower BD rate (…). The Delta rate loss is 10% or more.

[0014] Figure 1A This is a conceptual diagram illustrating the fixed GOP structure of the prior art. Fixed GOP 100 includes GOPs 101, 102, and 103, each consisting of an I (intra-coded picture) frame, followed by a first B (bidirectional prediction picture) frame, three P (predictive picture) frames, and a final B frame. For each of GOPs 101, 102, and 103, the number and order of I, B, and P frames are identical. Even when conventional software video coding techniques do use look-through portions to acquire some information, scene complexity information cannot be obtained, and coding is limited to a fixed quality.

[0015] Compared to conventional systems (such as those described above), dual look-ahead technology performs two look-ahead passes, enabling the use of adaptive GOP structures even when using MB-tree or CU-tree algorithms. The first look-ahead pass analyzes the video to extract (entropy) features, including scene changes and frame complexity information. Software video encoders utilize statistics from different reference frames using a single look-ahead pass to simultaneously generate dynamic GOP structures and QP graphs. Hardware video encoders cannot collect statistics from different reference frames, and a single look-ahead pass produces a fixed GOP structure. Dual look-ahead technology attempts to overcome the limitations of traditional software and hardware video encoders by generating a fixed GOP structure and statistics for each macroblock using the first look-ahead pass. The fixed GOP structure and statistics for each macroblock can be combined to provide frame-level statistics. A recombining operation can be performed after the first look-ahead pass to optimize the GOP structure before the second look-ahead pass. Recombining transforms the fixed GOP into an adaptive GOP structure, allowing one or more frames to use different GOP structures.

[0016] The second look-ahead pass involves repeating feature extraction based on the adapted GOP structure. Then, based on the extracted features and the adapted GOP structure, a QP map is generated at the end of the second look-ahead pass. Note that since the GOP structure becomes dynamic after the first pass, the QP map should only be computed after the second pass. Although the QP map can also be computed after the first pass, it cannot be used if the GOP structure has changed after the second pass. The second look-ahead pass also constructs spatial partitions (e.g., MB tree, CU tree, etc.) for each frame. The encoder uses the QP map, spatial partitions, and the adapted GOP structure to compress video frames.

[0017] Figure 1BThis is a conceptual diagram illustrating an adaptable GOP structure according to at least one embodiment. Adaptive GOP 110 includes GOPs 111, 112, and 113, each GOP consisting of a single I-frame followed by varying numbers and sequences of B-frames and P-frames. In adaptive GOP 110, the placement of I-frames is dynamically optimized compared to fixed GOP 100. Instead of placing I-frames at fixed intervals, the content of the frames is analyzed and each I-frame is inserted as needed to ensure that the trade-off between compression and video quality is improved or optimized. Specifically, GOP 111 consists of only a single B-frame followed by a single P-frame. Compared to GOP 112, which includes one P-frame followed by one B-frame and two more P-frames, GOP 111 may represent a less complex scene. GOP 113 can represent a more complex scene than GOPs 111 or 112. GOP 113 consists of two B-frames followed by one P-frame, two more B-frames, and a final P-frame.

[0018] When using an adaptive GOP structure, the encoder determines when to insert I-frames during processing based on factors such as scene changes, motion intensity, and / or camera switching. More specifically, for static scenes or scenes depicting little motion (e.g., static interview shots), the GOP can be extended (longer), meaning more frames are predicted using earlier frames (P-frames and B-frames), thereby reducing the number of bits required to encode frames and improving compression ratio. In one embodiment, the encoder generates shorter GOPs for scenes with high texture complexity, such as scenes depicting the movement of groups or groups of people (e.g., people, animals, leaves in the wind, etc.). In another embodiment, scenes with lower texture complexity (e.g., static backgrounds) can have longer GOPs.

[0019] For scenes with changing scenery or depicting high-motion scenes (e.g., sports or action sequences), the GOP can be shortened, and I-frames can be inserted more frequently to better capture new scenes or motion, thus reducing prediction errors. For scene transitions, where most content changes between frames, or where lighting, objects, or motion change significantly, the current GOP can be completed, and a new GOP can be started with an I-frame. Starting a new GOP resets the prediction for the new scene because relying on frames from the previous scene is inefficient and can introduce compression artifacts. Conversely, for smooth, low-motion scenes, the length of the current GOP can be lengthened, meaning fewer I-frames can be inserted, thus improving the compression ratio while maintaining visual quality.

[0020] Now, based on user expectations, we will elaborate on various optional architectures and features that can implement the aforementioned framework. It should be strongly noted that the following information is for illustrative purposes only and should not be construed as limiting in any way. Any of the following features may be optionally combined, while excluding or not excluding other features described.

[0021] Figure 2 This is a block diagram illustrating an example dual-look-through video coding system 200 according to at least one embodiment. The system includes a first look-through processing unit 210, an adaptation GOP unit 220, a second look-through processing unit 240, and an encoder unit 250, each of which can be implemented as software, hardware, or a combination thereof. A memory 230 stores information generated by 210, 220, and 240. It should be understood that this and other arrangements described herein are presented only as examples. Other arrangements and elements (e.g., machines, interfaces, functions, sequences, functional groupings, etc.) may be used in addition to or in place of the arrangements and elements shown, and some elements may be omitted entirely. Furthermore, many of the elements described herein are functional entities that can be implemented as discrete or distributed components or combined with other components, and can be implemented in any suitable combination and location. The various functions performed by the entities described herein can be performed by hardware, firmware, and / or software. For example, various functions can be implemented by a processor executing instructions stored in memory. Furthermore, those skilled in the art will understand that any system performing the operation of the dual-look-through video coding system 200 is within the scope and spirit of the embodiments of this disclosure.

[0022] The first look-ahead process performs a first look-ahead processing on the video frame through processing unit 210, generating a fixed GOP structure 215 and extracted features 245. In an embodiment, the extracted features include at least one of entropy change, detected scene change, and frame complexity information. A first frame resolution input and a second frame resolution input can be provided to enable processing of the video frame at a lower resolution, which is generally faster but may reduce compression quality. In an embodiment, the first and / or second frame resolution is reduced by processing only a subset of the video frames (e.g., discarding every nth frame). In one embodiment, the first frame resolution and / or the second frame resolution can be reduced compared to the original resolution of the video frame based on user-defined input and / or performance targets. User-defined inputs may correspond to compression ratio, storage capacity limits, or latency limits. Performance targets may specify compression ratio, bit rate, processing rate, bandwidth, etc.

[0023] The adaptive GOP unit 220 modifies the fixed GOP structure 215 based on the extracted features 245 to generate an adapted GOP structure 225. The second look-ahead process, through the processing unit 240, processes the video frames and the adapted GOP structure 225, repeatedly extracting features from the video frames based on the adapted GOP structure 225 and updating the extracted features 245. The second look-ahead process, through the processing unit 240, uses the adapted GOP structure 225 to construct an MB tree, CU tree, or other tree representation. In this embodiment, the resolution of the second frame is dynamically determined for one or more video frames based on the extracted features, making the computational complexity of the second look-ahead process dynamic as well.

[0024] Encoder unit 250 reads the tree representation, adapts the GOP structure 225 and QP graph from the extracted features 245, and encodes the video frames into a compressed format. In one embodiment, the parameters used by encoder unit 250 during the encoding process are adjusted based on the detected content information included in the extracted features.

[0025] In one embodiment, the operation of at least one of the first look-through processing unit 210, the adapting GOP unit 220, and the second look-through processing unit 240 is performed by a processor separate from the encoder unit 250. In another embodiment, the encoder unit 250 is remote relative to the first look-through processing unit 210, the adapting GOP unit 220, the second look-through processing unit 240, and the memory 230, and at least the adapting GOP structure 225 and the QP map are transmitted to the remote encoder unit 250 via a network.

[0026] In one embodiment, the dual-look-ahead video coding system 200 adjusts the bit rate and / or bandwidth consumption of the encoded video frames to perform real-time streaming. In another embodiment, the dual-look-ahead video coding system 200 is deployed in a surveillance system or live entertainment environment, where video frames are captured in real-time by a camera. In one embodiment, the dual-look-ahead video coding system 200 is deployed in a broadcast environment, and the video frames include live images and / or replay images.

[0027] Figure 3 A flowchart of method 300 according to at least one embodiment is shown. Each block of method 300 described herein includes a computational process that can be executed using any combination of hardware, firmware, and / or software. For example, various functions can be implemented by a processor executing instructions stored in memory. The method can also be embodied as computer-usable instructions stored on a computer storage medium. The method can be provided by a standalone application, service, or managed service (standalone or in combination with another managed service) or a plug-in to another product, to name a few. Furthermore, with Figure 2Method 300 is described using a dual-lookahead video coding system 200 as an example. However, this method may be performed by any other or alternative system or combination of systems, including but not limited to the systems described herein. Furthermore, those skilled in the art will understand that any system performing method 300 is within the scope and spirit of the embodiments of this disclosure.

[0028] At step 310, one or more frames of the video are processed during the first look-through to extract features and determine a fixed group of pictures (GOP) structure. In this embodiment, the extracted features include one or more detected scene changes and frame complexity information. At step 320, the fixed GOP structure is optimized based on the extracted features to generate an adapted GOP structure.

[0029] In step 330, one or more frames of the video are processed using an adapted GOP structure during the second look-through to update the extracted features based on the adapted GOP structure. In one embodiment, the resolution of one or more frames of the video is reduced before at least one of the first or second look-through. In one embodiment, the resolution is reduced to a lower resolution specified by a user-defined input. In one embodiment, the resolution is reduced based on a performance target. In one embodiment, at least one of the one or more frames is associated with a tree representation (e.g., a CU tree or MB tree) generated during the second look-through.

[0030] In step 340, one or more frames of the video are encoded based on the adapted GOP structure and the quantization parameter (QP) map contained in the extracted features. In one embodiment, encoding is performed using an encoder circuit system, and at least one of a first look-ahead pass, a second look-ahead pass, or optimization is performed by a processor separate from the encoder circuit system. In one embodiment, encoding is performed using a remote encoder circuit system, and at least the adapted GOP structure and QP map are transmitted to the remote encoder circuit system via a network.

[0031] In one embodiment, at least one of steps 310, 320, 330, or 340 is performed on a server or data center, and the encoded frame is streamed to a user device. In one embodiment, at least one of steps 310, 320, 330, or 340 is performed in a cloud computing environment. In one embodiment, at least one of steps 310, 320, 330, or 340 is performed for training, testing, or certifying a neural network used in a machine, robot, or autonomous vehicle. In one embodiment, at least one of steps 310, 320, 330, or 340 is performed on a virtual machine that includes a portion of a graphics processing unit. In one embodiment, at least one of steps 310, 320, 330, or 340 is implemented to include advanced error correction, fault tolerance, and self-healing capabilities.

[0032] Dual look-ahead coding enables the dynamic adjustment of the GOP length and structure based on extracted features such as scene changes, motion, and content complexity. By placing I-frames at optimal points and extending or shortening the GOP based on scene complexity, compression efficiency can be improved while maintaining high video quality. The result is more efficient bit usage, reduced artifacts in high-motion scenes, and better overall video quality for both real-time streaming and stored video content. Furthermore, dual look-ahead coding is compatible with existing standards and spatial partitioning techniques (CU tree and MB tree).

[0033] Parallel processing architecture

[0034] Figure 4 A parallel processing unit (“PPU”) 400 according to at least one embodiment is illustrated. PPU 400 can be used to implement a dual look-through video coding system 200. PPU 400 can be used to implement one or more of a first look-through processing unit 210, an adaptive GOP unit 220, a second look-through processing unit 230, and an encoder unit 250. In embodiments, a processor (e.g., PPU 400) can be configured to implement a neural network model. The neural network model can be implemented as software instructions executed by the processor, or, in other embodiments, the processor can include a matrix of hardware elements configured to process a set of inputs (e.g., electrical signals representing values) to generate a set of outputs that can represent activations of the neural network model. In other embodiments, the neural network model can be implemented as a combination of software instructions and processing executed by the hardware element matrix. Implementing a neural network model can include determining a set of parameters of the neural network model by, for example, supervised or unsupervised training, and, alternatively, using that set of parameters for inference to process new sets of inputs.

[0035] In one embodiment, PPU 400 is a multi-threaded processor implemented on one or more integrated circuit devices. PPU 400 is a latency-hiding architecture designed to process many threads in parallel. A thread (e.g., an execution thread) is an instantiation of an instruction set configured to be executed by PPU 400. In one embodiment, PPU 400 is a graphics processing unit (GPU) configured to implement a graphics rendering pipeline for processing three-dimensional (3D) graphics data to generate two-dimensional (2D) image data for display on a display device. In other embodiments, PPU 400 may be used to perform general-purpose computing. While an exemplary parallel processor is provided herein for illustrative purposes, it should be specifically noted that such a processor is illustrated for illustrative purposes only, and any processor may be employed to complement and / or replace this processor.

[0036] One or more PPU 400s can be configured to accelerate thousands of high-performance computing (HPC), data center, cloud computing, and machine learning applications. PPU 400s can be configured to accelerate numerous deep learning systems and applications used in autonomous vehicles, simulations, computational graphics such as ray or path tracing, deep learning, high-precision speech, image, and text recognition systems, intelligent video analytics, molecular simulations, drug discovery, disease diagnosis, weather forecasting, big data analytics, astronomy, molecular dynamics simulations, financial modeling, robotics, factory automation, real-time language translation, online search optimization, and personalized user recommendations, among others.

[0037] like Figure 4 As shown, PPU 400 includes an input / output (I / O) unit 405, a front-end unit 415, a scheduler unit 420, a job allocation unit 425, a hub 430, a crossbar (Xbar) 470, one or more general purpose processing clusters (GPCs) 450, and one or more memory partitioning units 480. PPU 400 can be connected to a host processor or other PPU 400 via one or more high-speed NVLink 410 interconnects. PPU 400 can be connected to a host processor or other peripheral devices via interconnect 402. PPU 400 can also be connected to local memory 404, which includes multiple memory devices. In one embodiment, local memory may include multiple dynamic random access memory (DRAM) devices. The DRAM devices may be configured as a high-bandwidth memory (HBM) subsystem, wherein multiple DRAM dies are stacked within each device.

[0038] The NVLink 410 interconnect enables the system to expand and include one or more PPUs 400 in conjunction with one or more CPUs, supporting cache coherency between the PPUs 400 and the CPU, as well as the CPU controller. Data and / or commands can be sent from or from the NVLink 410 to other units of the PPU 400 via hub 430, such as one or more copy engines, video encoders, video decoders, power management units, etc. (not explicitly shown). Figure 5B A more detailed description of the NVLink 410.

[0039] I / O unit 405 is configured to send and receive communications (e.g., commands, data, etc.) from a host processor (not shown) via interconnect 402. I / O unit 405 may communicate directly with the host processor via interconnect 402, or via one or more intermediate devices such as memory bridges. In one embodiment, I / O unit 405 may communicate with one or more other processors, such as one or more PPUs 400, via interconnect 402. In one embodiment, I / O unit 405 implements a Peripheral Component Interconnect High Speed ​​(PCIe) interface for communication via a PCIe bus, and interconnect 402 is a PCIe bus. In alternative embodiments, I / O unit 405 may implement other types of known interfaces for communication with external devices.

[0040] I / O unit 405 decodes data packets received via interconnect 402. In one embodiment, the data packets represent commands configured to cause PPU 400 to perform various operations. I / O unit 405 transmits the decoded commands to various other units of PPU 400 that these commands may specify. For example, some commands may be transmitted to front-end unit 415. Other commands may be transmitted to hub 430 or other units of PPU 400, such as one or more copy engines, video encoders, video decoders, power management units, etc. (not explicitly shown). In other words, I / O unit 405 is configured to route communication between and among the various logical units of PPU 400.

[0041] In one embodiment, a program executed by the host processor encodes a command stream in a buffer that provides a workload to the PPU 400 for processing. The workload may include instructions and data to be processed by those instructions. The buffer is an area of ​​memory accessible (e.g., read / write) by both the host processor and the PPU 400. For example, I / O unit 405 may be configured to access a buffer in system memory connected to interconnect 402 via a memory request transmitted through interconnect 402. In one embodiment, the host processor writes a command stream to the buffer and then transmits a pointer to the start of the command stream back to the PPU 400. Front-end unit 415 receives pointers to one or more command streams. Front-end unit 415 manages the one or more streams, reads commands from these streams, and forwards the commands to the respective units of the PPU 400.

[0042] Front-end unit 415 is coupled to scheduler unit 420, which configures various GPCs 450 to process tasks defined by the one or more streams. Scheduler unit 420 is configured to track status information related to the various tasks managed by scheduler unit 420. Status can indicate which GPC 450 a task is assigned to, whether the task is active or inactive, the priority associated with the task, etc. Scheduler unit 420 manages the execution of multiple tasks on the one or more GPCs 450.

[0043] Scheduler unit 420 is coupled to job allocation unit 425, which is configured to dispatch tasks for execution on GPC 450. Job allocation unit 425 can track several scheduled tasks received from scheduler unit 420. In one embodiment, job allocation unit 425 manages a pending task pool and an active task pool for each GPC 450. When GPC 450 completes the execution of a task, the task is evicted from the active task pool of GPC 450, and one of the other tasks from the pending task pool is selected and scheduled for execution on GPC 450. If an active task on GPC 450 is idle, for example, while waiting for data dependencies to be resolved, then the active task can be evicted from GPC 450 and returned to the pending task pool, while another task in the pending task pool is selected and scheduled for execution on GPC 450.

[0044] In one embodiment, the host processor executes a driver kernel that implements an application programming interface (API), enabling one or more applications executing on the host processor to be scheduled for operations to be performed on the PPU 400. In one embodiment, multiple computing applications are executed concurrently by the PPU 400, and the PPU 400 provides isolation, Quality of Service (QoS), and independent address spaces for the multiple computing applications. Applications may generate instructions (e.g., API calls) that cause the driver kernel to generate one or more tasks to be executed by the PPU 400. The driver kernel outputs the tasks to one or more streams being processed by the PPU 400. Each task may include one or more associated thread groups, referred to herein as warps. In one embodiment, a warp includes 32 associated threads that can execute in parallel. Cooperative threads may refer to multiple threads that include instructions for performing tasks and can exchange data via shared memory. These tasks may be assigned to one or more processing units within the GPC 450, and instructions are scheduled for execution by at least one warp.

[0045] The work allocation unit 425 communicates with one or more GPCs 450 via an XBar 470. The XBar 470 is an interconnect network that couples a plurality of units of the PPU 400 to other units of the PPU 400. For example, the XBar 470 can be configured to couple the work allocation unit 425 to a specific GPC 450. Although not explicitly shown, one or more other units of the PPU 400 may also be connected to the XBar 470 via a hub 430.

[0046] Tasks are managed by scheduler unit 420 and dispatched to GPC 450 by job allocation unit 425. GPC 450 is configured to process tasks and generate results. Results can be consumed by other tasks within GPC 450, routed to different GPC 450 via XBar 470, or stored in memory 404. Results can be written to memory 404 via memory partitioning unit 480, which implements a memory interface for reading data from and writing data to memory 404. Results can be transferred to another PPU 400 or CPU via NVLink 410. In one embodiment, PPU 400 includes U number of memory partitioning units 480, equal to the number of individual and distinct memory devices coupled to memory 404 of PPU 400. Each GPC 450 may include a memory management unit to provide virtual address to physical address translation, memory protection, and arbitration of memory requests. In one embodiment, the memory management unit provides one or more translation back buffers (TLBs) for performing virtual address to physical address translation in memory 404.

[0047] In one embodiment, memory partitioning unit 480 includes a raster operation (ROP) unit, a secondary (L2) cache, and a memory interface coupled to memory 404. The memory interface can implement 32-bit, 64-bit, 128-bit, or 1024-bit data buses for high-speed data transfer. PPU 400 can connect to up to Y memory devices, such as high-bandwidth memory stacks or graphics dual data rate, version 5, synchronous dynamic random access memory, or other types of persistent storage devices. In one embodiment, the memory interface implements an HBM2 memory interface, and Y equals half of U. In one embodiment, the HBM2 memory stack is located on the same physical package as PPU 400, providing significant power and area savings compared to conventional GDDR5 SDRAM systems. In one embodiment, each HBM2 stack includes four memory dies and Y equals 4, wherein each HBM2 stack includes two 129-bit channels per die, for a total of eight channels, and the data bus width is 1024 bits.

[0048] In one embodiment, memory 404 supports single error correction double detection (SECDED) error correction codes (ECC) to protect data. ECC provides a high level of reliability for computing applications sensitive to data corruption. Reliability is particularly important where the PPU 400 is handling very large datasets and / or in large-scale cluster computing environments where applications run for extended periods.

[0049] In one embodiment, PPU 400 implements a multi-level memory hierarchy. In one embodiment, memory partitioning unit 480 supports unified memory that provides a single, unified virtual address space for the CPU and PPU 400 memory, allowing data sharing between virtual memory systems. In one embodiment, the frequency with which PPU 400 accesses memory located on other processors is tracked to ensure that memory pages are moved to the physical memory of the PPU 400 that is accessing these pages more frequently. In one embodiment, NVLink 410 supports address translation services, allowing PPU 400 to directly access the CPU's page table and providing PPU 400 with full access to the CPU's memory.

[0050] In one embodiment, the replication engine transfers data between multiple PPUs 400 or between a PPU 400 and a CPU. The replication engine can generate page faults for addresses that are not mapped to a page table. The memory partitioning unit 480 can then repair the page faults, mapping these addresses to the page table, after which the replication engine can perform the transfer. In conventional systems, memory is fixed (e.g., non-pageable) for multiple replication engine operations across multiple processors, significantly reducing available storage. In the event of a hardware page fault, addresses can be passed to the replication engine without concern for whether memory pages reside, and the replication process is transparent.

[0051] Data from memory 404 or other system memory can be fetched by memory partitioning unit 480 and stored in an on-chip L2 cache shared among the various GPCs 450. As shown, each memory partitioning unit 480 includes a portion of the L2 cache associated with the corresponding memory 404. Low-level caches can then be implemented in various units within the GPC 450. For example, each processing unit within the GPC 450 can implement a Level 1 (L1) cache. The L1 cache is a private memory dedicated to a specific processing unit. The L2 cache is coupled to memory interface 470 and XBar 470, and data from the L2 cache can be fetched and stored in each of the L1 caches for processing.

[0052] In one embodiment, the processing unit within each GPC 450 implements a SIMD (Single Instruction Multiple Data) architecture, where each thread in a thread group (e.g., a thread bundle) is configured to process a different data set based on the same set of instructions. All threads in the thread group execute the same instructions. In another embodiment, the processing unit implements a SIMT (Single Instruction Multiple Thread) architecture, where each thread in the thread group is configured to process a different data set based on the same set of instructions, but individual threads in the thread group are allowed to diverge during execution. In one embodiment, maintaining a program counter, call stack, and execution state for each thread bundle allows for concurrency between thread bundles and serial execution within a thread bundle when threads diverge. In another embodiment, maintaining a program counter, call stack, and execution state for each individual thread allows for equal concurrency among all threads within and between thread bundles. When maintaining an execution state for each individual thread, threads executing the same instructions can aggregate and execute in parallel for maximum efficiency.

[0053] Cooperative groups are a programming model for organizing groups of communicating threads. They allow developers to express the granularity at which threads are communicating, enabling richer and more efficient expressions of parallel decomposition. The Cooperative Startup API supports synchronization between blocks of threads executing parallel algorithms. Conventional programming models provide a single, simple construct for synchronizing cooperative threads: a barrier across all threads in a thread block (e.g., the `syncthreads()` function). However, programmers often prefer to define thread groups smaller than thread blocks in the form of a collective group-wide function interface, and synchronize within the defined group to allow for greater performance, design flexibility, and software reuse.

[0054] Collaboration groups enable programmers to explicitly define thread groups (as small as a single thread) at the sub-block and multi-block granularity, and perform collective operations such as synchronization on threads within the collaboration group. This programming model supports clean composition across software boundaries, allowing libraries and utility functions to be safely synchronized within their local contexts without having to make assumptions about aggregation. The collaboration group primitive allows for the implementation of new cooperative parallelism patterns, including producer-consumer parallelism, opportunistic parallelism, and global synchronization across the entire mesh of thread blocks.

[0055] Each processing unit comprises a large number (e.g., 128, etc.) of different processing cores (e.g., functional units), which may be fully pipelined, single-precision, double-precision, and / or mixed-precision, and include floating-point arithmetic logic units and integer arithmetic logic units. In one embodiment, the floating-point arithmetic logic unit implements the IEEE 754-2008 standard for floating-point arithmetic. In one embodiment, the core comprises 64 single-precision (32-bit) floating-point cores, 64 integer cores, 32 double-precision (64-bit) floating-point cores, and 8 tensor cores.

[0056] Tensor cores are configured to perform matrix operations. Specifically, tensor cores are configured to perform deep learning matrix arithmetic, such as GEMM (matrix-matrix multiplication), for convolution operations during neural network training and inference. In one embodiment, each tensor core operates on a 4x4 matrix and performs matrix multiplication and accumulation operations, D = A × B + C, where A, B, C, and D are 4x4 matrices.

[0057] In one embodiment, matrix multiplication inputs A and B can be integer, fixed-point, or floating-point matrices, while accumulation matrices C and D can be integer, fixed-point, or floating-point matrices of equal or higher bit width. In one embodiment, the Tensor Core operates on 1-bit, 4-bit, or 8-bit integer input data using 32-bit integer accumulation. An 8-bit integer matrix multiplication requires 1024 operations and results in a full-precision product, which is then accumulated with other intermediate multiplications using 32-bit integer addition for an 8x8x16 matrix multiplication. In one embodiment, the Tensor Core operates on 16-bit floating-point input data using 32-bit floating-point accumulation. A 16-bit floating-point multiplication requires 64 operations and results in a full-precision product, which is then accumulated with other intermediate multiplications using 32-bit floating-point addition for a 4x4x4 matrix multiplication. In practice, the Tensor Core is used to perform operations on much larger two-dimensional or higher-dimensional matrices composed of these smaller elements. APIs such as the CUDA 9 C++ API expose specialized matrix loading, matrix multiplication and accumulation, and matrix storage operations for efficient use with Tensor Core from CUDA-C++ programs. At the CUDA level, the thread bundle-level interface takes a 16x16 matrix spanning all 32 threads of the thread bundle.

[0058] Each processing unit may also include M Special Function Units (SFUs) that perform special functions (e.g., attribute evaluation, reciprocal square root, etc.). In one embodiment, an SFU may include a tree traversal unit configured to traverse a hierarchical tree data structure. In one embodiment, an SFU may include a texture unit configured to perform texture map filtering operations. In one embodiment, a texture unit is configured to load texture maps (e.g., a 2D texture array) from memory 404 and sample these texture maps to produce sampled texture values ​​for use by a shader program executed by the processing unit. In one embodiment, the texture maps are stored in shared memory that may include or contain an L1 cache. The texture units use mip maps (e.g., texture maps with varying levels of detail) to implement texture operations such as filtering. In one embodiment, each processing unit includes two texture units.

[0059] Each processing unit also includes N Load Memory Units (LSUs) that implement load and store operations between shared memory and the register file. Each processing unit includes an interconnect network that connects each of the cores to the register file and connects the LSUs to the register file and the shared memory. In one embodiment, the interconnect network is a cross switch that can be configured to connect any of the cores to any register in the register file and connect the LSUs to memory locations in the register file and the shared memory.

[0060] Shared memory is an on-chip memory array that allows data storage and communication between processing units and between threads within a processing unit. In one embodiment, the shared memory includes 128KB of storage capacity and is located on the path from each of the processing units to memory partition unit 480. The shared memory can be used for caching reads and writes. One or more of the shared memory, L1 cache, L2 cache, and memory 404 serve as a backup cache.

[0061] Combining data caching and shared memory functionality into a single memory block provides optimal overall performance for both types of memory access. This capacity can be used as a cache by programs that do not use shared memory. For example, if shared memory is configured to use half its capacity, then texture and load / store operations can use the remaining capacity. Integration into shared memory allows it to function as a high-throughput conduit for streaming data, while providing high-bandwidth and low-latency access to frequently reused data.

[0062] When configured for general-purpose parallel computing, a simpler configuration can be used compared to graphics computing. Specifically, bypassing fixed-function graphics processing units (GPUs) creates a much simpler programming model. In this general-purpose parallel computing configuration, the work allocation unit 425 directly dispatches and assigns thread blocks to processing units within the GPC 450. Threads execute the same program using a unique thread ID during computation to ensure that each thread uses the executor program and the processing unit performing the computation, the shared memory for communication between threads, and the LSU (Local Subsystem for Memory) for reading and writing global memory via the shared memory and memory partitioning unit 480 to generate unique results. When configured for general-purpose parallel computing, processing units can also write commands, which the scheduler unit 420 can use to start new work on the processing unit.

[0063] Each of the PPUs 400 may include one or more processing cores and / or components thereof, such as a Tensor Core (TC), a Tensor Processing Unit (TPU), a Pixel Vision Core (PVC), a Ray Tracing (RT) Core, a Vision Processing Unit (VPU), a Graphics Processing Cluster (GPC), a Texture Processing Cluster (TPC), a Streaming Multiprocessor (SM), a Tree Traversal Unit (TTU), an Artificial Intelligence Accelerator (AIA), a Deep Learning Accelerator (DLA), an Arithmetic Logic Unit (ALU), an Application-Specific Integrated Circuit (ASIC), a Floating-Point Unit (FPU), Input / Output (I / O) Elements, Peripheral Component Interconnect (PCI) or Peripheral Component Interconnect High Speed ​​(PCIe) Elements and / or the like, and / or be configured to perform its functions.

[0064] The PPU 400 may be included in desktop computers, laptop computers, tablet computers, servers, supercomputers, smartphones (e.g., wireless, handheld devices), personal digital assistants (PDAs), digital cameras, vehicles, head-mounted displays, handheld electronic devices, and the like. In one embodiment, the PPU 400 is implemented on a single semiconductor substrate. In another embodiment, the PPU 400 is included in a system-on-a-chip (SoC) along with one or more other devices such as an additional PPU 400, memory 404, a reduced instruction set computer (RISC) CPU, a memory management unit (MMU), a digital-to-analog converter (DAC), and the like.

[0065] In one embodiment, PPU 400 may be included on a graphics card that includes one or more memory devices. The graphics card may be configured to interface with a PCIe slot on a desktop computer's motherboard. In yet another embodiment, PPU 400 may be an integrated graphics processing unit (iGPU) or a parallel processor included in the chipset of the motherboard. In yet another embodiment, PPU 400 may be implemented in reconfigurable hardware. In yet another embodiment, a portion of PPU 400 may be implemented in reconfigurable hardware.

[0066] Exemplary computing system

[0067] As developers expose and leverage more parallelism in applications such as artificial intelligence computing, systems with multiple GPUs and CPUs are being used across various industries. High-performance GPU-accelerated systems with tens to thousands of compute nodes are being deployed in data centers, research facilities, and supercomputers to solve increasingly complex problems. With the increasing number of processing devices within high-performance systems, communication and data transmission mechanisms need to be scaled to support the increased bandwidth.

[0068] Figure 5A The use according to at least one embodiment is shown. Figure 4 The PPU 400 implements a processing system 500. The exemplary system 500 can be configured to implement... Figure 3 The method 300 shown is described. The processing system 500 includes a CPU 530, a switch 510, multiple PPUs 400, and individual memories 404. An NVLink 410 provides a high-speed communication link between each of the PPUs 400. Although... Figure 5B The diagram illustrates a specific number of NVLink 410 and interconnect 402 connections, but the number of connections to each PPU 400 and CPU 530 can vary. Switch 510 forms an interface between interconnect 402 and CPU 530. PPU 400, memory 404, and NVLink 410 can reside on a single semiconductor platform to form a parallel processing module 525. In one embodiment, switch 510 supports two or more protocols to interface between various connections and / or links.

[0069] In another embodiment (not shown), NVLink 410 provides one or more high-speed communication links between each PPU 400 and CPU 530, and switch 510 forms an interface between interconnect 402 and each PPU 400. PPU 400, memory 404, and interconnect 402 may reside on a single semiconductor platform to form a parallel processing module 525. In yet another embodiment (not shown), interconnect 402 provides one or more communication links between each PPU 400 and CPU 530, and switch 510 uses NVLink 410 to form an interface between each PPU 400 to provide one or more high-speed communication links between PPUs 400. In another embodiment (not shown), NVLink 410 provides one or more high-speed communication links between PPU 400 and CPU 530 via switch 510. In yet another embodiment (not shown), interconnect 402 directly provides one or more communication links between each PPU 400. One or more of the NVLink 410 high-speed communication links can be implemented as physical NVLink interconnects or on-chip or die-on interconnects using the same protocol as the NVLink 410.

[0070] In the context of this specification, a single semiconductor platform can refer to a single semiconductor-based integrated circuit fabricated on a bare die or chip. It should be noted that the term "single semiconductor platform" can also refer to a multi-chip module with increased connectivity, simulating on-chip operation and representing a significant improvement over conventional bus implementations. Of course, the various circuits or devices can also be located individually within the semiconductor platform or in various combinations thereof, as desired by the user. Alternatively, the parallel processing module 525 can be implemented as a circuit board substrate, and each PPU 400 and / or memory 404 can be a packaged device. In one embodiment, the CPU 530, switch 510, and parallel processing module 525 reside on a single semiconductor platform.

[0071] In one embodiment, the signaling rate of each NVLink 410 is 20-25 gigabits per second, and each PPU400 includes six NVLink 410 interfaces (e.g., Figure 5A As shown, each PPU 400 includes five NVLink 410 interfaces. Each NVLink 410 provides a data transfer rate of 25 gigabits per second in each direction, and the six links provide 400 gigabits per second. NVLink 410s can be used as follows: Figure 5A It is used exclusively for PPU-to-PPU communication, or for a combination of PPU-to-PPU and PPU-to-CPU when the CPU 530 also includes one or more NVLink 410 interfaces.

[0072] In one embodiment, NVLink 410 allows direct load / store / atomic access from CPU 530 to memory 404 of each PPU 400. In one embodiment, NVLink 410 supports coherent operation, allowing data read from memory 404 to be stored in the cache hierarchy of CPU 530, reducing cache access latency of CPU 530. In one embodiment, NVLink 410 includes support for Address Translation Service (ATS), allowing PPU 400 to directly access page tables within CPU 530. One or more of NVLink 410s can also be configured to operate in low-power mode.

[0073] Figure 5B An exemplary system 565, according to at least one embodiment, is illustrated, in which various architectures and / or functions of various prior embodiments can be implemented. The exemplary system 565 can be configured to implement... Figure 3 Method 300 is shown. As shown, a system 565 is provided that includes at least one central processing unit 530 connected to a communication bus 575. The communication bus 575 may directly or indirectly couple one or more of the following devices: main memory 540, network interface 535, CPU 530, display device 545, input device 560, switch 510, and parallel processing system 525. The communication bus 575 may be implemented using any suitable protocol and may represent one or more links or buses, such as address bus, data bus, control bus, or a combination thereof. The communication bus 575 may include one or more bus or link types, such as Industry Standard Architecture (ISA) bus, Extended Industry Standard Architecture (EISA) bus, Video Electronics Standards Association (VESA) bus, Peripheral Component Interconnect (PCI) bus, Peripheral Component Interconnect High Speed ​​(PCIe) bus, HyperTransport, and / or another type of bus or link. In some embodiments, there are direct connections between components. As an example, CPU 530 may be directly connected to main memory 540. Furthermore, CPU 530 may be directly connected to parallel processing system 525. In cases where there is a direct or point-to-point connection between components, the communication bus 575 may include a PCIe link that implements the connection. In these examples, the PCI bus need not be included in the system 565.

[0074] Although using lines Figure 5BThe different blocks are shown connected via a communication bus 575, but this is not intended to be limiting and is merely for clarity. For example, in some embodiments, a presentation component such as a display device 545 can be considered an I / O component, such as an input device 560 (e.g., if the display is a touchscreen). As another example, the CPU 530 and / or the parallel processing system 525 may include memory (e.g., main memory 540 may represent storage devices other than the parallel processing system 525, the CPU 530, and / or other components). In other words, Figure 5B The term "computing device" is merely illustrative. No distinction is made between categories such as "workstation," "server," "laptop," "desktop," "tablet," "client device," "mobile device," "handheld device," "game console," "electronic control unit (ECU)," "virtual reality system," and / or other device or system types, as all of these are expected to fall under [the relevant category]. Figure 5B Within the scope of computing devices.

[0075] System 565 also includes main memory 540. Control logic (software) and data are stored in main memory 540, which can take the form of a variety of computer-readable media. Computer-readable media can be any available medium that can be accessed by system 565. Computer-readable media can include volatile and non-volatile media, as well as removable and non-removable media. For example and without limitation, computer-readable media can include computer storage media and communication media.

[0076] Computer storage media may include volatile and non-volatile media and / or removable and non-removable media implemented in any method or technology for storing information such as computer-readable instructions, data structures, program modules, and / or other data types. For example, main memory 540 may store computer-readable instructions such as an operating system (e.g., representing programs and / or program elements). Computer storage media may include, but is not limited to, RAM, ROM, EEPROM, flash memory or other memory technologies, CD-ROM, digital versatile disc (DVD) or other optical disc storage devices, magnetic cartridges, magnetic tape, disk storage devices or other magnetic storage devices, or any other medium that may be used to store desired information and that can be accessed by system 565. When used herein, computer storage media does not include the signal itself.

[0077] Computer storage media may contain computer-readable instructions, data structures, program modules, or other data types in modulated data signals such as carrier waves or other transmission mechanisms, and may include any information transport medium. The term "modulated data signal" may refer to a signal whose characteristics are set or altered in a manner that encodes information into that signal. For example and without limitation, computer storage media may include wired media such as wired networks or direct wired connections, and wireless media such as sound, RF, infrared, and other wireless media. Any combination of the above should also be included within the scope of computer-readable media.

[0078] When executed, the computer program enables system 565 to perform various functions. CPU 530 may be configured to execute at least some of the computer-readable instructions to control one or more components of system 565 to perform one or more of the methods and / or processes described herein. Each of CPUs 530 may include one or more cores (e.g., one, two, four, eight, twenty-eight, seventy-two, etc.) capable of processing numerous software threads simultaneously. Depending on the type of system 565 implemented, CPU 530 may include any type of processor and may include different types of processors (e.g., processors with fewer cores for mobile devices and processors with more cores for servers). For example, depending on the type of system 565, the processor may be an advanced RISC machine (ARM) processor implemented using Reduced Instruction Set Computing (RISC) or an x86 processor implemented using Complex Instruction Set Computing (CISC). In addition to one or more microprocessors or supplementary coprocessors such as math coprocessors, system 565 may include one or more CPUs 530.

[0079] In addition to or alternatively to CPU 530, parallel processing module 525 may be configured to execute at least some of the computer-readable instructions to control one or more components of system 565 to perform one or more of the methods and / or processes described herein. Parallel processing module 525 may be used by system 565 to render graphics (e.g., 3D graphics) or perform general-purpose computing. For example, parallel processing module 525 may be used for general-purpose computing on a GPU (GPGPU). In embodiments, CPU 530 and / or parallel processing module 525 may execute any combination of the methods, processes, and / or portions thereof, discretely or jointly.

[0080] System 565 also includes input device 560, parallel processing system 525, and display device 545. Display device 545 may include a display (e.g., a monitor, touchscreen, television screen, head-up display (HUD), other display types, or combinations thereof), speakers, and / or other presentation components. Display device 545 may receive data from other components (e.g., parallel processing system 525, CPU 530, etc.) and output that data (e.g., images, video, sound, etc.).

[0081] Network interface 535 enables system 565 to be logically coupled to other devices, including input device 560, display device 545, and / or other components, some of which may be embedded (e.g., integrated into) system 565. Illustrative input device 560 includes microphone, mouse, keyboard, joystick, gamepad, game controller, satellite dish, scanner, printer, wireless device, etc. Input device 560 can provide a natural user interface (NUI) that processes user-generated air gestures, voice, or other physiological input. In some instances, input can be transmitted to appropriate network elements for further processing. NUI can implement voice recognition, stylus recognition, facial recognition, biometric recognition, on-screen and adjacent-screen gesture recognition, air gestures, head-eye tracking, and touch recognition associated with the display of system 565 (described in more detail below). System 565 may include depth cameras for gesture detection and recognition, such as stereo camera systems, infrared camera systems, RGB camera systems, touchscreen technology, and combinations thereof. In addition, system 565 may include an accelerometer or gyroscope that allows motion detection (e.g., as part of an inertial measurement unit (IMU)). In some examples, the output of the accelerometer or gyroscope may be used by system 565 to render immersive augmented reality or virtual reality.

[0082] Furthermore, system 565 can be coupled to a network (e.g., a telecommunications network, a local area network (LAN), a wireless network, a wide area network (WAN) such as the Internet, a peer-to-peer network, a cable network, etc.) via network interface 535 for communication purposes. System 565 can be included in a distributed network and / or cloud computing environment.

[0083] Network interface 535 may include one or more receivers, transmitters, and / or transceivers, enabling system 565 to communicate with other computing devices via electronic communication networks, including wired and / or wireless communications. Network interface 535 may be implemented as a network interface controller (NIC) including one or more data processing units (DPUs) to perform operations such as (e.g., but not limited to) packet parsing and accelerating network processing and communication. Network interface 535 may include components and functions to enable communication over any of several different networks, such as wireless networks (e.g., Wi-Fi, Z-Wave, Bluetooth, Bluetooth LE, ZigBee, etc.), wired networks (e.g., communication over Ethernet or InfiniBand), low-power wide area networks (e.g., LoRaWAN, SigFox, etc.), and / or the Internet.

[0084] System 565 may also include an auxiliary storage device (not shown). The auxiliary storage device includes, for example, a hard disk drive and / or a removable storage drive representing a floppy disk drive, magnetic tape drive, compact disc drive, digital versatile disc (DVD) drive, recording device, Universal Serial Bus (USB) flash memory. The removable storage drive reads from and / or writes to the removable storage unit in a well-known manner. System 565 may also include a hard-wired power supply, a battery power supply, or a combination thereof (not shown). This power supply can supply power to System 565 to enable the components of System 565 to operate.

[0085] Each of the aforementioned modules and / or devices may even reside on a single semiconductor platform to form system 565. Alternatively, various different modules may be placed individually or located in various combinations of semiconductor platforms as desired by the user. Although various different embodiments have been described above, it should be understood that they are given by way of example only and without limitation. Therefore, the breadth and scope of preferred embodiments should not be limited to any of the exemplary embodiments described above, but should be defined only by the following claims and their equivalents.

[0086] Example network environment

[0087] A network environment suitable for implementing embodiments of this disclosure may include one or more client devices, servers, network-attached storage devices (NAS), other back-end devices, and / or other device types. Client devices, servers, and / or other device types (e.g., each device) may be... Figure 5A Processing system 500 and / or Figure 5B Implemented on one or more instances of the exemplary system 565, for example, each device may include similar components, features and / or functions of the processing system 500 and / or the exemplary system 565.

[0088] Components of a network environment can communicate with each other via a network, which can be wired, wireless, or a combination of both. A network can include multiple networks or a network of networks. For example, a network can include one or more wide area networks (WANs), one or more local area networks (LANs), one or more public networks—such as the Internet, and / or the Public Switched Telephone Network (PSTN), and / or one or more private networks. In cases where the network includes a wireless telecommunications network, components such as base stations, communication towers, or even access points (along with other components) can provide wireless connectivity.

[0089] A compatible network environment may include one or more peer-to-peer network environments—in which case the server may not be included in the network environment—and one or more client-server network environments—in which case one or more servers may be included in the network environment. In a peer-to-peer network environment, the functionality described herein regarding the server can be implemented on any number of client devices.

[0090] In at least one embodiment, the network environment may include one or more cloud-based network environments, distributed computing environments, combinations thereof, etc. A cloud-based network environment may include a framework layer, a job scheduler, a resource manager, and a distributed file system implemented on one or more servers, which may include one or more core network servers and / or edge servers. The framework layer may include a framework of one or more applications supporting a software layer and / or an application layer. The software or application may include web-based service software or applications, respectively. In embodiments, one or more client devices may use the web-based service software or application (e.g., by accessing the service software and / or application via one or more application programming interfaces (APIs)). The framework layer may be, but is not limited to, a free and open-source software web application framework type that can be used for large-scale data processing (e.g., "big data").

[0091] A cloud-based network environment can provide cloud computing and / or cloud storage to implement the computing and / or data storage functions (or one or more portions thereof) described herein. Any of these different functions can be distributed across multiple locations from a central or core server (e.g., a central or core server in one or more data centers, which may be distributed across states, regions, countries, globally, etc.). If the connection to a user (e.g., a client device) is relatively close to an edge server, then the core server can assign at least a portion of the functions to the edge server. A cloud-based network environment can be private (e.g., limited to a single organization), public (e.g., available to many organizations), and / or a combination thereof (e.g., a hybrid cloud environment).

[0092] Client devices may include Figure 5A Example processing system 500 and / or Figure 5B At least some of the components, features, and functions of the exemplary system 565. For example and without limitation, the client device may be implemented as a personal computer (PC), laptop computer, mobile device, smartphone, tablet computer, smartwatch, wearable computer, personal digital assistant (PDA), MP3 player, virtual reality headset, global positioning system (GPS) or device, video player, video camera, surveillance equipment or system, vehicle, ship, spacecraft, virtual machine, drone, robot, handheld communication device, hospital equipment, gaming equipment or system, entertainment system, vehicle computer system, embedded system controller, remote control, appliance, consumer electronics device, workstation, edge device, any combination of these defined devices, or any other suitable device.

[0093] Machine Learning

[0094] Deep neural networks (DNNs) developed on processors such as the PPU 400 have been used in a wide variety of use cases, from self-driving cars to faster drug development, from automatic image captioning in online image databases to intelligent real-time language translation in video chat applications. Deep learning is a technique that models the neural learning process of the human brain, which continuously learns, becomes smarter, and delivers more accurate results faster over time. Just as a child is initially taught by adults to correctly identify and classify various shapes, eventually becoming able to identify shapes without any guidance, a deep learning or neural learning system needs to be trained in object recognition and classification so that it becomes smarter and more efficient at identifying basic objects, occluded objects, and so on, while also attaching context to objects.

[0095] At its simplest level, neurons in the human brain receive various inputs, assigning a level of importance to each of these inputs, and the output is passed to other neurons to make a response. Artificial neurons, or perceptrons, are the most basic model of neural networks. In one example, a perceptron can receive one or more inputs representing various features of objects that the perceptron is being trained to recognize and classify, and each of these features is assigned a weight based on its importance in defining the shape of the object.

[0096] Deep neural network (DNN) models consist of multiple layers of numerous connected nodes (e.g., perceptrons, Boltzmann machines, radial basis functions, convolutional layers, etc.) that can be trained on massive amounts of input data to solve complex problems quickly and with high accuracy. In one example, the first layer of a DNN model breaks down an input image of a car into different segments and searches for basic patterns such as lines and angles. The second layer assembles these lines to find higher-level patterns such as wheels, windshields, and mirrors. The next layer identifies the type of vehicle, and the final few layers generate labels for the input image that identify the model of a specific car brand.

[0097] Once trained, a DNN can be deployed and used to identify and classify objects or patterns in a process called inference. Examples of inference (the process by which a DNN extracts useful information from a given input) include identifying handwritten digits on a check deposited into an ATM, identifying images of friends in a photograph, delivering movie recommendations to over 50 million users, identifying and classifying different types of cars, pedestrians, and road hazards in self-driving cars, or translating human language in real time.

[0098] During training, data flows through the DNN in the forward propagation phase until a prediction is produced indicating the label corresponding to the input. If the neural network does not correctly label the input, the error between the correct label and the predicted label is analyzed, and the weights are adjusted for each feature during the backpropagation phase until the DNN correctly labels the input as well as other inputs in the training dataset. Training complex neural networks requires significant parallel computing power, including floating-point multiplication and addition supported by a PPU400. Inference is less computationally intensive than training and is a latency-sensitive process where the trained neural network is applied to new inputs it has not seen before for tasks such as image classification, sentiment detection, label recommendation, language recognition and translation, and typically infers new information.

[0099] Neural networks rely heavily on matrix operations, and complex, multi-layered networks require massive amounts of floating-point performance and bandwidth for both efficiency and speed. Leveraging thousands of processing cores optimized for matrix operations and delivering tens to hundreds of TFLOPS of performance, the PPU 400 is a computing platform capable of providing the performance required for deep neural network-based artificial intelligence and machine learning applications.

[0100] Furthermore, images generated using one or more of the techniques disclosed herein can be used to train, test, or certify DNNs for recognizing real-world objects and environments. Such images can include driveways, factories, buildings, urban environments, rural environments, humans, animals, and any other physical objects or scenes of real-world environments. Such images can be used to train, test, or certify DNNs used in machines or robots to manipulate, process, or modify real-world physical objects. Additionally, such images can be used to train, test, or certify DNNs used in autonomous vehicles to navigate and move vehicles in the real world. Furthermore, images generated using one or more of the techniques disclosed herein can be used to communicate information to users of such machines, robots, and vehicles.

[0101] Figure 5C An example system 555, according to at least one embodiment, is illustrated for use in training machine learning models. As will be discussed, various components can be provided by a single computing system or various combinations of computing devices and resources, which may be under the control of a single entity or more entities. Furthermore, aspects may be triggered, initiated, or requested by different entities. In at least one embodiment, the training of the neural network may be guided by a vendor associated with vendor environment 506, while in at least one embodiment, training may be requested by a customer or other user capable of accessing the vendor environment via client device 502 or other such resources. In at least one embodiment, training data (or data to be analyzed by the trained neural network) may be provided by a vendor, user, or third-party content provider 524. In at least one embodiment, client device 502 may be, for example, a vehicle or object to be navigated on behalf of a user, who can submit requests and / or receive instructions that aid in device navigation.

[0102] In at least one embodiment, a request can be submitted via at least one network 504 for receipt by a vendor environment 506. In at least one embodiment, the client device can be any suitable electronic and / or computing device that enables a user to generate and send such requests, such as, but not limited to, desktop computers, laptop computers, computer servers, smartphones, tablets, game consoles (portable or otherwise), computer processors, computing logic, and set-top boxes. One or more networks 504 can include any suitable network for transmitting requests or other such data, such as the Internet, intranet, Ethernet, cellular network, local area network (LAN), wide area network (WAN), personal area network (PAN), self-organizing network providing direct wireless connectivity between peers, etc.

[0103] In at least one embodiment, a request may be received at interface layer 508, which in this example may forward data to training and inference manager 532. Training and inference manager 532 may be a system or service including hardware and software for managing services and requests corresponding to data or content. In at least one embodiment, training and inference manager 532 may receive a request to train a neural network and may provide data for the request to training module 512. In at least one embodiment, if the request is not specified, training module 512 may select an appropriate model or neural network to use and may train the model using the associated training data. In at least one embodiment, training data may be a batch of data stored in training data repository 514, received from client device 502, or obtained from third-party vendor 524. In at least one embodiment, training module 512 may be responsible for training the data. The neural network may be any suitable network, such as a recurrent neural network (RNN) or convolutional neural network (CNN). Once the neural network is trained and successfully evaluated, the trained neural network may be stored in, for example, model repository 516, which may store different models or networks for users, applications, or services, etc. In at least one embodiment, there may be multiple models for a single application or entity, which can be utilized based on multiple different factors.

[0104] In at least one embodiment, at a subsequent point in time, a request for content (e.g., path determination) or data that is at least partially determined or influenced by a trained neural network can be received from client device 502 (or another such device). This request may include, for example, input data to be processed using the neural network to obtain one or more inference or other output values, classifications, or predictions. Alternatively, in at least one embodiment, the input data may be received by interface layer 508 and directed to inference module 518, although different systems or services may also be used. In at least one embodiment, if not already locally stored in inference module 518, inference module 518 may obtain a suitably trained network, such as a trained deep neural network (DNN) as discussed herein, from model repository 516. Inference module 518 may provide data as input to the trained network, which may then generate one or more inferences as outputs. This may, for example, include the classification of instances of input data. In at least one embodiment, the inference may then be transmitted to client device 502 for display to a user or for other communication with the user. In at least one embodiment, user context data may also be stored in a user context data repository 522, which may include data about the user that can be used as network input to generate inference or determine data returned to the user after obtaining an instance. In at least one embodiment, relevant data, including at least some of the input or inference data, may also be stored in a local database 534 for processing future requests. In at least one embodiment, the user may use account information or other information to access resources or functions of the vendor environment. In at least one embodiment, user data may also be collected and used to further train the model, if permitted and available, to provide more accurate inference for future requests. In at least one embodiment, requests to a machine learning application 526 executed on a client device 502 may be received via a user interface, and the results may be displayed via the same interface. The client device may include resources such as a processor 528 and a memory 562 for generating requests and processing results or responses, and at least one data storage element 552 for storing data for the machine learning application 526.

[0105] In at least one embodiment, processor 528 (or the processor of training module 512 or inference module 518) will be a central processing unit (CPU). However, as mentioned above, resources in such an environment can utilize GPUs to process data for at least some types of requests. GPUs, such as the PPU 400, have thousands of cores and are designed to handle large amounts of parallel workloads, thus becoming popular in deep learning for training neural networks and generating predictions. While using GPUs for offline building allows for faster training of larger, more complex models, offline prediction generation means that request-time input features cannot be used, or predictions must be generated for all features and stored in a lookup table for real-time service requests. If the deep learning framework supports CPU mode and the model is small and simple enough that the feedforward can be performed on the CPU with reasonable latency, then a service on a CPU instance can host the model. In this case, training can be done offline on the GPU and inference can be performed in real-time on the CPU. If the CPU approach is not feasible, the service can run on a GPU instance. However, due to the different performance and cost characteristics of GPUs compared to CPUs, running a service that offloads runtime algorithms to the GPU may require it to be designed differently from a CPU-based service.

[0106] In at least one embodiment, video data can be provided from client device 502 for enhancement in vendor environment 506. In at least one embodiment, the video data can be processed for enhancement on client device 502. In at least one embodiment, the video data can be streamed from third-party content provider 524 and enhanced by third-party content provider 524, vendor environment 506, or client device 502. In at least one embodiment, video data can be provided from client device 502 for use as training data in vendor environment 506.

[0107] In at least one embodiment, supervised and / or unsupervised training may be performed by client device 502 and / or vendor environment 506. In at least one embodiment, a set of training data 514 (e.g., classified or labeled data) is provided as input for use as training data. In at least one embodiment, the training data may include instances of at least one type of object for which the neural network is to be trained, and information identifying that object type. In at least one embodiment, the training data may include a set of images, each image including a representation of an object of a type, wherein each image also includes, or is associated with, tags, metadata, classification, or other information identifying or identifying the type of object represented in the corresponding image. Various other types of data may also be used as training data, which may include text data, audio data, video data, and so on. In at least one embodiment, training data 514 is provided as training input to training module 512. In at least one embodiment, training module 512 may be a system or service including hardware and software, such as one or more computing devices executing a training application for training a neural network (or other model or algorithm, etc.). In at least one embodiment, training module 512 receives instructions or requests indicating the type of model to be used for training. In at least one embodiment, the model can be any suitable statistical model, network, or algorithm useful for such a purpose, which may include artificial neural networks, deep learning algorithms, learning classifiers, Bayesian networks, etc. In at least one embodiment, training module 512 may select an initial model or other untrained models from an appropriate repository and train the model using training data 514 to generate a trained model (e.g., a trained deep neural network) that can be used to classify similar types of data or generate other such inference. In at least one embodiment where training data is not used, an initial model can still be selected to train on the input data of each training module 512.

[0108] In at least one embodiment, the model can be trained in several different ways, which may depend in part on the type of model chosen. In at least one embodiment, a training dataset can be provided to a machine learning algorithm, wherein the model is a model artifact created through a training process. In at least one embodiment, each instance of the training data contains the correct answer (e.g., classification) that may be referred to as the target or target attribute. In at least one embodiment, the learning algorithm finds patterns in the training data that map input data attributes to the target—the answer to be predicted—and the machine learning model is the output that captures these patterns. In at least one embodiment, the machine learning model can then be used to obtain predictions for new data without a specified target.

[0109] In at least one embodiment, the training and inference manager 532 may select from a set of machine learning models, including binary classification, multi-class classification, generative, and regression models. In at least one embodiment, the type of model to be used may depend at least in part on the type of target to be predicted.

[0110] Graphics processing pipeline

[0111] In this embodiment, the PPU 400 includes a graphics processing unit (GPU). The PPU 400 is configured to receive commands specifying a shader program for processing graphics data. Graphics data can be defined as a set of primitives, such as points, lines, triangles, quadrilaterals, triangle strips, etc. Typically, primitives include data specifying multiple vertices of the primitive (e.g., in model space coordinates) and attributes associated with each vertex of the primitive. The PPU 400 can be configured to process the graphics primitives to generate framebuffers (e.g., pixel data for each pixel of a display).

[0112] The application writes model data of the scene (e.g., a set of vertices and attributes) to memory (e.g., system memory or memory 404). The model data defines each object visible on the display. The application then makes API calls to the driver kernel, requesting the rendering and display of the model data. The driver kernel reads the model data and writes commands to one or more streams to perform operations that process the model data. These commands may reference different shader programs to be implemented on processing units within the PPU 400, including one or more vertex shaders, shell shaders, domain shaders, geometry shaders, and pixel shaders. For example, one or more processing units may be configured to execute a vertex shader program that processes multiple vertices defined by the model data. In embodiments, different processing units may be configured to execute different shader programs concurrently. For example, a first subset of processing units may be configured to execute a vertex shader program, while a second subset of processing units may be configured to execute a pixel shader program. The first subset of processing units processes the vertex data to produce processed vertex data and writes the processed vertex data to L2 cache and / or memory 404. After the processed vertex data is rasterized (e.g., transformed from 3D data to 2D data in screen space) to produce fragment data, a second subset of processing units executes pixel shaders to produce the processed fragment data, which is then mixed with other processed fragment data and written to the frame buffer in memory 404. The vertex shader and pixel shader programs can execute concurrently, pipelinedly processing different data from the same scene until all model data for that scene has been rendered to the frame buffer. The contents of the frame buffer are then transferred to the display controller for display on the display device.

[0113] Images generated using one or more of the techniques disclosed herein can be displayed on a monitor or other display device. In some embodiments, the display device may be directly coupled to the system or processor that generates or renders the image. In other embodiments, the display device may be indirectly coupled to the system or processor, for example, via a network. Examples of such networks include the Internet, mobile telecommunications networks, Wi-Fi networks, and any other wired and / or wireless networking systems. When the display device is indirectly coupled, images generated by the system or processor can be streamed to the display device over the network. Such streaming allows, for example, video games or other applications that render images to execute on servers, data centers, or cloud-based computing environments, and the rendered images are transmitted and displayed on one or more user devices (e.g., computers, video game consoles, smartphones, other mobile devices, etc.) physically separate from the server or data center. Therefore, the techniques disclosed herein can be applied to enhance streamed images and services that stream images, such as NVIDIA GeForce Now (GFN), Google Stadia, etc.

[0114] Example streaming system

[0115] Figure 6 An exemplary streaming system 605 according to at least one embodiment is illustrated. Figure 6 Includes server 603 (which may include with Figure 5A Example processing system 500 and / or Figure 5B (Similar components, features and / or functions to exemplary system 565), client 604 (which may include similar ... Figure 5A Example processing system 500 and / or Figure 5B The exemplary system 565 has similar components, features, and / or functions to the network 606 (which may be similar to the network described herein). In some embodiments of this disclosure, system 605 may be implemented.

[0116] In one embodiment, streaming system 605 is a game streaming system and server 603 is a game server. In system 605, for a game session, client device 604 can simply receive input data in response to input from input device 626, send the input data to server 603, receive encoded display data from server 603, and display the display data on display 624. In this way, computationally intensive computation and processing are offloaded to server 603 (e.g., rendering of the game session's graphics output, especially ray or path tracing, is performed by the GPU 615 of server 603). In other words, the game session is streamed from server 603 to client device 604, thereby reducing the demands on client device 604 for graphics processing and rendering.

[0117] For example, regarding the instantiation of a game session, client device 604 can display frames of the game session on display 624 based on display data received from server 603. Client device 604 can receive input from one of input devices 626 and generate input data in response. Client device 604 can send the input data to server 603 via communication interface 621 and over network 606 (e.g., the Internet), and server 603 can receive the input data via communication interface 618. CPU 608 can receive the input data, process the input data, and send the data to GPU 615, which causes GPU 615 to generate a rendering of the game session. For example, the input data can represent the movement of a user character in the game, such as firing a weapon, reloading, passing a ball, turning a vehicle, etc. Rendering component 612 can render the game session (e.g., representing the result of the input data), and rendering capture component 614 can capture the rendering of the game session as display data (e.g., image data as frames of the captured game session rendering). The rendering of a game session may include lighting and / or shadow effects computed using one or more parallel processing units of server 603 (e.g., a GPU, which may further employ one or more dedicated hardware accelerators or processing cores to perform ray or path tracing techniques). Encoder 616 can then encode the display data to generate encoded display data, which can be sent to client device 604 via communication interface 618 through network 606. Client device 604 can receive the encoded display data via communication interface 621, and decoder 622 can decode the encoded display data to generate display data. Client device 604 can then display the display data via display 624.

[0118] It should be noted that the techniques described herein can be contained in executable instructions stored in a computer-readable medium for use by or in conjunction with a processor-based instruction execution machine, system, apparatus, or device. Those skilled in the art will appreciate that, for some embodiments, various different types of computer-readable media may be included for storing data. When used herein, “computer-readable medium” includes one or more of any suitable media for storing executable instructions of a computer program, such that an instruction execution machine, system, apparatus, or device can read (or retrieve) the instructions from the computer-readable medium and execute those instructions to implement the described embodiments. Suitable storage formats include one or more of electronic, magnetic, optical, and electromagnetic formats. A non-exhaustive list of conventional exemplary computer-readable media includes: portable computer disks; random access memory (RAM); read-only memory (ROM); erasable programmable read-only memory (EPROM); flash memory devices; and optical storage devices, including portable compact discs (CDs), portable digital video discs (DVDs), and the like.

[0119] It should be understood that the arrangement of components shown in the accompanying drawings is for illustrative purposes, and other arrangements are possible. For example, one or more of the elements described herein may be implemented wholly or partially as electronic hardware components. Other elements may be implemented in software, hardware, or a combination of software and hardware. Moreover, some or all of these other elements may be combined, some may be omitted entirely, and additional components may be added while still achieving the functionality described herein. Therefore, the subject matter described herein can be implemented in many different variations, and all such variations are contemplated to be within the scope of the claims.

[0120] To facilitate understanding of the topics described herein, many aspects are described in sequence of actions. Those skilled in the art will recognize that various actions can be performed by dedicated circuitry or circuit systems, by program instructions executed by one or more processors, or by a combination of both. The description of any sequence of actions herein is not intended to imply that a particular order in which the actions described for execution must be followed. All methods described herein can be performed in any suitable order unless otherwise indicated herein or otherwise clearly contradicted by the context.

[0121] In the context of describing the subject matter (especially in the context of the claims below), the use of the terms “a,” “an,” “this,” and similar designations should be interpreted to cover both the singular and plural, unless otherwise specified herein or obviously contradicted by the context. The use of the term “at least one” (e.g., at least one of A and B) followed by a list of one or more items should be interpreted to mean one item selected from the listed items (A or B), or any combination of two or more of the listed items (A and B), unless otherwise specified herein or obviously contradicted by the context. Furthermore, the foregoing description is for illustrative purposes only and not for limiting purposes, as the scope of protection sought is defined by the claims set forth thereafter with their equivalents. The use of any and all example or exemplary language provided herein (e.g., “such as”) is intended merely to better illustrate the subject matter and does not constitute a limitation on the scope of the subject matter, unless otherwise stated. The use of “based on,” and other similar phrases indicating conditions leading to the result, in both the claims and the written description, is not intended to exclude any other conditions leading to that result. The language in the description should not be interpreted as indicating that any unclaimed element is essential for the implementation of the claimed embodiment.

Claims

1. A method for encoding video, comprising: During the first look-through, one or more frames of the video are processed to extract features and determine the fixed group of pictures (GOP) structure. The fixed GOP structure is optimized based on the extracted features to generate an adaptive GOP structure; The adapted GOP structure is used, and one or more frames of the video are processed during the second look-through to update the extracted features based on the adapted GOP structure; as well as The one or more frames of the video are encoded according to the adapted GOP structure and the quantization parameter QP graph contained in the extracted features.

2. The method according to claim 1, wherein the extracted features include one or more detected scene changes and frame complexity information.

3. The method according to claim 1, further comprising: The resolution of one or more frames of the video is reduced before at least one of the first look-through or the second look-through.

4. The method of claim 3, wherein the resolution is reduced to a lower resolution specified by user-defined input.

5. The method of claim 1, wherein at least one of the one or more frames is associated with a CU tree or MB tree generated during the second look-through.

6. The method of claim 1, wherein the encoding is performed using an encoder circuit system, and at least one of the first look-through, the second look-through, or the optimization is performed by a processor separate from the encoder circuit system.

7. The method of claim 1, wherein the encoding is performed using a remote encoder circuit system, and at least the adapted GOP structure and the QP diagram are transmitted to the remote encoder circuit system via a network.

8. The method of claim 1, wherein at least one of the following is performed on a server or in a data center: processing the one or more frames of the video, the optimization, processing the one or more frames using the adapted GOP structure, or the encoding; and the encoded frames are streamed to a user equipment.

9. The method of claim 1, wherein at least one of the following is performed in a cloud computing environment: processing the one or more frames of the video, the optimization, processing the one or more frames using the adapted GOP structure, or the encoding.

10. The method of claim 1, wherein at least one of the following is performed for training, testing, or certifying a neural network used in a machine, robot, or autonomous vehicle: processing the one or more frames of the video, the optimization, processing the one or more frames using the adapted GOP structure, or the encoding.

11. The method of claim 1, wherein at least one of the following is performed on a virtual machine including a portion of a graphics processing unit: processing the one or more frames of the video, the optimization, processing the one or more frames using the adapted GOP structure, or the encoding.

12. The method of claim 1, wherein at least one of the following is implemented to include advanced error correction, fault tolerance, and self-healing capabilities: processing the one or more frames of the video, the optimization, processing the one or more frames using the adaptive GOP structure, or the encoding.

13. A system for encoding video, comprising: A memory that stores multiple frames of the video; as well as One or more processors, said processors coupled to said memory for performing operations including: During the first look-through, one or more frames of the video are processed to extract features and determine the fixed group of pictures (GOP) structure. The fixed GOP structure is optimized based on the extracted features to generate an adaptive GOP structure; During the second look-through, the adapted GOP structure is used to process one or more frames of the video to update the extracted features based on the adapted GOP structure; and The one or more frames of the video are encoded according to the adapted GOP structure and the quantization parameter QP graph contained in the extracted features.

14. The system of claim 13, wherein the extracted features include one or more detected scene changes and frame complexity information.

15. The system of claim 13, further comprising: The resolution of at least one of the one or more frames of the video is reduced before at least one of the first look-through and the second look-through.

16. The system of claim 13, wherein at least one of the one or more frames is associated with a CU tree or MB tree generated during the second look-through.

17. A non-transitory computer-readable medium storing computer instructions for encoding video, the computer instructions, when executed by one or more processors, causing the one or more processors to perform the following steps: During the first look-through, one or more frames of the video are processed to extract features and determine the fixed group of pictures (GOP) structure. The fixed GOP structure is optimized based on the extracted features to generate an adaptive GOP structure; Using the adapted GOP structure and processing one or more frames of the video during the second look-through to update the extracted features based on the adapted GOP structure; and The one or more frames of the video are encoded according to the adapted GOP structure and the quantization parameter QP graph contained in the extracted features.

18. The non-transitory computer-readable medium of claim 17, wherein the extracted features include one or more detected scene changes and frame complexity information.

19. The non-transitory computer-readable medium of claim 17, further comprising: The resolution of at least one of the one or more frames of the video is reduced before at least one of the first look-through and the second look-through.

Images