Generating statistical data for a coding unit tree to improve performance and compression efficiency

Abstract

Several embodiments include techniques for generating statistical data of a coding unit tree for video blocks contained in a media frame. The disclosed video encoder includes a first-in-first-out (FIFO) memory for storing coding data for multiple video blocks. A first set of units within the video encoder generates coding data for each video block. After storing the coding data in the FIFO memory, the first set of units can proceed with coding additional blocks of the media frame without having to wait for a second set of units within the video encoder to retrieve and process the coding data. Subsequently, the second set of units can efficiently retrieve and process data for multiple blocks of the media frame simultaneously.Furthermore, the video encoder can generate both forward-looking and backward-looking prediction data to encode a current block of the media frame, resulting in improved video quality.

Description

BACKGROUND Area of ​​the various implementation examples

[0001] Various implementation examples generally relate to video coding architectures and, in particular, to the generation of statistical data of a coding unit tree to increase performance and compression efficiency. Description of the state of the art

[0002] When streaming live or pre-recorded video, a first computer system, such as a server, data center, cloud storage system, and / or the like, transmits a video stream to a second computer system, such as a smartphone, tablet computer, laptop computer, and / or the like. Transmitting video streams between computer systems can consume a significant amount of network bandwidth, reducing the bandwidth available for other uses. Therefore, one goal of computer systems transmitting video streams is to compress and encode the video streams before transmission without significantly reducing the video quality. Computer systems receiving such video streams decompress and decode them before displaying them on one or more display devices.

[0003] When compressing and encoding a video stream, a computer system typically includes a hardware video encoder that divides each media frame (also called a media image) in the video stream into blocks, with each block containing a group of adjacent pixels from the media frame. Each block of adjacent pixels can be an 8×8 block of pixels, a 16×16 block of pixels, and / or the like. Depending on the video format used for encoding, these blocks are referred to as macroblocks, coding tree units (CTUs), and / or the like. The video encoder typically encodes the blocks as a set of lines, with the blocks in each line being encoded sequentially from left to right and the lines being encoded from top to bottom.

[0004] When a video encoder employs coding unit tree (CU tree) techniques, it can generate statistical data to optimize compression efficiency and / or improve video quality by producing specific statistical data associated with the media frames of the video stream being encoded. For example, the encoder can set a quantization parameter (QP) based on the predicted importance of the current coding block for encoding future frames. The encoder determines this predicted importance based on one or more of the following: intra-frame cost data, inter-frame cost data, and / or motion vector data. Conventional systems implementing these CU tree techniques may be subject to certain limitations.

[0005] First, conventional systems implementing CU-tree techniques transfer statistical data, such as intra-frame cost data, inter-frame cost data, and / or the like, by generating an interrupt at the coding block level. Generally, such interrupts require immediate attention and processing by a microcontroller or other processing unit associated with the video encoder. Consequently, to handle such an interrupt, conventional video encoders temporarily stall the execution of instructions by the microcontroller or other processing unit in the normal flow and sequence of operations.(also blocking, delaying), execute instructions associated with the interrupt to read and / or otherwise process the intra-frame cost data and / or inter-frame cost data, and then resume instruction execution in the normal flow and sequence. Because the normal flow of instruction execution is stalled (gestalted) each time a conventional video encoder handles an interrupt, the video encoder's performance decreases with an increasing number of interrupts.

[0006] Second, conventional systems implementing CU-tree techniques generate motion vector data partly based on motion vector surface data from previously processed blocks. For example, the video encoder can use motion vector data from a matching block of the previous media frame to generate a forward direction prediction that forecasts the current block of the current media frame. However, because the next media frame has not yet been processed, conventional systems cannot generate a backward prediction that forecasts the current block based on motion vector data from a matching block of the next media frame. This condition arises because the motion vector data for the next media frame has not yet been generated. Since motion vector data is only calculated in one direction, the quality of the motion vector data can be inaccurate when implementing CU-tree techniques, resulting in reduced video quality.

[0007] As the above illustrates, more effective techniques for generating intra-frame cost data, inter-frame cost data and motion vector data by a video encoder in a computing system are required in the prior art. SUMMARY

[0008] Various embodiments of the present disclosure provide a computer-implemented method for generating statistical data for a current media frame. The method includes generating initial interframe data for a first block of the current media frame. The method further includes generating initial intraframe data for the first block of the current media frame. The method further includes storing the initial interframe data and the initial intraframe data in a first-in-first-out (FIFO) memory. The method further includes generating a second interframe data and a second intraframe data for a second block of the current media frame, while the initial interframe data and the initial intraframe data are stored in the FIFO memory and are accessed without stalling (i.e., halting processing) for the initial interframe data and the initial intraframe data.for accessing or for accessing the first interframe data and the first intraframe data).

[0009] Other embodiments include, without limitation, a system implementing one or more aspects of the disclosed techniques, and one or more computer-readable media containing instructions for performing one or more aspects of the disclosed techniques, as well as a method for performing one or more aspects of the disclosed techniques.

[0010] At least one technical advantage of the disclosed techniques over the prior art is that, with the disclosed techniques, units within the video encoder do not have to pause for each block in a media frame until the interframe and intraframe data for each block are retrieved. Instead, units within the video encoder can retrieve and process the interframe and intraframe data for multiple blocks simultaneously, resulting in improved efficiency and performance compared to conventional approaches. Another advantage of the disclosed techniques is that, because the video encoder can retrieve data for multiple blocks in multiple media frames, it can encode a current block and determine the predicted importance of the current encoded block by accessing interframe data for matching blocks in a previous media frame and a subsequent media frame.As a result, the video encoder can generate both forward-direction motion vector data based on data from the previous media frame and reverse-direction motion vector data based on data from the next media frame. These advantages represent one or more technological improvements over prior art approaches. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] To better understand the features of the various embodiments mentioned above, a more detailed description of the inventive concepts summarized above can be provided with reference to various embodiments, some of which are illustrated in the accompanying drawings. It should be noted, however, that the accompanying drawings illustrate only typical embodiments of the inventive concepts and are therefore in no way intended to limit the scope, and that other equally effective embodiments exist. Fig. Figure 1 is a block diagram of a computer system configured to implement one or more aspects of the various embodiments; Fig. Figure 2 is a block diagram of a parallel processing unit (PPU) located in the auxiliary processing subsystem of Fig. 1 is included, according to various embodiments; Fig. Figure 3 is a block diagram of a general processing cluster (GPC) located in the parallel processing unit (PPU) of Fig. 2 is included, according to various embodiments; Fig. 4 is a block diagram of a video encoder configured to encode data for multiple blocks of a media frame for the computing system of Fig. 1- Fig. 3 to produce and process, according to various embodiments; Fig. Figure 5 illustrates a functional view of a video encoder that outputs a media frame for the computer system of Fig. 1- Fig. 4 can be coded according to various examples; Fig. Figure 6 illustrates the storage of forward prediction hint data for a media frame processed by the video encoders of Fig. 4- Fig. 5 is coded according to various examples; Fig. Figure 7 illustrates the storage of forward prediction hint data and backward prediction hint data for a media frame processed by the video encoders of Fig. 4- Fig. 5 is coded according to various examples; Fig. Figure 8 is a flowchart of process steps for generating statistical data of a coding unit tree for a media frame by the computing system of Fig. 1- Fig. 7, according to various embodiments; and Fig. Figure 9 is a flowchart of process steps for generating and storing hint data for a media frame by the computer system of Fig. 1- Fig. 7, according to various embodiments. DETAILED DESCRIPTION

[0012] The following description presents numerous specific details to provide a more thorough understanding of the various embodiments. However, it is obvious to a person skilled in the art that the inventive concepts can be implemented without one or more of these specific details. System overview

[0013] Fig. Figure 1 is a block diagram of a computing system 100 configured to implement one or more aspects of the various embodiments. As shown, the computing system 100 includes, without limitation, a central processing unit (CPU) 102 and a system memory 104, which is coupled to an auxiliary processing subsystem 112 via a memory bridge 105 and a communication path 113. The memory bridge 105 is further coupled to an I / O (input / output) bridge 107 via a communication path 106, and the I / O bridge 107 is in turn coupled to a switch 116.

[0014] In operation, the I / O bridge 107 is configured to receive user input information from input devices 108, such as a keyboard or mouse, and to forward the input information to the CPU 102 for processing via the communication path 106 and the memory bridge 105. In some examples, input devices 108 are used to verify the identities of one or more users, to allow access to the computer system 100 for authorized users, and to deny access to the computer system 100 for unauthorized users. The switch 116 is configured to provide connections between the I / O bridge 107 and other components of the computer system 100, such as a network adapter 118 and various add-in cards 120 and 121. In some examples, the network adapter 118 serves as the primary or exclusive input device to receive input data for processing via the disclosed techniques.

[0015] As also shown, the I / O bridge 107 is coupled to a system disk 114, which can be configured to store content, applications, and data for use by the CPU 102 and the auxiliary processing subsystem 112. Generally, the system disk 114 provides non-volatile memory for applications and data and can include fixed or removable hard disk drives, flash memory devices, and CD-ROM (Compact Disc Read-Only Memory), DVD-ROM (Digital Versatile Disc-ROM), Blu-ray, HD-DVD (High Definition DVD), or other magnetic, optical, or solid-state storage devices. Finally, although not explicitly shown, other components, such as Universal Serial Bus or other port connections, Compact Disc drives, Digital Versatile Disc drives, movie recording devices, and the like, can also be connected to the I / O bridge 107.

[0016] In various embodiments, the memory bridge 105 can be a northbridge chip, and the I / O bridge 107 can be a southbridge chip. Additionally, the communication paths 106 and 113, as well as other communication paths within the computer system 100, can be implemented using any technically suitable protocols, including, without limitation, Peripheral Component Interconnect Express (PCle), HyperTransport, or any other bus or point-to-point communication protocol known in the art.

[0017] In some embodiments, the auxiliary processing subsystem 112 comprises a graphics subsystem that supplies pixels to a display device 110, which may be any conventional cathode ray tube, liquid crystal display, light-emitting diode display, or the like. In such embodiments, the auxiliary processing subsystem 112 includes circuits optimized for graphics and video processing, including, for example, video output circuits. As shown below in Fig. As described in more detail in Section 2, such circuits can be contained via one or more auxiliary processors included in the auxiliary processing subsystem 112. An auxiliary processor contains any one or more processing units capable of executing instructions, such as a processor of a reduced instruction set computer (RISC), a central processing unit (CPU), or a parallel processing unit (PPU). Fig. 2 to Fig. 3, a graphics processing unit (GPU), a direct memory access unit (DMA), an intelligence processing unit (IPU), a neural processing unit (NAU), a tensor processing unit (TPU), a neural network processor (NNP), a data processing unit (DPU), a vision processing unit (VPU), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) and / or the like.

[0018] In some embodiments, the auxiliary processing subsystem 112 includes two processors, referred to herein as a primary processor (usually a CPU) and a secondary processor. Typically, the primary processor is a CPU and the secondary processor is a GPU. Additionally or alternatively, both the primary processor and the secondary processor can be any one or more of the types of auxiliary processors disclosed herein in any technically feasible combination. The secondary processor receives secure instructions from the primary processor via an unsecured communication path. The secondary processor accesses memory and / or another storage system, such as system memory 104, Compute eXpress Link (CXL) memory expanders, memory-managed disk storage, on-chip memory, and / or the like.The secondary processor accesses this memory and / or another storage system over an insecure connection. The primary and secondary processors can communicate with each other over a GPU-to-GPU communication channel, such as Nvidia Link (NVLink). Furthermore, the primary and secondary processors can communicate with each other over network adapter 118. Generally, the distinction between an insecure and a secure communication path is application-dependent. A given application program generally considers communications within a chip or packet to be secure. Communications of unencrypted data over a standard communication channel, such as PCIe, are considered insecure.

[0019] In some embodiments, the auxiliary processing subsystem 112 contains circuits optimized for general-purpose and / or arithmetic processing. These circuits may be accessed via one or more auxiliary processors within the auxiliary processing subsystem 112, configured to perform such general-purpose and / or arithmetic operations. In other embodiments, the one or more auxiliary processors within the auxiliary processing subsystem 112 may be configured to perform graphics processing, general-purpose processing, and arithmetic operations. The system memory 104 contains at least one device driver 103 configured to manage the processing operations of the one or more auxiliary processors within the auxiliary processing subsystem 112.

[0020] In various embodiments, the auxiliary processing subsystem 112 can be combined with one or more of the other elements of Fig. 1. be integrated to form a single system. For example, the auxiliary processing subsystem 112 can be integrated with the CPU 102 and other interconnect circuitry on a single chip to form a system-on-a-chip (SoC).

[0021] It is understood that the system shown herein is for illustrative purposes only and that variations and modifications are possible. The connection topology, including the number and arrangement of bridges, the number of CPUs 102, and the number of auxiliary processing subsystems 112, can be modified as desired. For example, in some embodiments, the system memory 104 could be directly connected to the CPU 102 instead of through the memory bridge 105, and other devices would communicate with the system memory 104 via the memory bridge 105 and the CPU 102. In other alternative topologies, the auxiliary processing subsystem 112 could be connected to the I / O bridge 107 or directly to the CPU 102 instead of to the memory bridge 105. In still other embodiments, the I / O bridge 107 and the memory bridge 105 could be integrated onto a single chip instead of existing as one or more discrete devices.Finally, in certain embodiments, one or more components that are in . Fig. The items shown in Figure 1 may not be present. For example, switch 116 could be omitted, and network adapter 118 and add-in cards 120 and 121 would connect directly to I / O bridge 107.

[0022] Fig. Figure 2 is a block diagram of a parallel processing unit (PPU) 202, which is located in the auxiliary processing subsystem 112 of Fig. 1 is included, according to various embodiments. Although Fig. 2. As shown above, a PPU 202 indicates that the auxiliary processing subsystem 112 can contain any number of PPUs 202. Furthermore, the PPU 202 is of Fig. 2 an example of an auxiliary processor located in the auxiliary processing subsystem 112 of Fig. 1 is included. Alternative auxiliary processors include, without limitation, RISCs, CPUs, GPUs, DMA units, IPUs, NPUs, TPUs, NNPs, DPUs, VPUs, ASICs, FPGAs and / or the like. The ones in the Fig. 2- Fig. The techniques disclosed in Section 3 relating to the PPU 202 apply equally to any type of auxiliary processor(s) included in the auxiliary processing subsystem 112, in any combination. As shown, the PPU 202 is coupled to a local parallel processing (PP) memory 204. The PPU 202 and the PP memory 204 can be implemented using one or more integrated circuit devices, such as programmable processors, application-specific integrated circuits (ASICs), or memory devices, or in any other technically feasible manner.

[0023] In some embodiments, the PPU 202 includes a graphics processing unit (GPU) that can be configured to implement a graphics rendering pipeline to perform various operations related to generating pixel data based on graphics data supplied by the CPU 102 and / or the system memory 104. When processing graphics data, the PP memory 204 can be used as graphics memory, storing one or more conventional frame buffers and, if required, one or more other render targets. Among other things, the PP memory 204 can be used to store and update pixel data and deliver final pixel data or display frames to the display device 110 for display. In some embodiments, the PPU 202 can also be configured for general-purpose processing and computational operations.

[0024] During operation, CPU 102 is the main processor of the computer system 100, controlling and coordinating the operations of other system components. Specifically, CPU 102 issues instructions that control the operation of PPU 202. In some embodiments, CPU 102 writes a stream of instructions for PPU 202 into a data structure (not explicitly in the system). Fig. 1 or Fig. (2 shown), which may be located in system memory 104, PP memory 204, or another memory location accessible to both CPU 102 and PPU 202. Additionally or alternatively, processors and / or auxiliary processors other than CPU 102 may write one or more streams of instructions for PPU 202 into a data structure. A pointer to the data structure is written to a push buffer to initiate the processing of the stream of instructions in the data structure. PPU 202 reads instruction streams from the push buffer and then executes instructions asynchronously with respect to the operation of CPU 102. In embodiments where multiple push buffers are created, execution priorities for each push buffer can be specified by an application program via device driver 103 to control the scheduling of the different push buffers.

[0025] As also shown, the PPU 202 contains an I / O (input / output) unit 205, which communicates with the rest of the computing system 100 via the communication path 113 and the memory bridge 105. The I / O unit 205 generates packets (or other signals) for transmission on the communication path 113 and also receives all incoming packets (or other signals) from the communication path 113, routing the incoming packets to appropriate components of the PPU 202. For example, commands relating to processing tasks can be routed to a host interface 206, while commands relating to memory operations (e.g., reading from or writing to the PP memory 204) can be routed to a crossbar unit 210. The host interface 206 reads each push buffer and transmits the instruction stream stored in the push buffer to a front end 212.

[0026] As above in connection with Fig. As mentioned in Section 1, the connection of the PPU 202 to the rest of the computing system 100 can be varied. In some embodiments, the auxiliary processing subsystem 112, which contains at least one PPU 202, is implemented as an add-in card that can be inserted into an expansion slot of the computing system 100. In other embodiments, the PPU 202 can be integrated on a single chip with a bus bridge, such as the memory bridge 105 or the I / O bridge 107. In still other embodiments, some or all of the elements of the PPU 202 can be included together with the CPU 102 in a single integrated circuit or a single system-on-a-chip (SoC).

[0027] During operation, the front end 212 transfers processing tasks received from the host interface 206 to a workload distribution unit (not shown) within the task / work unit 207. The workload distribution unit receives pointers to processing tasks, which are encoded as task metadata (TMD) and stored in memory. The pointers to TMDs are contained in an instruction stream, which is stored as a push buffer and received by the front end 212 from the host interface 206. Processing tasks, which can be encoded as TMDs, contain indices associated with the data to be processed, as well as state parameters and instructions that define how the data is to be processed. For example, the state parameters and instructions could define the program to be executed on the data.The task / work unit 207 receives tasks from the front end 212 and ensures that the GPCs 208 are configured to a valid state before initiating the processing task specified by each of the TMDs. A priority can be specified for each TMD, which is used to schedule the execution of the processing task. Processing tasks can also be received from the processing cluster array 230. Optionally, the TMD can include a parameter that controls whether the TMD is added to the head or the end of a list of processing tasks (or to a list of pointers to the processing tasks), thus providing another level of control over the execution priority.

[0028] The PPU 202 advantageously implements a highly parallel processing architecture based on a processing cluster array 230 containing a set of C general processing clusters (GPCs) 208, where C ≠ 1. Each GPC 208 is capable of executing a large number (e.g., hundreds or thousands) of threads concurrently, with each thread being an instance of a program. In different applications, different GPCs 208 can be allocated to process different types of programs or to perform different types of computations. The allocation of GPCs 208 can vary depending on the workload generated by each type of program or computation.

[0029] The memory interface 214 contains a set of D partition units 215, where D ≠ 1. Each partition unit 215 is coupled to one or more dynamic random access memories (DRAMs) 220 located within the PP memory 204. In one embodiment, the number of partition units 215 is equal to the number of DRAMs 220, and each partition unit 215 is coupled to a different DRAM 220. In other embodiments, the number of partition units 215 may differ from the number of DRAMs 220. Those skilled in the art will recognize that a DRAM 220 can be replaced by any other technically suitable storage device.During operation, various render targets, such as texture images and frame buffers, can be stored across the DRAMs 220, enabling the partition units 215 to write sections of each render target in parallel to efficiently utilize the available bandwidth of the PP memory 204.

[0030] A given GPC 208 can process data to be written to any of the DRAMs 220 within the PP memory 204. The crossbar unit 210 is configured to route the output of each GPC 208 to the input of any partition unit 215 or to any other GPC 208 for further processing. The GPCs 208 communicate with the memory interface 214 via the crossbar unit 210 to read from or write to different DRAMs 220. In one embodiment, in addition to a connection to the PP memory 204 via the memory interface 214, the crossbar unit 210 has a connection to the I / O unit 205, enabling the processing cores within the different GPCs 208 to communicate with the system memory 104 or other memory not local to the PPU 202. In the embodiment of Fig. In 2, the crossbar unit 210 is directly connected to the I / O unit 205. In various embodiments, the crossbar unit 210 can use virtual channels to separate traffic flows between the GPCs 208 and the partition units 215.

[0031] Again, the GPCs 208 can be programmed to perform processing tasks related to a wide variety of applications, including, without limitation, linear and nonlinear data transformations, filtering of video and / or audio data, modeling operations (e.g., applying laws of physics to determine the position, velocity, and other attributes of objects), image rendering operations (e.g., tessellation shaders, vertex shaders, geometry shaders, and / or pixel / fragment shader programs), general arithmetic operations, etc. In operation, the PPU 202 is configured to transfer data from system memory 104 and / or PP memory 204 to one or more on-chip memory units, process the data, and write the resulting data back to system memory 104 and / or PP memory 204.The result data can then be accessed by other system components, including CPU 102, another PPU 202 within the auxiliary processing subsystem 112, or another auxiliary processing subsystem 112 within the computing system 100.

[0032] As noted above, any number of PPUs 202 can be contained in an auxiliary processing subsystem 112. For example, multiple PPUs 202 can be provided on a single add-in card, or multiple add-in cards can be connected via the communication path 113, or one or more of the PPUs 202 can be integrated into a bridge chip. The PPUs 202 in a multi-PPU system can be identical or different from one another. For example, different PPUs 202 could have different numbers of processing cores and / or different amounts of PP memory 204. In implementations where multiple PPUs 202 are present, these PPUs can be operated in parallel to process data at a higher throughput than is possible with a single PPU 202.Systems containing one or more PPUs 202 can be implemented in a variety of configurations and form factors, including, without limitation, desktops, laptops, handheld personal computers or other handheld devices, servers, workstations, game consoles, embedded systems and the like.

[0033] Fig. Figure 3 is a block diagram of a general processing cluster (GPC) 208, which is located in the parallel processing unit (PPU) 202 of Fig. Section 2 is included, according to various embodiments. In operation, the GPC 208 can be configured to execute a large number of threads in parallel to perform graphics, general processing, and / or computational operations. As used herein, a "thread" refers to an instance of a particular program running on a specific set of input data. In some embodiments, single-instruction, multiple-data (SIMD) instruction output techniques are used to support the parallel execution of a large number of threads without providing multiple independent instruction units.In other embodiments, single-instruction, multiple-thread (SIMT) techniques are used to support the parallel execution of a large number of generally synchronized threads, using a common instruction unit configured to issue instructions to a set of processing engines within the GPC 208. Unlike a SIMD execution regime, where all processing engines typically execute identical instructions, SIMT execution allows different threads to more easily follow diverging execution paths through a given program. Those skilled in the field will recognize that a SIMD processing regime represents a functional subset of a SIMT processing regime.

[0034] The operation of the GPC 208 is controlled by a pipeline manager 305, which distributes processing tasks received by a work distribution unit (not shown) within the task / work unit 207 to one or more streaming multiprocessors (SMs) 310. The pipeline manager 305 can also be configured to control a work distribution crossbar 330 by specifying destinations for processed data output by the SMs 310.

[0035] In one embodiment, the GPC 208 contains a set of M SMs 310, where M ≥ 1. Each SM 310 also contains a set of functional execution units (not shown), such as execution units and load-store units. Processing operations specific to each of the functional execution units can be pipelined, allowing a new instruction to be issued for execution before a previous instruction has completed execution. Any combination of functional execution units within a given SM 310 can be provided. In various embodiments, the functional execution units can be configured to support a wide variety of different operations, including integer and floating-point arithmetic (e.g., addition and multiplication), comparison operations, Boolean operations (e.g.,AND, OR, XOR), bit shifting, and computation of various algebraic functions (e.g., planar interpolation and trigonometric, exponential, and logarithmic functions, etc.). Advantageously, the same functional execution unit can be configured to perform different operations.

[0036] In operation, each SM 310 is configured to process one or more thread groups. As used herein, a "thread group" or "warp" refers to a group of threads that simultaneously execute the same program on different input data, with one thread of the group being assigned to a different execution unit within an SM 310. A thread group can contain fewer threads than the number of execution units within the SM 310, in which case some execution may be idle during cycles when that thread group is being processed. A thread group can also contain more threads than the number of execution units within the SM 310, in which case processing may occur over successive clock cycles. Since each SM 310 can support up to G thread groups concurrently, it follows that up to G*M thread groups can be executed in the GPC 208 at any given time.

[0037] Additionally, a large number of related thread groups (at different stages of execution) can be active simultaneously within an SM 310. The size of a given CTA is equal to m*k, where k is the number of concurrently executing threads in a thread group, typically an integer multiple of the number of execution units within the SM 310, and m is the number of concurrently active thread groups within the SM 310. In various embodiments, a software application written in the CUDA programming language (CUDA = Compute Unified Device Architecture) describes the behavior and operation of threads running on the GPC 208, including each of the behaviors and operations described above. A given processing task can be specified in a CUDA program, allowing the SM 310 to be configured to perform and / or manage general-purpose computing operations.

[0038] Although in Fig. Not shown in Figure 3, each SM 310 contains a Level One (L1) cache or uses space in a corresponding L1 cache outside the SM 310 to support, among other things, load and store operations performed by the execution units. Each SM 310 also has access to Level Two (L2) caches (not shown) shared among all GPCs 208 in the PPU 202. The L2 caches can be used to transfer data between threads. Finally, the SMs 310 also have access to a “global” memory outside the chip, which may contain PP memory 204 and / or system memory 104. It is understood that any memory outside the PPU 202 can be used as global memory. Additionally, as shown in Figure 3, the SMs 310 also have access to a global memory outside the chip, which may contain PP memory 204 and / or system memory 104. Fig. As shown in Figure 3, a Level One Point Five (L1.5) cache 335 can be contained within the GPC 208 and configured to receive and hold data requested from the memory via the memory interface 214 by the SM 310. Such data can include, without limitation, instructions, uniform data, and constant data. In embodiments with multiple SMs 310s within the GPC 208, the SMs 310 can advantageously share common instructions and data that are cached in the L1.5 cache 335.

[0039] Each GPC 208 can have an associated memory management unit (MMU) 320, which is configured to map virtual addresses to physical addresses. In various embodiments, the MMU 320 can be located either within the GPC 208 or within the memory interface 214. The MMU 320 contains a set of page table entries (PTEs) used to map a virtual address to a physical address of a tile or memory page, and optionally a cache row index. The MMU 320 can include address translation buffers (TLBs) or caches, which may be located within the SMs 310, within one or more L1 caches, or within the GPC 208.

[0040] In graphics and computing applications, the GPC 208 can be configured so that each SM 310 is coupled with a Texture Unit 315 to perform texture mapping operations, such as determining texture sample positions, reading texture data, and filtering texture data.

[0041] In operation, each SM 310 transfers a processed task to the work-distribution crossbar 330 to make the processed task available to another GPC 208 for further processing or to store the processed task in an L2 cache (not shown), PP memory 204, or system memory 104 via the crossbar unit 210. Additionally, a pre-raster operations (Pre-ROP) unit 325 is configured to receive data from the SM 310, route data to one or more raster operations (ROP) units within the partition units 215, perform color mixing optimizations, organize pixel color data, and perform address translations.

[0042] It is understood that the core architecture described herein is for illustrative purposes only and that variations and modifications are possible. Among other things, any number of processing units, such as SMs 310, texture units 315, or pre-ROP units 325, can be included within the GPC 208. As mentioned above in conjunction with Fig. As described in Section 2, the PPU 202 can further contain any number of GPCs 208 configured to be functionally similar to one another, so that the execution behavior does not depend on which GPC 208 receives a particular processing task. Furthermore, each GPC 208 operates independently of the other GPCs 208 in the PPU 202 to perform tasks for one or more application programs. In light of the foregoing, experts in the field will recognize that the Fig. 1- Fig. The architecture described in 3 does not in any way limit the scope of the various embodiments of the present disclosure.

[0043] It should be noted that, as used herein, references to shared memory may include any or more technically feasible memories, including, without limitation, local memory shared by one or more SMs 310, or memory accessible via the memory interface 214, such as cache memory, PP memory 204, or system memory 104. It should also be noted that, as used herein, references to cache memory may include any or more technically feasible memories, including, without limitation, an L1 cache, an L1.5 cache, and the L2 caches. Video coding architecture for generating statistical data of a coding unit tree for a media frame

[0044] Several embodiments include a video encoder with a first-in-first-out (FIFO) memory that stores intraframe cost data and / or interframe cost data for multiple blocks contained within a media frame of a video stream. Various units within the video encoder perform motion estimation, including full-pixel motion estimation and subpixel motion estimation, and motion compensation to generate interframe data, including an interframe candidate and interframe cost data. Interframe data, interframe candidates, and interframe cost data are also referred to herein as interdata, intercandidates, and intercost data, respectively.Various units within the video encoder generate motion vector data, including forward motion vector prediction data based on matching blocks in one or more reference frames, including matching blocks in a previous media frame, and backward motion vector prediction data based on matching blocks in a subsequent media frame. These units search for intercandidate blocks in the reference frame(s) and determine an intercost value for each intercandidate block. The intercost value of an intercandidate block is inversely proportional to the degree to which the intercandidate block matches the current block of the current media frame. The winning intercandidate block is the one with the lowest intercost value.

[0045] Furthermore, various units within the video encoder perform intraframe estimation and prediction to generate intraframe data, including an intraframe candidate and intraframe cost data. Intraframe data, intraframe candidates, and intraframe cost data are also referred to herein as intradata, intracandidates, and intracost data, respectively. These units within the video encoder search for intracandidate blocks in the current media frame that are adjacent to the current block and determine an intracost value for each intracandidate block. The intracost value of an intracandidate block is inversely proportional to the degree to which the intracandidate block matches the current block of the current media frame. The winning intracandidate block is the intracandidate block with the lowest intracost value.The video encoder stores the interframe and intraframe data for the blocks in the FIFO memory, thus avoiding stalling of the units in the video encoder until the interframe and intraframe data for each block are retrieved. After the video encoder generates the interframe and intraframe data for the first block of the current media frame and stores the data in the FIFO memory, it can generate the interframe and intraframe data for a second block of the current media frame while the interframe and intraframe data for the first block are still stored in the FIFO memory. This access to the interframe and intraframe data for the first block is possible without stalling.

[0046] After interframe and intraframe data for several blocks have been stored in the FIFO memory, various other units in the video encoder retrieve the interframe and intraframe data stored in the FIFO memory. These other units perform specific functions, including selecting between the interframe and intraframe candidates, generating motion vector data, filtering the selected candidate, and performing entropy coding to generate bits for the output encoded video stream.

[0047] Fig. Figure 4 is a block diagram of a video encoder 400 configured to encode data for multiple blocks of a media frame for the computer system 100. Fig. 1- Fig. 3 to generate and process, according to various embodiments. As shown, the video encoder 400 includes, without limitation, a controller 405, a full-pixel search (FPS) unit 410, a sub-pixel search (SPS) unit 415, a motion compensation filter type selection (MCT) unit 420, a rate distortion optimization (RDO) unit 425, a recon unit 430, a filter 435, an entropy encoder 440, an intermediate link 445, a pixel direct memory access (DMA) unit 450, a read collocated motion vector (RCOL) unit 455, and an external motion vector analysis ( RHINT-) unit 457, a Read Motion Vector Predictor (RMVP) unit 459, a cache memory (Cache) 460,a Mode Decision Processor (MDP) 465, a Multimedia Pipeline Encoder B (MPEB) unit 466, a Multimedia Pipeline Encoder C (MPEC) unit 467, a History (HIST) unit 470, a Motion Estimation DMA (MEDA) unit 475, a Write Motion Vector Predictor (WMVP) unit 477, and a Frame Buffer Interface (FB) unit 480.

[0048] Various units of the 400 video encoder communicate with each other via the 445 connection. These various units include the 410 FPS unit, the 415 SPS unit, the 420 MCT unit, the 425 RDO unit, the 455 RCOL unit, the 457 RHINT unit, the 459 RMVP unit, the 460 cache unit, the 465 MDP unit, the 466 MPEB unit, the 467 MPEC unit, the 470 history unit, the 475 MEDMA unit, the 477 WMVP unit, and / or the like. The 445 connection can include any suitable interconnect bus, mesh network, point-to-point connection, and / or the like for transmitting and receiving data between and among these units of the 400 video encoder.

[0049] The Video Encoder 400 can be configured to encode video in accordance with any one or more video encoding formats. In some embodiments, the Video Encoder 400 can encode a video stream compatible with the Advanced Video Coding (AVC) format, also known as the H.264 format or the Motion Picture Experts Group 4 (MPEG-4) Part 10 format. Additionally or alternatively, the Video Encoder 400 can encode a video stream compatible with the High Efficiency Video Coding (HEVC) format, also known as the H.265 format or the Motion Picture Experts Group High Efficiency (MPEG-H) Part 2 format. Additionally or alternatively, the Video Encoder 400 can encode a video stream compatible with the Video Compression 9 (VP9) format and / or the Alliance for Open Media (AOMedia) Video 1 (AV1) format.Additionally or alternatively, the Video Encoder 400, as is and / or with minor modification, can encode a video stream compatible with any other technically feasible video coding format, including, without limitation, Motion Joint Pictures Experts Group (JPEG) 2000 (MJ2), MPEG-2 or H.262, H.263v2 or H.263+, Video Coding 1 (VC-1 or SMPTE 421), Versatile Video Coding (VVC or H.266), VP8, VP10 and / or the like.

[0050] In some embodiments, certain units of the Video Encoder 400 can be general-purpose coding units that support operations for encoding video into multiple encoding formats. Additionally or alternatively, certain units of the Video Encoder 400 can be format-specific coding units that support operations for encoding video into a single encoding format or into two or three related encoding formats. For example, general-purpose coding units of the Video Encoder 400 can include, without limitation, the FPS unit 410, the SPS unit 415, the MCT unit 420, the PDMA unit 450, the RCOL unit 455, the RHINT unit 457, the RMVP unit 459, the cache memory 460, the MEDMA unit 475, the WMVP unit 477, and / or the like. Additionally or alternatively, format-specific coding units of the Video Encoder 400 can support operations for encoding video into the H.264 format and / or the H.Supporting the 265 format, without restriction it may include the MDP unit 465, the MPEB unit 466, the MPEC unit 467, the history unit 470, and / or the like. Additionally or alternatively, format-specific coding units of the video encoder 400, which support operations for encoding video into the AV1 format and / or the VP9 format, may without restriction include the RDO unit 425, the reconstruction unit 430, the filter 435, the entropy encoder 440, and / or the like.

[0051] In operation, the Controller 405 encodes media frames in a video stream in conjunction with other units and / or components of the Video Encoder 400. The Controller 405 may contain any one or more processors capable of executing instructions, including, without limitation, a microcontroller, a RISC processor, a CPU, a PPU, a GPU, a DMA unit, an IPU, a NAU, a TPU, an NNP, a DPU, a VPU, an ASIC, an FPGA, and / or the like. The Controller 405 may include memory for storing instructions that can program the Controller 405 to perform various operations described herein. The Controller 405 may further include memory for storing data associated with these operations.In this respect, the CPU 102, the auxiliary processing subsystem 112, and / or similar components can store instructions and / or data in the memory of the controller 405 via the memory bridge 105 and the communication path 113. Similarly, the controller 405 can communicate with the memory bridge 105 via the communication path 113. Through the memory bridge 105, the controller 405 can communicate with various other units and / or components of the computer system 100.

[0052] Furthermore, the controller 405 can communicate with various units and / or components of the video encoder 400, including, but not limited to, the FPS unit 410, the SPS unit 415, the MCT unit 420, the RDO unit 425, the reconstruction unit 430, the filter 435, the entropy encoder 440, and / or the like. The controller 405 can configure one or more of the units of the video encoder 400. Furthermore, the controller 405 can control the execution and operation of one or more of the units of the video encoder 400, including various operations for encoding media frames of a video stream. The controller 405 can receive data from the units of the video encoder 400 resulting from the execution of these various operations.

[0053] Media frames can be subdivided into block rows, with each block row containing a set of blocks extending from the left edge to the right edge of the media frame. Depending on the format used by the Video Encoder 400, the blocks may be referred to as macroblocks, coding tree units (CTUs), coding tree blocks (CTBs), superblocks, and / or the like. In some embodiments, the Video Encoder 400 encodes a video stream in the H.264 format, with media frames subdivided into 16 × 16 pixel macroblocks. In some embodiments, the Video Encoder 400 encodes a video stream in the HEVC format, with media frames subdivided into 32 × 32 pixel coding tree blocks. In some embodiments, the 400 video encoder encodes a video stream in the AV1 format, with media frames subdivided into 64 × 64 pixel superblocks.In various formats, including HEVC, AV1, and / or the like, the video encoder generates 400 predictions or indications of a coding unit granularity. In some embodiments, a coding unit is a square pixel block of various sizes, including, without limitation, 16 × 16 pixel coding units, 32 × 32 pixel coding units, 64 × 64 pixel coding units, and / or the like. In various formats, including HEVC, AV1, and / or the like, the video encoder generates 400 predictions or indications of a prediction unit granularity. In some embodiments, a prediction unit is a rectangular pixel block of various sizes, including, without limitation, 8 × 16 pixel prediction units, 16 × 8 pixel prediction units, 16 × 32 pixel prediction units, 32 × 16 pixel prediction units and / or the like.In some embodiments, a prediction unit is a square pixel block of various sizes, including, without limitation, 16 × 16 pixel prediction units, 32 × 32 pixel prediction units, 64 × 64 pixel prediction units and / or the like.

[0054] The controller 405, in conjunction with other units and / or components of the video encoder 400, can control various operations for generating inter-frame candidates for media frames of a video stream. To generate an inter-frame candidate, the controller 405 performs motion estimation and / or motion compensation for a block contained within a media frame of the video stream. The controller 405 performs motion estimation and / or motion compensation to generate an inter-frame candidate for the specified block based on temporal redundancy between media frames. Specifically, the controller 405 performs one or both of a full-pixel search in conjunction with the FPS unit 410 and / or a subpixel search in conjunction with the SPS unit 415.Control 405 performs one or both of these searches by looking at a reference media frame, such as the previous media frame and / or the next media frame, for blocks that match corresponding blocks in the current media frame being encoded. A block in the reference frame matches the block in the current frame if the pixel data of the block in the reference frame is the same as or similar to the pixel data of the current block in the current media frame. The matching block is the block in the reference frame with pixel data that is closest to the pixel data of the block in the current frame. If the objects in the scene have not moved and the camera view has not changed between the reference media frame and the current media frame, then the position of the current block within the current frame can be the same as the position of the matching block in the reference frame.However, if objects in the scene have moved and / or the camera view has changed between the reference media frame and the current media frame, then the position of the current block within the current frame may differ from the position of the corresponding block in the reference frame. Control 405 generates a motion vector for the current block in the current media frame that identifies the position of the corresponding block in the reference frame. Motion compensation predicts the pixels of a current media frame based on a previous media frame and / or a future media frame by determining effects caused by movement of the camera recording the video stream and / or movement of objects within the scene being recorded by the camera.Control 405 generates difference data, also known as residual data, which specifies the differences between the pixel data of the matching block in the reference frame and the pixel data of the current block in the current media frame. As the similarity between the pixel data of the block in the reference frame and the pixel data of the current block increases, the amount of difference data decreases, resulting in lower inter-frame costs. Conversely, as the similarity between the pixel data of the block in the reference frame and the pixel data of the current block decreases, the amount of difference data increases, resulting in higher inter-frame costs.

[0055] Control 405 generates a motion vector for the current block, which predicts the block of the current media frame from the corresponding block of a reference media frame. Control 405, in conjunction with other units and / or components of the Video Encoder 400, generates motion vector prediction data, also known as motion vector hint data or simply hint data, for each media frame. This hint data includes forward prediction hint data and backward prediction hint data. Control 405 generates forward prediction hint data based on block data for the current block and block data for a corresponding block of the previous media frame. Similarly, Control 405 generates backward prediction hint data based on block data for the current block and block data for a corresponding block of the next media frame.The 400 video encoder can use the motion vector data as an inter-frame candidate. Based on the motion vector data, the 405 controller generates motion-compensated pixels for the inter-frame candidate.

[0056] Depending on the currently used encoding format, the controller 405 transmits motion vector data and / or pixel block data to the MDP unit 465, the MPEB unit 466, and / or the MCT unit 420. When encoding in certain formats, such as H.264, H.265, and / or similar, the controller 405 transmits motion vector data and / or pixel block data to the MDP unit 465 and the MPEB unit 466. When encoding in certain other formats, such as AV1 and / or similar, the controller 405 transmits motion vector data and / or pixel block data to the MCT unit 420.

[0057] Furthermore, the controller 405, in conjunction with other units of the video encoder 400, can control various operations for generating intra-frame candidates for media frames of a video stream. To generate an intra-frame candidate, the controller 405 performs an intra-frame estimation and / or an intra-frame prediction to generate an intra-frame candidate for the specified block based on spatial redundancy within a media frame.

[0058] To perform intraframe estimation, the controller 405 selects an intraframe prediction mode based on the current pixels in the current media frame and the neighboring pixels of the reconstructed current media frame. In some embodiments, the controller 405 can select the intraframe prediction mode that best predicts the pixels of the current block. The controller 405 can select the intraframe prediction mode that results in the lowest rate distortion cost based on the sum of squared errors (SSE) distortion for the current block, as determined by the rate distortion optimization unit 425. The number and type of available prediction modes can vary based on the block size. For example, the number and type of available prediction modes may vary among 4 × 4 pixel blocks, 8 × 8 pixel blocks, 16 × 16 pixel blocks, 32 × 32 pixel blocks and / or the like.In this respect, the controller 405 can select different intraframe prediction modes for each of the possible block sizes based on which intraframe prediction mode results in the lowest rate distortion cost value, as determined by the rate distortion optimization unit 425 for that block size. Furthermore, the controller 405 can select different intraframe prediction modes for the luma values ​​in the block compared to the chroma samples in the block. In some embodiments, the intraframe prediction mode determines the order in which the pixels in the current block are sampled to generate the predicted intraframe candidate.For example, the intraframe prediction mode can specify vertical scanning, horizontal scanning, diagonal downward-left scanning, diagonal downward-right scanning, vertical left scanning, vertical right scanning, horizontal downward scanning, horizontal upward scanning and / or the like.

[0059] To perform an intraframe prediction, the 405 control generates an intraframe candidate based on the selected intraframe prediction mode. The 405 control samples the pixel values ​​in the current block in the order specified by the selected intraframe prediction mode. For each sampled pixel, the 405 control determines a predicted pixel value based on the differences between that pixel value and the pixel values ​​of adjacent pixels. From these predicted pixel values, the 405 control generates the intraframe candidate.

[0060] The controller 405 uses various other units contained within the video encoder 400 to perform the operations described herein. Specifically, the controller 405 uses the FPS unit 410 to perform full-pixel motion estimation. The FPS unit 410 performs a full-pixel search using integer pixel addresses to generate motion estimation data between pixels of a current media frame and corresponding pixels of a previous and / or next media frame on a pixel-by-pixel basis. Similarly, the controller 405 uses the PLC unit 415 to perform subpixel motion estimation.The PLC unit 415 performs a subpixel search using fractional pixel addresses to generate motion estimation data between subpixels of a current media frame and corresponding subpixels of a previous and / or next media frame on a subpixel-by-subpixel basis. Each subpixel can be half the size of a full pixel, a quarter the size of a full pixel, and / or the like.

[0061] The controller 405 uses the MCT unit 420 to perform motion compensation. The MCT unit 420 selects a filter type to perform motion compensation prediction. Specifically, the MCT unit 420 selects a motion compensation filter type to account for the interpolation of subpixels resulting from fractional motion vectors. Subpixels can be determined by filtering full pixels and full-pixel motion vectors. Motion compensation filter types can include bicubic filtering, bilateral filtering, and / or similar methods.

[0062] The RCOL unit 455 reads collocated motion vector data from memory and stores this collocated motion vector data in memory for access during motion estimation and / or other encoding operations. This collocated motion vector data for a previous media frame is written to memory beforehand by the MPEC unit 467.

[0063] The RMVP unit 459 reads motion vector prediction data, also known as motion vector hint data, from memory. This motion vector hint data is further processed in conjunction with Fig. 6- Fig. 7 described. In some embodiments, this motion vector guidance data is previously written to memory by the WMVP unit 477.

[0064] The MDP unit 465 determines the mode for encoding each block of the media frame. The potential modes can be an interframe mode, an intraframe mode, and / or the like. In some embodiments, the MDP unit 465 encodes the current block according to several encoding modes and selects the mode that results in the lowest cost, with the lowest cost resulting in the smallest amount of residual data for the block. The MDP unit 465 determines the final selection from among all candidates from the motion estimation, determines the optimal partitioning for interframe encoding, and determines the final choice between interframe encoding and intraframe encoding. Furthermore, the MDP unit 465 can generate motion estimation result data, motion compensation result data, and / or the like.

[0065] The MPEB unit 466 performs and / or supports various functions for the video encoder 400. These functions may include, without limitation, intraframe prediction, block size search and / or subblock size search, reconstruction, deblocking filter, sample adaptive offset (SAO) filtering, and / or the like. Furthermore, these functions may include, without limitation, the transformation, quantization, inverse quantization, and inverse transformation described herein.

[0066] The MPEC unit 467 performs and / or supports various functions for the video encoder 400. These functions can include, without limitation, certain entropy coding modes, such as context-adaptive variable length coding (CAVLC), context-based adaptive binary arithmetic coding (CABAC), and / or the like.

[0067] The history unit 470 stores data in memory and loads data from memory, including spatial clues, intra-frame predictions, SAO filter data, and entropy coding data for the current block row and / or the previous block row encoded by the video encoder 400. The history unit 470 can receive data from processed blocks in a block row, which is then accessed again while encoding blocks in the next block row. As a result, different units of the video encoder 400 can access data for adjacent blocks in the block row above the current block row being encoded.

[0068] The WMVP unit 477 writes motion vector prediction data, also known as motion vector hint data, to memory for each media frame. This motion vector hint data is further used in conjunction with Fig. 6- Fig. 7 described. This motion vector cue data can be used as cue data when the video encoder 400 encodes the following media frames. The RMVP unit 459 can read this motion vector cue data from memory.

[0069] Rate Distortion Optimization Unit 425 performs rate distortion optimization on the blocks contained within a media frame of the video stream. Rate Distortion Optimization Unit 425 selects a winning candidate for a block from between the interframe candidate for that block and the intraframe candidate for that block. Rate Distortion Optimization Unit 425 selects a winning candidate partly based on a predicted importance score generated from forward-prediction clue data and backward-prediction clue data. Rate Distortion Optimization Unit 425 further receives the reconstructed pixels of the block in the reconstructed current media frame from Reconstruction Unit 430 via the feedback loop from Reconstruction Unit to FPS Unit 410.Based on the reconstructed pixels of the block in the reconstructed current media frame, the Rate Distortion Optimization Unit 425 determines a rate distortion cost value based on the sum of squared error (SSE) distortion for the current block. The Rate Distortion Optimization Unit 425 selects the winning candidate, at least in part, based on the rate distortion cost value for the current block, as determined from the interframe cost value and the intraframe cost value. In some embodiments, the Rate Distortion Optimization Unit 425 further performs a transformation operation and / or a quantization operation on the block as part of the encoding process. The Rate Distortion Optimization Unit 425 may further determine various mode selections, including, without limitation, block size and / or type selection, transformation size and / or type selection, and / or the like.

[0070] Reconstruction Unit 430 performs image reconstruction for the blocks contained in a media frame of the video stream, based on mode selection results received from Rate Distortion Optimization Unit 425. Reconstruction Unit 430 performs image reconstruction on frequency coefficients that were previously transformed and quantized during the encoding process. Reconstruction Unit 430 performs an inverse quantization function to reverse the quantization previously performed on the block. Reconstruction Unit 430 also performs an inverse transformation function to reverse the transformation previously performed on the block. In doing so, Reconstruction Unit 430 produces reconstructed residual data.The Reconstruction Unit 430 sums the reconstructed residual data with the winning candidate generated by the Rate Distortion Optimization Unit 425 to produce the Reconstructed Current Frame Block. The Reconstructed Current Frame Block is a proxy of the corresponding block of the media frame that a video decoder produces when the video stream generated by the video encoder 400 is decoded. In some embodiments, the Reconstruction Unit 430 can improve the visual quality of the video stream by performing a secondary fent block and / or size search for interframe encoding. The Reconstruction Unit 430 can also improve the visual quality of the video stream by performing an intraframe encoding mode search based on accurate neighbor pixel data.

[0071] Filter 435 performs one or more filtering techniques on the blocks contained within a media frame of the video stream. These techniques may include unblocking filtering, sample adaptive offset filtering, and / or similar methods. With unblocking filtering, Filter 435 improves the visual quality of the reconstructed current block of the media frame by smoothing the sharp edges resulting from the transformation and / or quantization performed by the Rate Distortion Optimization Unit 425 during encoding, followed by the inverse quantization and / or inverse transformation performed by the Reconstruction Unit 430 during reconstruction.With sample adaptive offset filtering, filter 435 further filters the reconstructed current block of the media frame by selectively adding offsets to the pixel values ​​of the reconstructed current block of the media frame based on the pixel value of a given pixel and / or the pixel values ​​of one or more neighboring pixels.

[0072] The entropy encoder 440 generates the final encoded bitstream for the video encoder 400 from the encoded blocks produced by the filter 435. In some embodiments, the entropy encoder 440 generates the final encoded bitstream, i.e., the output video stream, using a lossless compression technique. Additionally or alternatively, the entropy encoder 440 generates the final encoded bitstream using a lossless compression technique. The entropy encoder 440 encodes the blocks of a media frame sequentially in raster sampling order. The entropy encoder 440 waits for the final winning candidate data for each sequential block to be generated before encoding the bitstream for that block. In this way, the entropy encoder 440 encodes the blocks of each block row of the image sequentially and individually in raster sampling order.In raster sampling order, the entropy encoder 440 encodes blocks on each block row of the media frame from left to right and encodes the block rows of the media frame from top to bottom. Since the entropy encoder 440 completes the encoding of the blocks in each media frame, it stores the encoded blocks at a suitable position in the frame buffer memory via the frame buffer interface 480.

[0073] As described herein, various units and / or components generate block data, including, without limitation, interframe cost data, motion vector data, and / or the like, for each block of a media frame in a video stream. In some embodiments, the video encoder 400 generates such block data when media frames are encoded in certain formats, such as BH264, H.265, and / or the like. In such embodiments, the FPS unit 410, the SPS unit 415, the MCT unit 420, the MDP unit 465, the MPEB unit 466, the MPEC unit 467, and / or the like generate the block data for each media frame in the video stream. These units store the block data in a FIFO memory 468. After storing the block data in the FIFO memory 468, these units can proceed with encoding additional blocks without waiting for the controller 405 to read the block data for the current block.For example, after the video encoder 400 generates interframe and intraframe data for a first block of the current media frame and stores the data in the FIFO memory 468, the video encoder 400 can generate interframe and intraframe data for a second block of the current media frame while the interframe and intraframe data for the first block are stored in the FIFO memory 468. This can be accessed without pausing for the interframe and intraframe data for the first block. In this way, the execution of instructions by the controller 405 is decoupled from operations performed by other units of the video encoder 400. In some embodiments, any one or more other processors included in the computing system 100 can perform the operations associated with the FIFO memory 468, in addition to or as an alternative to the controller 405.

[0074] The units and / or components of the video encoder 400 can store block data for multiple blocks in the FIFO memory 468 before the controller 405 reads the block data from the FIFO memory 468. For example, and without limitation, the FIFO memory 468 can have a capacity to store block data for at least 64 blocks. When block data for a threshold number of blocks is stored in the FIFO memory 468, such as 32 blocks, the FIFO memory 468 notifies the controller 405 by means of a logic signal, a trigger instruction, an interrupt, and / or the like. In response, the controller 405 reads block data for the multiple blocks stored in the FIFO memory 468. The controller 405 can continue reading block data from the FIFO memory 468 until one or more threshold conditions are met.These threshold conditions may include the FIFO memory 468 no longer containing any stored block data, the controller 405 having read block data for a threshold number of blocks from the FIFO memory 468, and / or the like. While the controller 405 is reading block data from the FIFO memory 468, the units and / or components of the video encoder 400 may continue to store block data for additional blocks in the FIFO memory 468. The block data stored in the FIFO memory 468 may correspond to any type, size, and / or shape of blocks as described herein.

[0075] In some embodiments, the video encoder 400 generates such block data when media frames are encoded in certain formats, such as AV1, VP9, ​​and / or the like. In such embodiments, the FPS unit 410, the SPS unit 415, the MCT unit 420, the RDO unit 425, the reconstruction unit 430, the filter 435, the entropy encoder 440, and / or the like generate the block data for each media frame in the video stream. These units store the block data in a FIFO memory 441. After storing the block data in the FIFO memory 441, these units can proceed with encoding additional blocks without waiting for the controller 405 to read the block data for the current block.For example, after the video encoder 400 generates interframe and intraframe data for a first block of the current media frame and stores the data in the FIFO memory 441, the video encoder 400 can generate interframe and intraframe data for a second block of the current media frame while the interframe and intraframe data for the first block are stored in the FIFO memory 441. This can be accessed without stalling the interframe and intraframe data for the first block. In this way, the execution of instructions by the controller 405 is decoupled from operations performed by other units of the video encoder 400. In some embodiments, any one or more other processors included in the computing system 100 can perform the operations associated with the FIFO memory 441, in addition to or as an alternative to the controller 405.

[0076] The units and / or components of the video encoder 400 can store block data for multiple blocks in the FIFO memory 441 before the controller 405 and / or another processor contained in the computing system 100 reads the block data from the FIFO memory 441. For example, and without limitation, the FIFO memory 441 can have a capacity to store block data for at least 64 blocks. When block data for a threshold number of blocks is stored in the FIFO memory 441, such as 32 blocks, the FIFO memory 441 notifies the controller 405 by means of a logic signal, a trigger instruction, an interrupt, and / or the like. In response, the controller 405 reads block data for the multiple blocks stored in the FIFO memory 441. The controller 405 can continue reading block data from the FIFO memory 441 until one or more threshold conditions are met.These threshold conditions may include the FIFO memory 441 no longer containing any stored block data, the controller 405 having read block data for a threshold number of blocks from the FIFO memory 441, and / or the like. While the controller 405 is reading block data from the FIFO memory 441, the units and / or components of the video encoder 400 may continue to store block data for additional blocks in the FIFO memory 441. The block data stored in the FIFO memory 441 may correspond to any type, size, and / or shape of blocks as described herein.

[0077] The Video Encoder 400 contains various memory-related units and / or components, including, without limitation, DMA engines and cache memory 460. DMA engines, such as the PDMA unit 450, the RHINT unit 457, and the MEDMA unit 475, can perform block copies of data and / or instructions from one location in memory to another. More precisely, DMA engines can copy a block of data and / or instructions within a specific memory location or between one memory location and another. Therefore, DMA engines can copy a block of data and / or instructions within or between any one or more shared memory locations, PP memory 204, system memory 104, and / or the like.

[0078] Specifically, the PDMA unit 450 is a pixel DMA unit that loads original media frame data from memory. The PDMA unit 450 can buffer multiple blocks of the original media frame pixel data for motion estimation operations and for the MPEC unit 467. The PDMA unit 450 stores this original media frame data in local memory for access by the FPS unit 410 and / or other units and components of the video encoder 400. The RHINT unit 457 loads external motion vector hint data from memory. The RHINT unit 457 stores this external motion vector hint data in local memory for access by the FPS unit 410 and / or other units and components of the video encoder 400. The MEDMA unit 475 stores data generated by the MDP unit 465 in memory.In particular, the MEDMA unit 475 can store motion estimation result data, motion compensation result data, original pixel data and / or the like in a dedicated MEDMA buffer in memory.

[0079] Cache 460 can store short-term data and / or commands that have recently been accessed, or are predicted to be accessed soon, by various units and / or components of the video encoder 400. These units include, without limitation, the FPS unit 410, the SPS unit 415, the MCT unit 420, and / or the like. In particular, cache 460 can store reference pixels contained in reference media frames for the units of the video encoder 400. The data and / or commands stored in cache 460 may be copies of data and / or commands stored in other memory, including, without limitation, shared memory, PP memory 204, system memory 104, and / or the like.Typically, access times for loading data from and / or storing data into the cache memory 460 are lower than loading data from and / or storing data into these other stores.

[0080] The units and / or components of the video encoder 400 can access the frame buffer memory (not shown) via the frame buffer interface 480. The frame buffer memory can be a special memory for storing image data or it can be a section of another memory, including, without limitation, the PP memory 204, the system memory 104, and / or the like. In some embodiments, the frame buffer interface 480 can support data write operations from units of the video encoder 400 to the memory simultaneously with data read operations from the memory to the video encoder 400.

[0081] In some embodiments, the video encoder 400 can include feedback loops from a later stage to an earlier stage. For example, the visual quality of the output video stream can be improved by a feedback loop from the rate distortion optimization unit 425 to the FPS unit 410, and from the FPS unit 410 to the SPS unit 415 and the MCT unit 420. With such a feedback loop, the FPS unit 410, the SPS unit 415, and the MCT unit 420 can generate motion vector data for the current block in order to generate the motion vector for the next block. In this way, the video encoder 400 can generate a motion vector for the current block based on pixel data from the current block, as well as the motion vector from the previous block and / or the motion vector from the next block, resulting in improved motion estimation.This improved motion estimation can in turn lead to improved motion compensation.

[0082] Fig. Figure 5 illustrates a functional view of a video encoder 500, which outputs a media frame for the computer system 100. Fig. 1- Fig. 4. The Video Encoder 500 can encode a video stream compatible with the High Efficiency Video Coding (HEVC) standard, also known as the H.265 format or the Motion Picture Experts Group High Efficiency (MPEG-H) Part 2 format. Additionally or alternatively, the Video Encoder 500, as is and / or with minor modifications, can encode a video stream compatible with any other technically feasible video coding standard.

[0083] As shown, the 500 video encoder receives an input media frame to be encoded. This received media frame is designated as the current media frame (Fn ) 505. The current media frame (F n ) 505 and other media frames processed by the 500 video encoder are divided into multiple blocks. Each block contains a group of adjacent pixels, such as an 8x8 block of pixels, a 16x16 block of pixels, and / or the like. Each block is further subdivided into partitions, with each partition containing luminance pixels (lumapixels) and / or chrominance pixels (chromapixels). Lumapixels contain the luma or Y pixel values ​​for the pixels in the block. Chromapixels contain the chroma pixel values ​​for the pixels in the block. Chromapixel values ​​are typically color difference values ​​and can be of two types: (1) red color difference (U or C) r )-pixel values; and (2) blue color difference-(V or C) b )-pixel values.

[0084] The 500 video encoder also contains a reconstructed media frame based on the previously received and encoded media frame. This reconstructed media frame is known as the reference media frame (F'). n-1 ) 510. Based on the current pixels in the current media frame (F n ) 505 and on the reference pixels in the reference media frame (F' n-1The Motion Estimator (ME) 510 generates a motion vector for the current block, predicting the block of the current media frame from the corresponding block of the reference media frame. The reference media frame can be a previous media frame and / or a next media frame. The video encoder 500 can use the motion vector as an inter-frame candidate. The Motion Estimator 515 passes the inter-frame candidate to the Motion Compensation Unit (MC) 520. The Motion Compensation Unit 520 generates motion-compensated pixels for the inter-frame candidate. The Motion Compensation Unit 520 passes the motion-compensated pixels for the inter-frame candidate to the "Inter" input of the selector 525.

[0085] Additionally, the intraframe estimation unit selects 570 based on the current pixels in the current media frame (F n) 505 and on the adjacent pixels of the reconstructed current media frame uF' nThe data received by the summing unit 565 selects an intraframe prediction mode. In some embodiments, the intraframe estimating unit 570 can select the intraframe prediction mode that best predicts the pixels of the current block. The intraframe estimating unit 570 can select the intraframe prediction mode that results in the lowest rate distortion cost based on the sum of squared error (SSE) distortion for the current block, as determined by the rate distortion optimization unit of the selector 525. The number and type of available prediction modes can vary based on the block size. For example, the number and type of available prediction modes can differ among 4 × 4 pixel blocks, 8 × 8 pixel blocks, 16 × 16 pixel blocks, 32 × 32 pixel blocks, and / or the like.In this respect, the Intraframe Estimator 570 can select different prediction modes for each of the possible block sizes based on which prediction mode results in the lowest rate distortion cost, as determined by the rate distortion optimization unit for that block size. Furthermore, the Intraframe Estimator 570 can select different prediction modes for the luma values ​​in the block compared to the chroma samples in the block. In some embodiments, the prediction mode determines the order in which the pixels in the current block are sampled to generate the predicted intraframe candidate.For example, the prediction mode can specify vertical scanning, horizontal scanning, diagonal downward-left scanning, diagonal downward-right scanning, vertical left scanning, vertical right scanning, horizontal downward scanning, horizontal upward scanning and / or the like.

[0086] Based on the selected intraframe prediction mode, the intraframe prediction unit 575 generates an intraframe candidate. The intraframe prediction unit 575 samples the pixel values ​​in the current block in the order specified by the selected intraframe prediction mode. For each sampled pixel, the intraframe prediction unit 575 determines a predicted pixel value based on the differences between that pixel value and the pixel values ​​of neighboring pixels. From these predicted pixel values, the intraframe prediction unit 575 generates the intraframe candidate. The intraframe prediction unit 575 passes the intraframe candidate to the "Intra" input of the selector 525.

[0087] Selector 525 determines whether to select the compensated pixels for the inter-frame candidate received by the motion compensation unit 520 or the intra-frame candidate received by the intra-frame prediction unit 575. This determination of whether to select the inter-frame candidate or the intra-frame candidate can be made at any level of granularity, including, but not limited to, a block-by-block, media-frame-by-media-frame basis, and / or the like. The technique for determining whether to select the inter-frame candidate or the intra-frame candidate can be relatively simple or relatively complex. Typically, the more complex the technique used to determine whether to select the inter-frame candidate or the intra-frame candidate, the higher the video quality of the resulting encoded stream.The selected candidate between the interframe candidate and the intraframe candidate is called the winning candidate. In some embodiments, the selector 525 determines the winning candidate solely based on lumapixel values. In other embodiments, the selector 525 determines the winning candidate based on both lumapixel and chromapixel values. In general, basing the selection on both lumapixel and chromapixel values ​​can be more accurate and therefore result in higher visual quality than basing the selection on lumapixel values ​​alone.

[0088] In some embodiments, the selector 525 can also perform rate distortion optimization (RDO) when selecting the winning candidate. The rate distortion optimization unit (in Fig. (5 not shown) of selector 525 receives the inter-frame candidate and the intra-frame candidate. The rate distortion optimization unit further receives the reconstructed pixels of the reconstructed current media frame uF'. n , which are received by the inverse quantization unit 555, the inverse transformation unit 560, and the summing unit 565. Based on the reconstructed pixels of the reconstructed actual media frame uF' n The Rate Distortion Optimization Unit determines a rate distortion cost value based on the sum of squared errors (SSE) distortion for the current block. Selector 525 selects the winning candidate, at least partially, based on the rate distortion cost value for the current block, as determined by the Rate Distortion Optimization Unit. Selector 525 then passes the winning candidate to summer 530 and summer 565.

[0089] The summing unit 530 inverts the winning candidate received by the selector 525 before the winning candidate is combined with the current media frame (F n ) 505 is combined. As a result, the summator 530 determines the difference resulting from subtracting the winning candidate from the current media frame (F). n ) 505. This difference is called the residual pixels, residual data, or more generally, the residual D. n The summing mixer 530 carries over the remainder D. n to the transformation unit (T) 535.

[0090] The transformation unit 535 converts the remainder D nThe data received by the summing unit 530 is transformed into an array of frequency coefficients representing the image segment contained in each block. The transformation unit 535 transfers the frequency coefficients to the quantization unit (Q) 540. The quantization unit 540 reduces the total number of unique frequency coefficients received by the transformation unit 535 by quantizing the frequency coefficients according to defined frequency ranges or bins. The quantization unit 540 transfers the quantized frequency coefficients X to the reordering unit 545. The reordering unit 545 sorts the quantized frequency coefficients X in descending order of value, so that all coefficients with a value of zero (,0') are sorted to the end of the set of frequency coefficients.The reordering unit 545 transmits the sorted quantized frequency coefficients to the entropy encoder 550. The entropy encoder 550 generates the final encoded bitstream for the video encoder 500. In some embodiments, the entropy encoder 550 generates the final encoded bitstream, i.e., the output video stream, using a lossless compression technique. Additionally or alternatively, the entropy encoder 550 generates the final encoded bitstream using a lossless compression technique. The final encoded bitstream generated by the video encoder 500 can then be decoded by a suitable video decoder (not shown).

[0091] In addition to transferring the quantized frequency coefficients X to the reordering unit 545, the quantization unit 540 transfers the quantized frequency coefficients X to the inverse quantization unit (Q). -1) 555. The inverse quantization unit 555 performs an inverse quantization function to reverse the quantization performed by the quantization unit 540. The inverse quantization unit 555 transfers the inverse quantized frequency coefficients to the inverse transformation unit (T). -1 ) 560. The inverse transformation unit 560 performs an inverse transformation function to reverse the transformation performed by the transformation unit 535. In doing so, the inverse transformation unit 560 generates reconstructed residual data D. n The inverse transformation unit 560 transfers the reconstructed residual data D' n to the summing unit 565.

[0092] The summing unit 565 adds the reconstructed residual data D' n to the winning candidate generated by selector 525 to reconstruct the current media frame uF' nto generate. The reconstructed current media frame uF' n is a proxy of the media frame that a video decoder generates when the video stream generated by video encoder 500 is decoded. As described herein, summing encoder 565 transmits the reconstructed current media frame uF'. n to the intraframe estimation unit 570 to generate the intraframe candidate in conjunction with the intraframe prediction unit 575. Additionally, the summer 565 transmits the reconstructed current media frame uF'. n to filter 580. In some embodiments, filter 580 is an unblocking filter that improves the visual quality of the reconstructed actual media frame uF'. nImproved. Filter 580 improves visual quality by smoothing the sharp edges resulting from the transformation performed by the Transformation Unit 535 and / or the quantization performed by the Quantization Unit 540, followed by the inverse quantization performed by the Inverse Quantization Unit 555 and / or the inverse transformation performed by the Inverse Transformation Unit 560. Filter 580 passes the filtered image to the Sample Adaptive Offset (SAO) Filter 585. The Sample Adaptive Offset Filter 585 further filters the reconstructed actual media frame uF'. n by selectively adding offsets to the pixel values ​​of the reconstructed current media frame uF' nbased on the pixel value of a given pixel and / or the pixel values ​​of one or more neighboring pixels. The Sample Adaptive Offset Filter 585 stores the SAO-filtered image as the final reconstructed current media frame (F'). n ) 590.

[0093] After the video encoder 500 has finished processing the current media frame (F n Once 505 has completed, the video encoder 500 receives the next input media frame, which then becomes the new current media frame (F n ) 505 will be. Furthermore, the reconstructed current media frame (F') will be n ) 590 the new reference media frame (F' n-1 ) 510. The 500 video encoder uses this new reference media frame (F' n-1 ) 510, to select the inter-frame candidate for the new current media frame (F n ) to generate 505.

[0094] In some embodiments, the visual quality of the output video stream can be further improved by a feedback loop (not shown) from selector 525 to motion estimation unit 515. When selecting the winning candidate, selector 525 determines the final motion vector for the current block. Selector 525 transmits this final motion vector to motion estimation unit 515. Motion estimation unit 515 can use this final motion vector for the current block to generate the motion vector for the next block. In this way, motion estimation unit 515 can generate a motion vector for the current block based on pixel data from the current block, as well as the motion vector from the previous block and / or the motion vector from the next block, resulting in improved motion estimation.This improved motion estimation can in turn lead to improved motion compensation, as performed by the motion compensation unit 520, and improved selection accuracy, as performed by the selector 525.

[0095] In some embodiments, a given block can contain multiple subblocks or partitions. The subblocks can be of different sizes. For example, a 16 × 16 pixel block can contain 8 × 16 pixel subblocks, 16 × 8 pixel subblocks, 8 × 8 pixel subblocks, and / or the like in any combination. In such embodiments, the motion estimation unit 515 can generate a motion vector for each subblock and combine the motion vectors from the different subblocks to generate a final motion vector for the block.

[0096] In some embodiments, the 500 video encoder can be combined with the architecture of the 400 video encoder. Fig. 4 can be implemented. In such embodiments, media frames, including, without limitation, the current media frame (F), can be used. n ) 505, of the reference media frame (F' n-1 ) 510 and the reconstructed current media frame (F' n ) 590, in any technically feasible memory. In particular, these media frames can be stored in shared memory, cache memory 460, frame buffer memory, and / or the like. The motion estimation unit 515 and / or the motion compensation unit 520 can, without restriction, control the controller 405, the FPS unit 410, the PLC unit 415, and the MCT unit 420 of Fig. 4 represent. The Intraframe Estimation Unit 570 and / or the Intraframe Forecasting Unit 575 can, without restriction, control the Controller 405, the MPEB Unit 466 and the Progress Unit 470 of Fig. 4 represent. One or more of the selector 525 (including the rate distortion optimization unit of selector 525), the summer 530, the transformation unit 535 and / or the quantization unit 540 can, without restriction, control the rate distortion optimization unit 425 of Fig. 4 represent. The inverse quantization unit 555, the inverse transformation unit 560 and / or the summing unit 565 can, without restriction, represent the reconstruction unit 430 of Fig. 4 represent. The Filter 580 and / or the Sample Adaptive Offset Filter 585 can, without restriction, replace the Filter 435 of Fig. 4 represent. The entropy encoder 550 can, without restriction, replace the entropy encoder 440 of Fig. 4 represent.

[0097] It is understood that the system shown herein is for illustrative purposes only and that variations and modifications are possible. The techniques described herein can be performed by one or more alternative auxiliary processors, including, without limitation, CPUs, GPUs, video encoders, DMA units, IPUs, NPUs, TPUs, NNPs, DPUs, VPUs, ASICs, FPGAs, and / or the like, in any combination. More generally, the techniques described herein can be applied to any CPU 102, PPU 202, video encoder, and / or any other processing unit in any combination.

[0098] Fig. Figure 6 illustrates the storage of forward prediction hint data for a media frame generated by the video encoders 400 and 500 of Fig. 4- Fig. 5 is encoded according to various embodiments. As shown, the controller 405, in conjunction with other units and / or components of the video encoder 400, generates motion vector prediction data, also referred to as motion vector cue data or simply cue data, for each media frame. The controller 405 stores the cue data in a cue block line 600. The cue data for the cue block line 600 includes N cue blocks 670(0), 670(1), 670(2),..., 670(N-1), where each cue block 670 contains 16 forward-predicted cues. The cue block 650 is an expanded view of the cue block 670(0). Hint block 650 contains 16 forward-predicted hints, labelled L0, arranged as 4 lines 610(0), 610(1), 610(2), 610(3), with each line 610 containing 4 forward-predicted hints.In some examples, each hint block 670 represents a superblock of 64 × 64 pixels, and each hint represents a 16 × 16-pixel subblock within the superblock. The controller 405, in conjunction with other units and / or components of the video encoder 400, generates forward-predicted hints and stores the forward-predicted hints in the L0 slots of each hint block 670. In some embodiments, the controller 405 generates the forward-predicted hints block by block. In such embodiments, the controller generates and stores forward-predicted notes for note block 670(0) at the left end of note block line 600, followed by forward-predicted notes for note block 670(1), followed by forward-predicted notes for note block 670(2), and so on for note blocks ending with forward-predicted notes for note block 670(N-1).

[0099] After generating the cue blocks 670 for cue block line 600, the controller 405 generates forward-predicted cues for the other block lines from the top of the media frame to the bottom. Once the forward-predicted cue data is stored in all cue blocks 670 of cue block line 600, the WMVP unit 477 stores the forward-predicted cue data for cue block line 600 in memory. In some embodiments, the WMVP unit 477 stores the cue data line by line. In such embodiments, the WMVP unit 477 stores the cue data for line 660(0) from left to right, followed by the cue data for line 660(1), followed by the cue data for line 660(2), followed by the cue data for line 660(3).The RMVP unit 459 then loads the forward-predicted hint data for hint block line 600 into local memory for further processing, as described herein.

[0100] Fig. Figure 7 illustrates the storage of forward prediction hint data and backward prediction hint data for a media frame processed by the video encoders 400 and 500 of Fig. 4- Fig. 5 is encoded according to various embodiments. As shown, the controller 405, in conjunction with other units and / or components of the video encoder 400, generates motion vector prediction data, also referred to as motion vector cue data or simply cue data, for each media frame. The controller 405 stores the cue data in a cue block line 700. The cue data for the cue block line 700 includes N cue blocks 770(0), 770(1), 770(2),..., 770(N-1), where each cue block 770 contains 16 forward-predicted cues and 16 backward-predicted cues. The cue block 750 is an extended view of the cue block 770(0). The hint block 750 contains 16 forward-predicted hints, labelled L0, and 16 backward-predicted hints, labelled L1, arranged as 4 lines 710(0), 710(1), 710(2), 710(3).Each line 710 contains 4 forward-predicted hints and 4 backward-predicted hints. In some examples, each hint block 770 represents a superblock of 64 × 64 pixels, and each hint represents a 16 × 16-pixel subblock within the superblock. The controller 405, in conjunction with other units and / or components of the video encoder 400, generates forward-predicted hints and stores the forward-predicted hints in the L0 slots of each hint block 770. Furthermore, the controller 405, in conjunction with other units and / or components of the video encoder 400, generates backward-predicted hints and stores the backward-predicted hints in the L1 slots of each hint block 770. In some embodiments, the controller 405 generates the forward-predicted hints and the backward-predicted hints block by block.In such embodiments, the controller generates and stores forward-predicted hints and backward-predicted hints for hint block 770(0) at the left end of the hint block line 700, followed by forward-predicted hints and backward-predicted hints for hint block 770(1), followed by forward-predicted hints and backward-predicted hints for hint block 770(2), and so on for the hint blocks that end with forward-predicted hints and backward-predicted hints for hint block 770(N-1).

[0101] After generating the cue blocks 770 for cue block line 700, the controller 405 generates forward-predicted cues and backward-predicted cues for the other block lines from the top of the media frame to the bottom. Once the forward-predicted cue data and the backward-predicted cue data are stored in all cue blocks 770 of cue block line 700, the WMVP unit 477 stores the forward-predicted cue data and the backward-predicted cue data for cue block line 700 in memory. In some embodiments, the WMVP unit 477 stores the cue data line by line. In such embodiments, the WMVP unit 477 stores the cue data for line 760(0) from left to right, followed by the cue data for line 760(1), followed by the cue data for line 760(2), followed by the cue data for line 760(3).The RMVP unit 459 then loads the forward-predicted and backward-predicted cue data for cue block line 700 into local memory for further processing, as described herein. Generating, storing, and subsequently loading both forward-predicted and backward-predicted cue data can result in improved visual quality of the encoded video stream compared to the approach that uses only forward-predicted cue data, as described in conjunction with [reference missing]. Fig. 6 described.

[0102] It is understood that the system shown herein is for illustrative purposes only and that variations and modifications are possible. As in Fig. 6 and Fig. As shown in Figure 7, each clue in the clue data represents a 16 × 16 pixel subblock, and each block is a 64 × 64 pixel superblock. As a result, the clue data for each superblock contains 16 clues or clue pairs. In this context, the clue data for each block in Fig. 6 16 forward forecast hints. Likewise, the hint data for each block includes Fig. 7 16 forward-prediction hints and 16 backward-prediction hints. However, the disclosed techniques can be applied to any type, size, and / or shape of blocks as described herein. Furthermore, each hint in the hint data can be for any size and / or shape of a pixel subblock. As a result, each block can correspond to any number of hints, where the number of hints per block is equal to the size of the block divided by the size of the subblock for a hint.

[0103] Fig. Figure 8 is a flowchart of process steps for generating statistical data of a coding unit tree for a media frame by the computing system 100. Fig. 1- Fig. 7, according to various embodiments. Additionally or alternatively, the process steps can be performed by one or more alternative auxiliary processors, including, without limitation, microcontrollers, RISC processors, CPUs, GPUs, DMA units, IPUs, NPUs, TPUs, NNPs, DPUs, VPUs, ASICs, FPGAs and / or the like, in any combination. Although the process steps in conjunction with the systems of Fig. 1- Fig. As described in section 7, experts in the field will recognize that any system configured to perform the procedural steps in any order is within the scope of this disclosure.

[0104] As shown, a procedure 800 begins at step 802, where a video encoder, such as video encoder 400 and / or video encoder 500, receives a block of a current media frame from a video stream. The video encoder can receive the block of the current media frame via a DMA engine, which loads original media frame data from memory and stores this original media frame data in local memory for access by units and / or components of the video encoder. Furthermore, the DMA engine can buffer multiple blocks of the original media frame pixel data for access by units and / or components of the video encoder.

[0105] In step 804, the video encoder generates interframe data based on the block of the current media frame and a matching block from a previous and / or next media frame. Various units within the video encoder perform motion estimation, including full-pixel and subpixel motion estimation, and motion compensation to generate interframe data. The interframe data may include an interframe candidate and interframe cost data.

[0106] The video encoder performs motion estimation by searching a reference media frame, such as the previous and / or the next media frame, for blocks that match corresponding blocks in the current media frame being encoded. A block in the reference frame matches the block in the current frame if the pixel data of the block in the reference frame is equal to or similar to the pixel data of the corresponding block in the current media frame. The matching block is the block in the reference frame with pixel data that is closest to the pixel data of the block in the current frame. If the objects in the scene have not moved and the camera view has not changed between the reference media frame and the current media frame, then the position of the corresponding block within the current frame may be the same as the position of the corresponding block in the reference frame.However, if the objects in the scene have moved and / or the camera view has changed between the reference media frame and the current media frame, then the position of the current block within the current frame may differ from the position of the corresponding block in the reference frame. The video encoder generates a motion vector for the current block in the current media frame, which identifies the position of the corresponding block in the reference frame.

[0107] Motion compensation predicts the pixels of a current media frame based on a previous and / or the next media frame by determining effects caused by movement of the camera recording the video stream and / or movement of objects within the scene being captured by the camera. The video encoder generates difference data, also known as residual data, which specifies the differences between the pixel data of the matching block in the reference frame and the pixel data of the current block in the current media frame. As the similarity between the pixel data of the block in the reference frame and the pixel data of the current block increases, the amount of difference data decreases, resulting in lower inter-frame costs.Conversely, if the similarity of the pixel data of the block in the reference frame to the pixel data of the current block decreases, the amount of difference data increases, leading to high inter-frame costs.

[0108] In step 806, the video encoder generates intraframe data based on the block of the current media frame and neighboring pixels of that block. Various units within the video encoder perform intraframe estimation and prediction to generate this data. The intraframe data may include an intraframe candidate and intraframe cost data.

[0109] In step 808, the video encoder determines whether the FIFO memory has sufficient capacity to store the interframe and intraframe data. If the FIFO memory does not have sufficient capacity, procedure 800 returns to step 808 until the video encoder determines that the FIFO memory has sufficient capacity. However, if the FIFO memory does have sufficient capacity to store the interframe and intraframe data, the video encoder proceeds to step 810, where it stores the interframe and intraframe data in the FIFO memory.

[0110] In step 812, the video encoder determines whether the amount of data stored in the FIFO buffer has reached a threshold. The threshold can be set to reduce or eliminate the likelihood that the video encoder will wait in step 808 for the FIFO buffer to have sufficient capacity. For example, and without limitation, the FIFO buffer may have a capacity to store block data for at least 64 blocks, and the threshold may be set to a capacity to store block data for 32 blocks. In such an example, the amount of data stored in the FIFO buffer has reached the threshold when at least half of the FIFO buffer is storing interframe and / or intraframe data.If the amount of data stored in the FIFO memory has not reached the threshold, then procedure 800 returns to step 802 described above to receive additional blocks of the current media frame.

[0111] When the amount of data stored in the FIFO memory reaches the threshold, procedure 800 proceeds to step 814, where the video encoder sends a notification to a controller contained within the video encoder and / or another processor. The notification indicates that the amount of data stored in the FIFO memory has reached the threshold. The video encoder can notify the controller and / or another processor by a logic signal, a trigger instruction, an interrupt, and / or the like. Procedure 800 then returns to step 802, as described above, to receive additional blocks of the current media frame.

[0112] In response to receiving the notification, the controller and / or another processor reads block data for the multiple blocks stored in the FIFO memory. The controller and / or another processor can continue reading block data from the FIFO memory until one or more threshold conditions are met. These threshold conditions may include the FIFO memory no longer containing any stored block data, the controller and / or another processor having read block data for a threshold number of blocks from the FIFO memory, and / or the like. While the controller and / or another processor is reading block data from the FIFO memory, the video encoder units and / or components can continue storing block data for additional blocks in the FIFO memory.

[0113] Fig. Figure 9 is a flowchart of process steps for generating and storing hint data for a media frame by the computer system 100. Fig. 1- Fig. 7, according to various embodiments. Additionally or alternatively, the process steps can be performed by one or more alternative auxiliary processors, including, without limitation, microcontrollers, RISC processors, CPUs, GPUs, DMA units, IPUs, NPUs, TPUs, NNPs, DPUs, VPUs, ASICs, FPGAs and / or the like, in any combination. Although the process steps in conjunction with the systems of Fig. 1- Fig. As described in section 7, experts in the field will recognize that any system configured to perform the procedural steps in any order is within the scope of this disclosure.

[0114] As shown, a procedure 900 begins at step 902, where a video encoder, such as video encoder 400 and / or video encoder 500, retrieves block data for multiple blocks of a current media frame from a media stream. The video encoder can receive the block data of the current media frame via a DMA engine, which loads original media frame data from memory and stores this original media frame data in local memory for access by units and / or components of the video encoder. Furthermore, the DMA engine can buffer multiple blocks of the original media frame pixel data for access by units and / or components of the video encoder.

[0115] In step 904, the video encoder generates a forward prediction based on block data for a current block of the current media frame and block data for a matching block of the previous media frame. In step 906, the video encoder generates a backward prediction based on block data for the current block of the current media frame and block data for a matching block of the next media frame. The matching block is the block in the reference frame with pixel data that is closest to the pixel data of the block in the current frame. The reference media frame can be the previous media frame and / or the next media frame. If the objects in the scene have not moved and the camera view has not changed between the reference media frame and the current media frame, then the position of the current block within the current frame can be the same as the position of the matching block in the reference frame.However, if objects in the scene have moved and / or the camera view has changed between the reference media frame and the current media frame, then the position of the current block within the current frame may differ from the position of the corresponding block in the reference frame. Motion compensation predicts the pixels of a current media frame based on the reference media frame by determining effects caused by movement of the camera recording the video stream and / or movement of objects within the scene being recorded by the camera. The video encoder generates a motion vector for the current block that predicts the block of the current media frame from the corresponding block in the reference media frame. The video encoder generates motion vector prediction data, also known as motion vector hint data or simply hint data, for each media frame.This guidance data includes forward-prediction guidance data and backward-prediction guidance data. The video encoder can use the motion vector data as an inter-frame candidate. Based on the motion vector data, the video encoder generates motion-compensated pixels for the inter-frame candidate.

[0116] In step 908, the video encoder generates a predicted importance score for the current block based on forward and backward prediction. In step 910, the video encoder selects a winning candidate based on the predicted importance score. The video encoder performs rate distortion optimization on the blocks contained within a media frame of the video stream. The video encoder selects a winning candidate for a block from among the inter-frame candidate for that block and the intra-frame candidate for that block. The selection of the winning candidate may be based on the inter-frame cost score associated with the inter-frame candidate. The inter-frame cost score may be partially based on the predicted importance score generated from the forward and backward prediction clue data.Similarly, the selection of the winning candidate can be based on the intra-frame cost value associated with the intra-frame candidate.

[0117] In step 912, the video encoder encodes the current block based on the winning candidate. The video encoder includes an entropy encoder that generates the final encoded bitstream for the video encoder. In some embodiments, the entropy encoder generates the final encoded bitstream, i.e., the output video stream, using a lossless compression technique. Additionally or alternatively, the entropy encoder generates the final encoded bitstream using a lossless compression technique.

[0118] In step 914, the video encoder sets (determines) the next block as the current block. This allows the video encoder to encode the next block and any other remaining blocks of the current media frame sequentially. In step 916, the video encoder determines whether sufficient data is stored in memory to encode the new current block. For example, the video encoder determines whether the block data buffered by the DMA engine is sufficient to process the new current frame. If sufficient data is stored in memory to encode the new current block, then procedure 900 returns to step 904 to encode the new current block. However, if insufficient data is stored in memory to encode the new current block, then procedure 900 returns to step 902 to retrieve additional block data before encoding the new current block.

[0119] Overall, a video encoder includes a first-in-first-out (FIFO) memory that stores intraframe cost data and / or interframe cost data for multiple blocks contained within a media frame of a video stream. Various units within the video encoder perform motion estimation, including full-pixel motion estimation and subpixel motion estimation, and motion compensation to generate interframe data, including an interframe candidate and interframe cost data. Interframe data, interframe candidates, and interframe cost data are also referred to herein as interdata, intercandidates, and intercost data, respectively.Various units within the video encoder generate motion vector data, including forward motion vector prediction data based on matching blocks in one or more reference frames, including matching blocks in a previous media frame, and backward motion vector prediction data based on matching blocks in a subsequent media frame. These units search for intercandidate blocks in the reference frame(s) and determine an intercost value for each intercandidate block. The intercost value of an intercandidate block is inversely proportional to the degree to which the intercandidate block matches the current block of the current media frame. The winning intercandidate block is the one with the lowest intercost value.

[0120] Furthermore, various units within the video encoder perform intraframe estimation and prediction to generate intraframe data, including an intraframe candidate and intraframe cost data. Intraframe data, intraframe candidates, and intraframe cost data are also referred to herein as intradata, intracandidates, and intracost data, respectively. These units within the video encoder search for intracandidate blocks in the current media frame that are adjacent to the current block and determine an intracost value for each intracandidate block. The intracost value of an intracandidate block is inversely proportional to the degree to which the intracandidate block matches the current block of the current media frame. The winning intracandidate block is the intracandidate block with the lowest intracost value.The video encoder stores the interframe and intraframe data for the blocks in the FIFO memory, thus avoiding stalling of the units in the video encoder until the interframe and intraframe data for each block are retrieved. After the video encoder generates the interframe and intraframe data for the first block of the current media frame and stores the data in the FIFO memory, it can generate the interframe and intraframe data for a second block of the current media frame while the interframe and intraframe data for the first block are still stored in the FIFO memory. This access to the interframe and intraframe data for the first block is possible without stalling.

[0121] After interframe and intraframe data for several blocks have been stored in the FIFO memory, various other units in the video encoder retrieve the interframe and intraframe data stored in the FIFO memory. These other units perform specific functions, including selecting between the interframe and intraframe candidates, generating motion vector data, filtering the selected candidate, and performing entropy coding to generate bits for the output encoded video stream.

[0122] At least one technical advantage of the disclosed techniques over the prior art is that, with the disclosed techniques, units within the video encoder are not paused (also known as gestalt, blocked, or delayed) for each block in a media frame until the interframe and intraframe data for each block are retrieved. Instead, units within the video encoder can retrieve and process the interframe and intraframe data for multiple blocks simultaneously, resulting in improved efficiency and performance compared to conventional approaches.Another advantage of the disclosed techniques is that, since the video encoder can retrieve data for multiple blocks across multiple media frames, it can encode a current block and determine the predicted importance of that block by accessing interframe data for matching blocks in a previous and a subsequent media frame. As a result, the video encoder can generate both forward-direction motion vector data based on data from the previous media frame and reverse-direction motion vector data based on data from the subsequent media frame. These advantages represent one or more technological improvements over prior art approaches.

[0123] Any and all combinations of any of the claim elements mentioned in any of the claims and / or any elements described in this application, in any way, fall within the intended scope of this disclosure and protection.

[0124] The descriptions of the various embodiments are presented for illustrative purposes only and are not intended to be exhaustive or limited to the disclosed embodiments. Many modifications and variations will be obvious to those skilled in the art without deviating from the scope and spirit of the described embodiments.

[0125] Aspects of the present embodiments may be embodied as a system, method, or computer program product. Accordingly, aspects of the present disclosure may take the form of a fully hardware embodiment, a fully software embodiment (including firmware, resident software, microcode, etc.), or an embodiment that combines software and hardware aspects, all of which may generally be referred to herein as a “module” or “system.”

[0126] Any combination of one or more computer-readable media can be used. The computer-readable medium can be a computer-readable signaling medium or a computer-readable storage medium. A computer-readable storage medium can be, for example, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system; an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor device; or any suitable combination of the foregoing.More specific examples (a non-exhaustive list) of computer-readable storage media would include the following: an electrical connection with one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.For the purposes of this document, a computer-readable storage medium can be any tangible medium that can contain or store a program for use by or in conjunction with an instruction execution system, instruction execution device, or instruction execution device.

[0127] Aspects of the present disclosure are described above with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the disclosure. It is understood that each block of the flowcharts and / or block diagrams, and combinations of blocks in the flowcharts and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device to create a machine such that the instructions executed by the processor of the computer or other programmable data processing device enable the implementation of the functions / steps specified in the block or blocks of the flowcharts and / or block diagrams.Such processors can be, without limitation, general-purpose processors, special-purpose processors, application-specific processors, or field-programmable gate arrays.

[0128] The flowcharts and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, procedures, and computer program products according to various embodiments of the present disclosure. In this respect, each block in the flowcharts or block diagrams can represent a module, segment, or section of code comprising one or more executable instructions for implementing the specified logical function(s). It should also be noted that in some alternative implementations, the functions specified in the block may occur out of the order shown in the figures. For example, two blocks shown consecutively may actually be executed essentially simultaneously, or the blocks may sometimes be executed in reverse order, depending on the functionality involved.It is also noted that each block of the block diagrams and / or flowcharts and combinations of blocks in the block diagrams and / or flowcharts can be implemented by specialized hardware-based systems that perform the specified functions or steps, or combinations of specialized hardware and computer instructions.

[0129] While the foregoing is directed to embodiments of the present disclosure, other and further embodiments of the disclosure may be designed without deviating from its basic scope, and the scope thereof is determined by the following claims.

Claims

[1] Computer-implemented method for generating statistical data for a current media frame, wherein the method comprises: Generating initial internet data for a first block of the current media frame; Generating initial intradata for the first block of the current media frame; Storing the first inter-data and the first intra-data in a first-in, first-out (FIFO) storage system; and Generating a second interdata and second intradata for a second block of the current media frame, while the first interdata and first intradata are stored in the FIFO memory, and without pausing for the first interdata and first intradata to be accessed. [2] Computer-implemented method according to claim 1, further comprising: Determine that a certain amount of data stored in the FIFO memory has reached a threshold; and In response, a notification is transmitted to a controller indicating that the amount of data stored in the FIFO memory has reached the threshold. [3] Computer-implemented method according to claim 2, wherein the notification comprises at least one logic signal, a trigger instruction or an interrupt. [4] Computer-implemented method according to claim 2 or 3, wherein, in response to receiving the notification, the controller reads block data for a plurality of blocks, including the first block, from the FIFO memory. [5] Computer-implemented method according to claim 4, wherein, while the controller reads block data for the plurality of blocks from the FIFO memory, the second inter-data and the second intra-data are stored in the FIFO memory. [6] Computer-implemented method according to claim 4 or 5, wherein the controller continues to read block data until at least one of the FIFO memory no longer has any stored block data or the controller has read block data for a threshold number of blocks from the FIFO memory. [7] Computer-implemented method according to any one of the preceding claims, wherein: Generating the first internet data for the first block includes: Searching for candidate interblocks in a reference media frame to determine a candidate intercost value for each candidate interblock relative to the first block; Setting a first intercost value as the lowest candidate intercost value; and Setting a first intercandidate block as the candidate interblock that corresponds to the first intercost value, where the first interdata comprise the first intercandidate block and the first intercost value, and Generating the first intranet data for the first block includes: Searching for candidate intrablocks in the current media frame to determine a candidate intracost value for each candidate intrablock relative to the first block; Setting a first intracost value as the lowest candidate intracost value; and Setting a first intracandidate block as the candidate intrablock that corresponds to the first intracost value, where the first intra-data includes the first intra-candidate block and the first intra-cost value. [8] Computer-implemented method according to claim 7, further comprising: Selecting a winning candidate from the first intercandidate block and the first intracandidate block based on the first intercost value and the first intracost value; and Coding the first block by coding the winning candidate. [9] Computer-implemented method according to claim 8, wherein the first block is encoded according to the H.264 format, the H.265 format, the Video Compression 9 (VP9) format, the Alliance for Open Media (AOMedia) Video 1 (AV1) format or the Versatile Video Coding (VVC) format. [10] Computer-implemented method according to any one of the preceding claims, further comprising: Generating a forward prediction based on block data for the first block and on block data for a second block of a previous media frame; Storing the forward prediction as forward guidance data in memory; Generating a backward prediction based on block data for the first block of the current media frame and on block data for a third block of the next media frame; and Storing the backward prediction as backward hint data in memory. [11] Computer-implemented method according to claim 10, further comprising: Generating a predicted importance score based on the forward hint data and the backward hint data; and encoding the first block based at least partially on the predicted importance score. [12] System, encompassing: a first-in-first-out (FIFO) storage system; and a video encoder configured to: Generating initial internet data for a first block of a current media frame; Generating initial intradata for the first block of the current media frame; Storing the first inter-data and the first intra-data in the FIFO memory; and Generating a second interdata and second intradata for a second block of the current media frame, while the first interdata and first intradata are stored in the FIFO memory, and without pausing for the first interdata and first intradata to be accessed. [13] System according to claim 12, wherein the video encoder is further configured to: Determine that a certain amount of data stored in the FIFO memory has reached a threshold; and as a response, transmitting a notification to a controller indicating that the amount of data stored in the FIFO memory has reached the threshold, wherein the notification includes at least one logic signal, a trigger instruction or an interrupt, In response to receiving the notification, the controller reads block data for a large number of blocks, including the first block, from the FIFO memory. [14] System according to claim 13, wherein, while the controller reads block data for the plurality of blocks from the FIFO memory, the video encoder further stores the second interdata and the second intradata in the FIFO memory. [15] System according to claim 14, wherein the controller continues to read block data until at least one of the FIFO memory no longer has any stored block data or the controller has read block data for a threshold number of blocks from the FIFO memory. [16] System according to any one of claims 12 to 15, wherein: to generate the first internet data for the first block, the video encoder is configured to: Searching for candidate interblocks in a reference media frame to determine a candidate intercost value for each candidate interblock relative to the first block; Setting a first intercost value as the lowest candidate intercost value; and Setting a first intercandidate block as the candidate interblock that corresponds to the first intercost value, where the first interdata comprise the first intercandidate block and the first intercost value, and to generate the first intranet data for the first block, the video encoder is configured to: Searching for candidate intrablocks in the current media frame to determine a candidate intracost value for each candidate intrablock relative to the first block; Setting a first intracost value as the lowest candidate intracost value; and Setting a first intracandidate block as the candidate intrablock that corresponds to the first intracost value, where the first intra-data includes the first intra-candidate block and the first intra-cost value. [17] System according to claim 16, wherein the video encoder is further configured to: selects a winning candidate from the first intercandidate block and the first intracandidate block based on the first intercost value and the first intracost value; and Coding the first block by coding the winning candidate. [18] System according to claim 17, wherein the first block is encoded according to the H.264 format, the H.265 format, the Video Compression 9 (VP9) format, the Alliance for Open Media (AOMedia) Video 1 (AV1) format or the Versatile Video Coding (VVC) format. [19] System according to any one of claims 12 to 18, wherein the video encoder is further configured to: Generating a forward prediction based on block data for the first block and on block data for a second block of a previous media frame; Storing the forward prediction as forward guidance data in memory; Generating a backward prediction based on block data for the first block of the current media frame and on block data for a third block of the next media frame; and Storing the backward prediction as backward hint data in memory. [20] System according to claim 19, wherein the video encoder is further configured to: Generating a predicted importance score based on forward and backward guidance data; and Encoding the first block based at least partially on the predicted importance value.

Images