Hardware codec accelerator for high performance video encoding

Abstract

Apparatuses, systems, and techniques for real-time codec encoding of video files using a hardware-assisted accelerator that leverages a combination of parallel and sequential processing, in which at least part of intra block prediction is performed with parallel processing are disclosed.

Description

Technical Field

[0001] At least one embodiment relates to computational techniques for performing and facilitating efficient compression of video files. For example, at least one embodiment relates to utilizing hardware video codec accelerators for efficient, real-time encoding of high-definition video files using modern video coding standards that, while providing numerous tools and options for improving video quality, do so at the expense of increased computational complexity. Background Art

[0002] Video files in raw (source) pixel format can take up very large amounts of memory and require significant network bandwidth, making them impractical for storage and / or live streaming. For example, a typical HD video display has approximately 30 frames per second, which are typically very similar to each other. Much of the information in these frames is necessarily redundant, allowing for efficient compression. On the one hand, relatively small changes occur between subsequent frames (temporal redundancy). On the other hand, various regions within a given frame are often similar to each other (spatial redundancy); for example, an image of the sky can extend over a large portion of the frame. Therefore, instead of transmitting the actual pixel information (luminance and chrominance) for each pixel, the codec can identify reference blocks similar to the block being encoded ("predicted") and provide the decoder with a suitable and compact mathematical representation of the difference ("delta") between the actual source block and the predicted block. The reference block can be a block from a different frame (e.g., the previous or even the next frame), a block from the same frame, or even a synthetic block generated according to some predetermined scheme (pattern) based on a small number of reference pixels. Then, instead of storing or live streaming actual frames of pixels, the codec can output a bitstream of encoded data, which primarily contains instructions to the decoder on how to generate an approximation of the frame whose visual appearance is indistinguishable from or very similar to the source frame. BRIEF DESCRIPTION OF THE DRAWINGS

[0003] Figure 1A is a schematic block diagram of an example codec accelerator system for efficient encoding operations according to at least some embodiments;

[0004] Figure 1B According to at least some embodiments Figure 1A A schematic block diagram of a motion estimation functional group of an example codec accelerator system;

[0005] Figure 2 is a schematic block diagram of a rate-distortion optimization functional group according to at least some embodiments;

[0006] Figure 3 is a schematic diagram illustrating the operation of initial intra-frame sequential processing and parallel processing of a codec accelerator system for efficient encoding operations according to at least some embodiments;

[0007] Figure 4 is a schematic diagram illustrating the operation of final intra-frame processing of a codec accelerator system for efficient encoding operations according to at least some embodiments;

[0008] Figure 5A is a flow chart of an example method for real-time compression encoding of a video file using hardware parallel processing according to at least some embodiments;

[0009] Figure 5B is a flow chart of an example method for reconstruction of a predicted PB for efficient real-time compression encoding of a video file in accordance with at least some embodiments;

[0010] Figure 6 is a flow chart of another example method for real-time compression encoding of a video file using hardware parallel processing, according to at least some embodiments. DETAILED DESCRIPTION

[0011] High-performance codecs such as H.264 (Advanced Video Coding, AVC), H.265 (High Efficiency Video Coding, HEVC), and VP9 allow for efficient coding based on finding similar blocks in the same frame (called intra prediction) and / or different frames (called inter prediction). The search for intra prediction blocks can be performed along multiple spatial directions (e.g., 8 directions in AVC and 33 directions in HEVC). Similarly, the search for inter prediction blocks can be performed with sub-pixel precision, where translation vectors (rotations, affine transformations, etc.), referred to herein as motion vectors, specify the relative displacement of similar blocks in different frames. Blocks of multiple sizes can be processed to identify the best prediction, for example (in HEVC) 64×64 pixel blocks, 32×32 pixel blocks, and 16×16 pixel blocks. Multiple block sizes allow for more efficient coding (larger portions of one or more images that are similar to each other) and finer resolution (portions of one or more images that contain unique details). The VP9 standard allows for more variations in block sizes, including 8×8 square blocks, 64×32 rectangular blocks, 32×64, 32×16, 16×32, 16×8, and 8×16 pixels. Newer codec standards, such as AOMedia Video1 (AV1), allow for more prediction options. For example, while AV1 inherits the block sizes of VP9, ​​it allows for 58 spatial directions for intra-frame prediction. In addition, AV1 expands the types of mathematical transformations that can be used to encode residual blocks, as well as the number and types of filters that can be used to mitigate block boundary artifacts, noise artifacts, motion vector artifacts, and more.

[0012] The increase in the number of available prediction, encoding, and filtering options has led to higher quality video encoding, but at the expense of increased computational complexity and processor utilization. While the computational performance is acceptable for offline applications, real-time encoding (e.g., live streaming of high-definition video) requires improvements to existing technologies.

[0013] Conventional AV1 (or other codec) encoding software can process various superblocks (e.g., 64x64 or 128x128 pixel blocks) sequentially. For example, starting with the top-left 64x64 superblock of the current frame, the software can partition the superblock into all possible partitions (approved by the codec standard). A given partition of the superblock can contain blocks of various sizes. For example, the top-left quadrant of the superblock can be represented as a single 32x32 block, the top-right quadrant can be partitioned into 16 blocks of 8x8 pixels each, the bottom-left quadrant can be partitioned into two blocks of 32x16 pixels, and the bottom-right quadrant can be partitioned into eight blocks of 16x8 pixels. (As described below, many other partitioning schemes have also been selected and evaluated.) Each block in the selected partitioning scheme (regardless of size) can then be approximated (predicted) using inter- and intra-frame prediction.

[0014] During inter-frame prediction, the block most similar to the current block can be identified in one or more reference frames (previous and / or next frames). Frames that are closely spaced (in time) are likely to be similar to each other, with objects experiencing relatively small displacements. Therefore, it is usually sufficient to explore relatively close neighboring blocks of the current frame block in one or more reference frames. The best inter-frame prediction candidate block can be selected by minimizing the error value of a cost function that characterizes the difference between the pixels of the inter-frame prediction candidate block and the source pixels of the current block.

[0015] During intra prediction, an approximation can be made to a block based on its reference pixels. Typically, the reference pixels are boundary pixels of adjacent blocks (e.g., pixels close to the top and left boundaries of the current block). The pixels of the current block are inferred from the reference pixels by extrapolation (e.g., from the top row or left column of the reference pixels) or interpolation (from the top row and left column) along various directions recognized by the codec standard. The extrapolation or interpolation of the current block pixels is performed according to specific formulas that depend on the direction (mode) and are defined as part of the codec standard. Similar to inter prediction, one or more best intra prediction candidates can be selected by minimizing the error value of the cost function and based on how close each intra prediction candidate is to the source pixels of the current block.

[0016] The best inter-prediction candidate block and the best intra-prediction candidate block (generated using the corresponding winning intra-prediction mode) are then compared to each other to select the final winning candidate block. The resulting final winning block typically has pixels that differ from the source pixels of the current block. Subtracting the pixel values ​​of the prediction block from the source block yields the residual block. Even though the residual block may have a variety of pixel values ​​distributed throughout the entire area of ​​the block, this variation is typically smooth. This means that the discrete Fourier transform (or some other discrete transform) has components concentrated near the long-wavelength end of the transform. The short-wavelength end of the transform is less noticeable to the human eye and can be truncated (using a so-called quantized transform), significantly reducing the number of (Fourier) coefficients required to encode the residual block. These coefficients form the basis of the encoded codec bitstream (subject to additional filtering and entropy coding).

[0017] Certain pixels in the prediction block are used as reference pixels for the prediction of the next block, such as the neighboring block to the right of the current block and the neighboring block below the current block. Since the decoder receives truncated Fourier coefficients (instead of the exact residual block), these reference pixels must also be modified (reconstructed) on the encoder side. Specifically, the application first performs an inverse quantization transform, then an inverse Fourier transform on the truncated distribution, and the resulting reconstructed residual block is added to the prediction block to obtain a reconstructed block. The reconstructed block is the same pixel block that the decoder subsequently generates (and presents to the viewer). Accordingly, the encoder uses the boundary pixels of the reconstructed block as (reconstructed) reference pixels for approximating (predicting) subsequent blocks. The described process continues until all blocks of the image are predicted and reconstructed.

[0018] The described process has multiple interdependencies. Superblock reconstruction is performed one after the other, with the next superblock being processed after the previous one has been predicted and reconstructed. Furthermore, similar predictions and reconstructions must be performed for the individual blocks (which may be of different sizes) within each superblock. For each superblock, there may be a large number of block partitions. Furthermore, within each partition, there may be a large number of spatial intra prediction modes (directions of interpolation). Such dependencies make deployment of conventional AV1 (and other computationally expensive) codec embodiments problematic for real-time encoding and streaming applications.

[0019] Aspects and embodiments of the present disclosure address these and other technical challenges of real-time video encoding in modern advanced codecs by reducing the dependencies encountered during the encoding process. A novel hardware architecture for efficient codec operation is also disclosed. In certain embodiments, hardware acceleration of encoding is achieved by performing sequential processing on consecutive blocks of the current frame while restricting the size of the blocks to a specific subset of identified block sizes (e.g., a specific (first) block size). The same processing as above can be performed on blocks of the first size, and a set of reconstructed blocks can be obtained. The reconstructed blocks can then be used as a reference pixel source for processing of all other block sizes, this time in parallel with each other. More specifically, each partition of each super block (e.g., a 64×64 pixel block) can be processed by separate processing threads, concurrently with other super blocks. This parallel processing can determine the optimal partitioning of each super block into smaller blocks, the best inter-frame prediction candidate for each block, and one or more best candidate intra-frame prediction interpolation modes for each block.

[0020] Subsequent parallel compression of the remaining superblocks would generate codec data that is undecodable because a decoder operating according to existing codec standards would reconstruct the blocks sequentially. To address this issue, after parallel processing, a codec accelerator can perform a final sequential reconstruction of the identified best blocks using a reduced number of identified best candidate modes. Advantages of the disclosed embodiments over the prior art include a significant speedup of the encoding process. Both the initial and final sequential processing stages can be performed much faster than conventional methods: the initial sequential processing is limited to a much smaller number of fixed block sizes (e.g., one), while the final sequential processing is limited to blocks of a determined optimal size and a reduced number of best modes for each block.

[0021] System Architecture

[0022] Figure 1Ais a schematic block diagram of an example codec accelerator system 100 for efficient encoding operations according to at least some embodiments. Codec accelerator system 100 can be part of or in communication with a host computer device 102, which can be a desktop computer, a laptop computer, a smartphone, a tablet computer, a local server, a cloud server, a dedicated video processing server, a collection of multiple computing devices, a distributed computing system, a smart television, an augmented reality device, or any other suitable computing device (or collection of computing devices) capable of executing the techniques described herein. Host computer device 102 can include one or more applications, including a video application 104, which can be any application capable of creating or managing video files. For example, video application 104 can be a camera application, a video game application, a movie making application, a video streaming application, a social networking application, or any other application capable of creating, rendering, downloading, receiving, and / or processing video files, including but not limited to high-definition video files.

[0023] like Figure 1A As described, the codec accelerator system 100 may include a controller 110 that is communicatively coupled to several components or functional groups, such as a motion estimation group (ME) 120, a rate-distortion optimization group (RDO) 130, a reconstruction group (RECON) 140, a filtering group (FILTER) 150, and an entropy processing group (EP) 160. Each functional group may be implemented by one or more electronic circuits. In some embodiments, any functional group may be combined with any other functional group on the same silicon die. In some embodiments, all functional groups may be implemented on the same die (together with or separately from the controller 110). Any or all functional groups may be (or include) a number of configurable logic circuits. The logic circuitry of any or all of the functional groups may be configured by the controller 110, which may be an application specific integrated circuit (ASIC), a finite state machine (FSM), a field programmable gate array (FPGA), a central processing unit (CPU), a graphics processing unit (GPU), or some other processing device, or a combination of the foregoing. The various functional groups of the codec accelerator system 100 may also be communicatively coupled to a memory 170, which may include one or more memory components such as cache memory, dynamic random addressable memory (“DRAM”), static random addressable memory (“SRAM”), non-volatile memory (e.g., flash memory), or other suitable data storage.

[0024] The controller 110 may receive data from the host device ( Figure 1AController 110 may receive an instruction (not explicitly described in the preceding text) that identifies a video file to be encoded, for example, by the file's storage location in memory 170. In response to receiving the instruction, controller 110 may initialize the functional groups of codec accelerator system 100. Codec accelerator system 100 may sequentially process each frame in the video file. Each frame may be processed using one or more reference frames. Memory 170 may maintain a status register (not explicitly described) that indicates the frames of the video file designated as reference frames. Controller 110 or ME 120 may manage (e.g., access and control) the status register and, from time to time, change the designation of reference frames, remove reference designations from certain frames, and designate other frames as reference frames. In some embodiments, while processing a current frame, ME 120 may access (receive from memory 170) reference pixels 172 of one or more reference frames, which may include one or more previous frames and / or one or more subsequent frames. The ME 120 may further access the source pixels 174 of the current frame and partition the current frame into blocks of various sizes according to the codec standard.

[0025] Figure 1B According to at least some embodiments Figure 1A Schematic block diagram of the motion estimation functional group (ME 120) of the example codec accelerator system 100. Figure 1B As shown, ME 120 may include several functional circuit blocks that can be implemented by separate circuits. Alternatively, any (or all) functional circuit blocks may be implemented on an integrated circuit. For each block in the current frame, ME 120 may identify a motion vector that specifies the motion transformation from the best candidate block (inter-frame prediction candidate) in one of the reference frames to a specific block in the current frame. In some embodiments, the motion transformation may be a translation specified by a two-dimensional vector (X, Y) in units of the number of pixels X and Y in the horizontal and vertical directions. The units may be integers or decimals. In some embodiments, the motion transformation may be an affine transformation that preserves straight lines and parallelism but does not necessarily preserve angles or distances. The affine transformation may be specified by a combination of coordinate translation and matrix transformation.

[0026] More specifically, full pixel search (FPS) functionality 122 can identify a motion vector based on integer pixel values ​​X and Y. Sub-pixel search (SPS) functionality 124 can then correct the X and Y values ​​using fractional pixel precision (e.g., ¼ pixel precision, ⅛ pixel precision, ⅛ pixel precision, etc.). In some embodiments, FPS 122 and SPS 124 can determine the motion vector based on pixel brightness (luminance, or simply, luma) data of luma reference pixel 176-1 (e.g., stored in memory 170). Pixel color (chromaticity, or simply, chroma) data can be processed separately, as described below. The determined fractional pixel motion vector (or, in some embodiments, an integer pixel motion vector if a lower resolution is sufficient) can be provided to luma motion compensation (luma MC) functionality 126. Luma MC functionality 126 can specify one or more motion compensation filters to be applied to the inter-prediction candidate block. Luma MC function block 126 may provide the determined motion vector and filter type information to RDO 130. Determining the motion vector based on luma (rather than luma and chroma) may save computational effort if RDO 130 does not select an inter-predicted block.

[0027] Continue to refer Figure 1A , information output by ME 120 (e.g., by luma MC function circuit block 126) may be received by RDO 130, which performs the following dual functions: i) performs initial intra block prediction, and ii) compares the initial intra prediction with the inter prediction generated by ME 120. RDO 130 may then select the best block (intra or inter) candidate. More specifically, initial intra prediction may include sequential processing of blocks of a first size (e.g., a 16×16 pixel block or a 32×32 pixel block), followed by parallel processing of blocks of various other sizes, as described below. Figure 3 As described in more detail below, during the sequential processing of blocks of a first size, initial reference pixels (reference pixel grid) are generated and stored in memory 170 (reference pixel storage 176). The initial reference pixels are then used during the parallel processing of blocks of other sizes.

[0028] In some instances, after comparing the intra prediction and the inter prediction, the RDO 130 may determine that the inter prediction block is the winner. For example, the inter prediction may have a smaller cost value representing the difference between the prediction block and the corresponding source block. In these instances, the selected inter prediction winner is returned to the ME 120, and additional motion compensation filtering is performed on the chroma pixel values ​​of the winner block. Referring again to Figure 1B, the motion vector may be provided to the chroma MC functional circuit block 128. The chroma MC functional circuit block 128 may access the chroma reference pixel 176-2, identify a chroma motion compensation filter, and pass the identified filter back to the RDO 130. In some embodiments, the chroma data may have a reduced (compared to luma) resolution. For example, chroma values ​​(e.g., chroma values ​​for red and blue, or red and green, or other suitable chroma values) may be defined for each pair of pixels, each 2x2 pixel block, each four pixels (4x4 block), or according to any other suitable scheme. Even in cases where the winning inter-frame prediction block is determined based solely on luma data, the motion compensation filter may be applied independently of the luma filter (previously applied by the ME 120). Return to Reference Figure 1A In instances where an inter-prediction block is not selected, the motion vector may not be returned to the ME 120 for chroma motion compensation because the winner intra-prediction block is provided to the RECON 140 .

[0029] As described above, although motion vector determination can be based solely on luma pixels, in other embodiments, motion vectors can be determined based on both luma and chroma source pixel data, and the luma error, red chroma error, and blue chroma error can be evaluated using the same or different cost functions. The relative weights assigned to the luma error and chroma error can be determined empirically through testing. Various cost functions can be used, including the sum of absolute transform differences (SATD), the sum of squared differences (SSD), or other suitable cost functions. For example, the selection of a luma motion compensation filter and / or a chroma motion compensation filter (or both) can be performed based on an SSD cost function that characterizes the difference between the luma (and / or chroma) source pixel values ​​of the current frame and the pixel values ​​of the selected inter-frame prediction block.

[0030] RECON 140 receives prediction blocks from RDO 130 and ME 120. Some of the received prediction blocks may be inter-prediction blocks identified (and motion compensated) by ME 120. Some of the received prediction blocks may be intra-prediction blocks. More than one intra-prediction block may be received for the same region of a frame. More specifically, an initial prediction block of a first size (e.g., identified by RDO 130 in an initial sequential phase) may be received along with one or more additional candidate blocks of other sizes (e.g., identified by RDO 130 in a parallel processing phase). If an intra-prediction block is received, RECON 140 may first identify a final prediction block (from the received initial prediction block and the additional prediction blocks). RECON 140 then performs final block reconstruction, for example, starting sequentially from the identified block at the top left (or in some other order, if consistent with the codec standard). Specifically, RECON 140 calculates a residual block by subtracting the final prediction block from the source blocks of various sizes and positions. RECON 140 then identifies the residual block and applies a discrete transform to the residual block to obtain a transformed block, and finally applies a quantized transform to the transformed block. If the winning block is an inter-frame predicted block, RECON 140 performs the same operations except that it does not need to identify the final predicted block (since the inter-frame predicted block becomes the final predicted block). The encoded data, including the transformed (and quantized) residual block, is passed to EP 160 along with the identification of the final predicted block for use in generating the encoded output 178. The identification of the final predicted block may include the block size and the final interpolation mode (if it is an intra-frame final predicted block) or the motion vector (if it is an inter-frame final predicted block).

[0031] Similarly, RECON 140 can also identify other final prediction blocks and reconstruct them. The boundary pixels of the reconstructed block become reference pixels for subsequent block reconstructions and are stored in memory 170. In some embodiments, in the reference pixel register 176, the reference pixels from the reconstructed block replace the original reference pixel block previously used by RDO 130. To match the frame that the decoder will receive from the encoded output 178, the reference pixels used by RECON 140 may be pixels of the reconstructed block (which will also be used by the decoder). More specifically, a reconstructed residual block can be obtained from the transformed and quantized residual block by applying an inverse quantization transform and an inverse discrete transform to obtain a reconstructed residual block, which is then added to the final prediction block. This process continues until all final prediction blocks have been similarly reconstructed.

[0032] The set of reconstructed blocks of the frame is not yet intended for display to a viewer (at the decoder). Additionally, the reconstructed blocks may be processed by a set of filters 150. Filters 150 may select from a number of filters identified by the codec standard being used and further determine individual filter parameters to enhance the visual quality of the reconstructed blocks, including removing boundaries and other artifacts generated during block prediction and reconstruction. In some embodiments, available filters may include a deblocking filter, which can remove visible block boundaries between adjacent blocks. For example, a deblocking filter may identify discontinuities in luma and / or chroma values ​​across a boundary and propagate this value across several pixels. The degree of spreading may be within parameters determined by filter 150. Furthermore, filter 150 may apply a constrained directional enhancement filter (CDEF) to remove ringing artifacts near the depiction of sharp edges of various objects. More specifically, because the quantization transform irreversibly reduces or eliminates some short-wavelength harmonics (which are important for imaging with sharp edges), CDEF can compensate for the loss of these harmonics. CDEF can identify the most likely edge directions, for example, by identifying lines of constant intensity and lines of maximum intensity gradient. In some embodiments, identification of these lines can be performed by identifying the direction of the best matching directional pattern for the block and minimizing the SSD. After identifying the direction of the block, CDEF can sharpen the depiction of edges in the block by selecting filter strengths along the determined direction and across this direction. The filter 150 can also apply a loop restoration (LR) filter to the block. The LR filter further improves edge quality and reduces noise using several filters, such as self-steering filters, Wiener filters, or a combination of self-steering filters and Wiener filters. The scalared filter parameters (for deblocking, CDEF, LR filter or other suitable filters, or any combination thereof) are then transmitted to the EP 160 for inclusion in the encoded output 178.

[0033] In some embodiments, to facilitate generation of the encoded output 718, filter 150 may perform filtering on certain super-blocks before all blocks within a given super-block are reconstructed by RECON 140. For example, if a super-block has a size of 64×64 pixels, filter 150 may determine filtering parameters based on a reconstructed region of reduced size, such as 48×48 pixels, 48×56 pixels, 48×54 pixels, 52×56 pixels, or some other region of reduced size. The reduced size region may be completely within the corresponding super-block or may touch at least one boundary of the super-block. The size of the reduced size region may be dynamically variable, for example, based on the size of the reconstructed blocks within the super-block.

[0034] The output of filter 150 includes information sufficient for a decoder to reconstruct the image frame (as well as multiple previous and subsequent frames), including identification of various inter-frame and intra-frame prediction blocks, the sizes and interpolation methods used for these blocks, identification of the discrete transform used to encode the blocks, identification of filters used to improve the visual quality of the blocks, and so on. The output of filter 150 can be provided to EP 160 for entropy coding. Specifically, EP 160 can replace each fixed-length input symbol with a variable-length prefix-free output codeword. The length of each codeword can be approximately proportional to the negative logarithm of the probability of the codeword occurring, so that the most frequently used symbols are represented using the shortest code. EP 160 can use various encoding techniques, including Huffman coding and arithmetic coding. The encoded output 178 can be stored in memory 170 and / or streamed live via the Internet or other suitable network, including a local area network, a wide area network, a personal area network, a public network, a private network, etc.

[0035] In some embodiments, the functions described above are performed by circuit blocks. In another embodiment, certain functions (of one or more circuits 120-160) may be implemented in one or more software or firmware modules executed by a processing device, such as a central processing unit (CPU), a graphics processing unit (GPU), a field programmable gate array (FPGA), an application-specific integrated circuit (ASIC), or some other suitable logic device. For example, a first software / firmware module may implement the functions of filter 150, a second software / firmware module may implement the functions of EP 160, and ME 120, RDO 130, and RECON 140 may be implemented by a dedicated functional circuit group. In some embodiments, ME 120, RDO 130, RECON 140, filter 150, and EP 160 may all be implemented as software / firmware modules.

[0036] Figure 2 is a schematic block diagram of a rate-distortion optimization (RDO) functional group in accordance with at least some embodiments. Figure 2 The RDO described in can be Figure 1A The RDO 130 of the example codec accelerator system 100 is shown in FIG. Figure 2As depicted, RDO 130 may include several functional circuit blocks, which may be implemented using the same circuit or separate circuits. The functional circuit blocks may include an intra-frame RDO functional circuit block 132, a shared RDO functional circuit block 134, and an inter-frame RDO functional circuit block 138. Intra-frame RDO functional circuit block 132 may apply various interpolation modes to a given block of a current image using reference pixels and provide an indication of a predicted block (e.g., interpolated pixel values) to shared RDO functional circuit block 134 for cost evaluation 136. Shared RDO functional circuit block 134 may access source pixels of the current frame using a peripheral direct memory access controller (PDMA) 135. Cost evaluation 136 may evaluate the differences between pixels of the predicted block and pixels of the corresponding source block using one or more cost functions (e.g., SATD, SSD, or any other suitable cost function). The shared RDO 134 may then provide the determined cost value back to the intra-RDO functional circuit block 132 , which may select one or more lowest-cost interpolation modes for further processing by the RECON 140 .

[0037] Inter-frame RDO functionality circuit block 138 may utilize a history data module (HIST) 137, which collects data from previously processed inter-frame prediction blocks for reuse during processing of subsequent blocks (e.g., neighboring blocks). Inter-frame RDO functionality circuit block 138 may obtain a motion vector from luma MC 126 and generate a representation (e.g., pixel values) for the inter-frame prediction block based on the received motion vector. The generated representation may be sent to shared RDO functionality circuit block 134 for cost evaluation 136. Shared RDO functionality circuit block 134 may utilize PDMA 135 to access source pixels of the current frame and one or more reference frames. Cost evaluation 136 may evaluate the differences between the pixels of the inter-frame prediction block and the pixels of the corresponding source blocks. Shared RDO functionality circuit block 134 may then provide a determined cost value back to inter-frame RDO functionality circuit block 138. Based on the determined cost, shared RDO functionality circuit block 134 may further select between the inter-frame prediction block and the intra-frame prediction block. The winner (the lowest cost predicted block) is then provided for further processing by RECON 140 .

[0038] The inter-frame RDO function circuit block 138 can also select a discrete transform type to be applied to the difference between the pixels in the selected inter-frame prediction block and the corresponding source pixels in the current frame. The selection can be made by cost evaluation 136 and can be based on any suitable cost function (e.g., SSD, SATD, etc.). The discrete transform can be reversible and can include a discrete Fourier transform, a discrete cosine transform (DCT), a horizontal DCT, a vertical DCT, an asymmetric discrete sine transform (ADS), a flipped ADS transform, an identity transform (e.g., for imaging sharp transitions / edges), or other suitable discrete transforms.

[0039] Figure 3 is a schematic diagram illustrating the operation of initial intra-frame sequential processing 301 and parallel processing 302 of a codec accelerator system for efficient encoding operations according to at least some embodiments. In some embodiments, Figure 3 The operations depicted can be performed by Figure 1A In some embodiments, the RDO 130 of the example codec accelerator system 100 performs Figure 3 The depicted operations may be performed based on instructions generated by a software or firmware codec application.

[0040] A sequential process 301 may be performed to identify a reference pixel grid (RGP) 318 that may be used as input to the parallel process 302. In some embodiments, the RPG may include a pixel line lattice having a period determined by the size of the smallest block identified by the codec standard. For example, in an AV1 embodiment, the RPG 318 may be a pixel line lattice having a period corresponding to an 8×8 block (e.g., Figure 3 ). In at least one embodiment, the RPG 318 can be identified without sequential processing 301, for example, by selecting source pixels at various locations entering the RPG 318. In those embodiments in which sequential processing 301 is performed, the sequential processing 301 can involve blocks of any size (referred to herein as a first size) identified by the codec standard, for example, 16x16 blocks, 8x8 blocks, 32x32 blocks, and so on, although blocks of any other size can be used instead of or in addition to blocks of the first size. In some embodiments, the first size can refer to rectangular blocks, for example, 8x16 blocks or 16x32 blocks, or any other suitable block size. In some embodiments, the first size can be selected based on the type and context of the video being encoded. When higher detail resolution is required, smaller blocks can be used, while when computational cost is to be minimized, larger blocks can be used.

[0041] Figure 3Schematically depicted is an initial prediction for block N 304. Block N can be any block of a first size in the current frame. The output of the initial prediction is initial prediction block N 312. Prediction is performed using reference pixels 306-1, which may include certain reference pixels determined during the prediction process for previous blocks 1...N-1. Reference pixels 306-1 may include pixels located to the left / top of block N (although some other suitable pixels may be used as reference pixels). In these examples, blocks to the left / top of block N have not yet been processed, and some reference pixels 306-1 may be source pixels or pixels having certain predetermined values ​​(e.g., values ​​at 50% of maximum intensity). The initial block N prediction 304 may include evaluating several available intra-prediction modes, including, but not limited to, a vertical mode (where predicted pixels are extrapolated downward vertically from the top row of reference pixels 306-1), a horizontal mode (where predicted pixels are extrapolated right horizontally from the left column of reference pixels 306-1), a planar mode (where predicted pixels are interpolated from the top row and left column of reference pixels 306-1), a DC mode (where predicted pixels are the average of the top row and left column of reference pixels 306-1), certain angular modes (where predicted pixels are extrapolated from the top row or left column of reference pixels 306-1, or from a combination of the top row and left column of reference pixels 306-1), etc. In some embodiments, all possible modes recognized by the codec standard may be evaluated, while in other embodiments, only a subset of all possible modes may be evaluated (e.g., a subset of 28 of the 56 angular modes of the AV1 standard may be evaluated).

[0042] Figure 3 Blocks predicted using different interpolation modes are depicted using different shading patterns. These modes can be evaluated using a cost function 308, which can be a SATD, SSD, or any other suitable cost function. Cost function 308 can be used to compare (e.g., pixel by pixel) block N predicted using a particular interpolation mode with the corresponding source block 310 of the current frame. Based on the comparison, the best mode can be selected for the initial prediction block N 312. A subset of pixels from the prediction block, such as reference pixel 306-2, can be used to predict subsequent blocks (and can be included in RPG 318). Reference pixel 306-2 can be a right boundary pixel (e.g., used when predicting the right neighbor of block N) and / or a bottom boundary pixel (e.g., used when predicting the bottom neighbor of block N). Reference pixel 306-2, added to some reference pixels 306-1, can be used as reference pixel 306-3 for the initial prediction of block N+1 314. This process can continue until the entire frame is predicted. This initial prediction frame 316 , partitioned into prediction blocks of a first size, may then be used as input to the parallel processing 302 .

[0043] The parallel processing 302 may use an RPG 318 of the initial prediction frame 316. The RPG 318 may include pixels of the initial prediction frame 316 that are located on the grid lines of the RPG 318 (e.g., along the boundaries of the smallest size blocks, such as 8x8 blocks). Additional block processing 320 may then be performed in parallel using the pixels of the RPG 318 as reference pixels. The processing may be performed similarly to the processing described above in conjunction with 304-312, with the difference that the reference pixels are not updated. Each (second, third, fourth, etc.) sized block may be processed using at least some (or in some embodiments, all) interpolation modes recognized by the codec standard. For each size, a predetermined number M of top candidate interpolation modes may be selected, where the top candidate mode is selected based on having a lower cost determined by a cost function. The cost function used to evaluate the modes may be SATD, SSD, or any other suitable cost function. In some embodiments, blocks of the first size are not re-evaluated during the additional block processing 320. In such an embodiment, Figure 3 The output of the intra-frame RDO processing depicted in includes one best mode for a block of a first size and M (top) candidate modes for each other size. In some embodiments, additional processing of the blocks of the first size may be further performed and M top candidate blocks may be output for each block of all sizes (including the first size). In some embodiments, M=2 or M=3, even though in other embodiments, a very large number of candidate modes may be output. Even different blocks of the same size may have different candidate modes. For example, a particular block of size 16x8 may have a horizontal mode, a planar mode, and a DC mode as candidate modes, while a neighboring block of the same size 16x8 located immediately above may have a vertical mode and two corner modes as candidate modes.

[0044] It should be understood that Figure 3 Only intra-frame prediction operations are described. Figure 1A 、 Figure 1B and Figure 2 As described above, before the RECON 140 group performs the final block reconstruction, the RDO 130 may then select among the intra and inter candidate modes, as described below in conjunction with Figure 4 shown.

[0045] Figure 4 is a schematic diagram illustrating the operation of final intra-frame processing 400 of a codec accelerator system for efficient encoding operations according to at least some embodiments. In some embodiments, Figure 4 The depicted operations may be performed by RECON 140 of the example codec accelerator system 100 of FIG. 1 . In some embodiments, Figure 4The depicted operations may be performed by one or more general-purpose CPUs or one or more GPUs based on instructions generated by a software or firmware codec application.

[0046] exist Figure 3 After the parallel processing 302 has identified the top M candidate modes for each block of the current image (each mode is associated with a specific prediction block, also referred to herein as a candidate block), those modes that have not been eliminated as intra prediction candidates can be subjected to the final intra processing 400. Figure 4 As schematically depicted, the final prediction of block N 404 may be performed using reconstructed (reference) pixels 406-1, which may include certain reconstructed pixels determined during the previous reconstruction of the final blocks 1...N-1. It will be appreciated that even though the same symbol N is used to enumerate Figure 4 and Figure 3 The blocks reconstructed as part of the final intra process 400 may also be different from the blocks predicted as part of the sequential process 301. In particular, blocks of different (e.g., all) sizes may be reconstructed during the final intra process 400.

[0047] The reconstructed pixels 406-1 may include pixels located to the left / top of block N (or other suitable reference pixels, which may be specified by the codec standard). In instances where the blocks to the left / top of block N have not yet been reconstructed, some of the reconstructed pixels 406-1 may be source pixels or reference pixels 306-1, 306-2, 306-3, etc. obtained during the sequential processing 301. Prediction of the final block N (404) may include evaluating the M remaining candidates output by the parallel processing 302 and the best mode for the block of the first size determined during the sequential processing 301 (in those embodiments in which no additional modes are identified for the block of the first size). In some embodiments, prediction of the final block N (404) may also include evaluating interpolation modes that were not previously evaluated during the parallel processing 302 or the sequential processing 301.

[0048] A cost function 408 may be used to select various candidate modes, which may be Figure 3 308 in the same or different. In some embodiments, the cost function 308 may be a cost function based on SATD, while the cost function 408 may be a cost function based on SSD (or vice versa). The cost function 408 can be used to compare (e.g., pixel by pixel) the final block candidate calculated using a given candidate mode with each source block 410 of the current frame. Although the source block 410 is depicted as a square, if the block N is rectangular, a rectangular source block of the corresponding size can be used for reconstruction.

[0049] Based on the comparison results, a final mode can be selected for the final prediction block N 412. RECON 140 (or any other suitable device performing the final intra-frame processing 400) then performs the final block reconstruction, for example, sequentially, starting with the top left block (or other order recognized by the codec standard). Specifically, the final prediction block N 412 can be subtracted (operation 418) from the corresponding source block 410 of each size and position. This subtraction is performed pixel by pixel, resulting in a residual block 420, Δ(x, y) = Source(x, y) - Pred.Block(x, y). RECON 140 (or other suitable device) then selects and performs a discrete transform on the residual block 420. The discrete transform can be reversible and can include a discrete Fourier transform, a discrete cosine transform (DCT), a horizontal DCT, a vertical DCT, an asymmetric discrete sine (ADS) transform, a flipped ADS transform, an identity transform (e.g., for imaging sharp transitions / edges), or other suitable discrete transforms. The transform can be selected using a transform cost function 422. The selected transform is used to generate a transformed (residual) block 424, Δ(x,y)→Δ(k x ,k y ). Subsequently, a (forward) quantization transform may be applied to the transformed block 424, resulting in a quantized block 426, Δ(k x ,k y )→Δ Q (k x ,k y ). The quantization transform may be irreversible and, in some embodiments, may include dividing the transformed block 424 (element-wise) by the table of quantization coefficients and applying a subsequent rounding operation to the result of the division operation. Consequently, small components of the transformed block 424 may become zero, such that the quantized block 426 includes fewer (in some instances very few) non-zero elements than the transformed block 424.

[0050] Quantized blocks and other coded data may be provided for filtering and coding (428), e.g., by Figure 1A The example codec accelerator system 100 includes filters 150 and EP 160. Other encoded data may include identification of the final predicted block (including the size and position of the block), identification of the type of prediction used (inter or intra), motion vectors and motion compensation filters (for inter prediction), the final interpolation mode (for intra prediction), the type of discrete transform used, identification of the quantization coefficient table, and various other data.

[0051] The quantized block 426 Δ can then be used Q (k x,ky) to obtain the reconstructed block N 413. RECON 140 (or some other device performing the final intra-frame processing 400) may perform an inverse quantization transform to obtain an inverse quantized transformed block 430. The inverse quantization transform may include multiplying the quantized block 426 (e.g., element-wise) by a table of quantization coefficients. Due to rounding performed during the forward quantization transform, the inverse quantized transformed block 430, Δ'(k x ,ky) is usually the same as the transformed block 424, Δ(k x Therefore, the inverse discrete transform is applied to the inverse quantized transformed block 430, Δ'(k x ,ky)→Δ'(x,y), thus generating a reconstructed residual block 432, Δ'(x,y), which is different from the residual block 420, Δ(x,y). The reconstructed residual block 432 (operation 434) is then added to the final prediction block N 412 to obtain a reconstructed block N 413, for example, as follows: Recon.Block(x,y)=Pred.Block(x,y)+Δ'(x,y).

[0052] and Figure 3 Similar to the boundary pixels of the prediction blocks 304 and 314, certain boundary pixels of the reconstructed block N 413 become reference pixels for subsequent block reconstruction. For example, a subset of pixels of the reconstructed block N 413 (e.g., the bottom row and rightmost column of the reconstructed block) can be used as reconstructed (reference) pixels 406-2 for the final block N+1 prediction 414. The final prediction block N+1 416 can be processed in a similar manner to block N. This process can continue until the entire frame is reconstructed.

[0053] Figure 5A 、 Figure 5B and Figure 6 Flowcharts of example methods 500, 501, and 600, respectively, of video codec acceleration operations according to at least some embodiments. In some embodiments, methods 500, 501, and 600 may be performed by Figure 1A In some embodiments, methods 500, 501, and 600 may be performed by one or more circuits that may communicate with one or more memory devices. In some embodiments, at least some operations of methods 500, 501, and 600 may be performed by multiple (e.g., in parallel) hardware threads, each thread executing one or more separate functions, routines, subroutines, or operations of the methods. In some embodiments, the processing threads implementing methods 500 and 600 may be synchronized (e.g., using semaphores, critical sections, and / or other thread synchronization mechanisms). Alternatively, the hardware threads implementing methods 500, 501, and 600 may execute asynchronously with respect to one another. Figure 5A 、 Figure 5B and Figure 6 The various operations of methods 500, 501, and 600 may be performed in a different order than shown. Certain operations of methods 500, 501, and 600 may be performed concurrently with other operations. In some embodiments, the operations may not be performed. Figure 5A 、 Figure 5B and Figure 6 One or more operations shown in .

[0054] Figure 5A Flowchart of an example method 500 for real-time compression encoding of a video file using hardware parallel processing, according to at least some embodiments. Method 500 may be performed to encode a video file into the AV1 codec format, the VP9 codec format, or any other suitable video codec format. The video file may include multiple image frames. Method 500 may include sequential and parallel processing of various blocks (partitions) of the image frame. The following description of the operation of method 500 uses an example image frame for illustration, but it should be understood that other frames of the video file may be similarly processed. At block 510, one or more circuits executing method 500 may partition the image frame of the video file into a first plurality of partitions of a first size. For example, the first size may be measured in pixels and may refer to 16×16 pixels, 32×32 pixels, 8×8 pixels, etc. In some embodiments, the first size may be non-square (e.g., rectangular), such as 16×8 pixels, 64×16 pixels, etc. The first plurality of partitions may be used for initial (e.g., sequential) processing of the image frame. Furthermore, the image frame may be partitioned into a second plurality of partitions of a second size. Here, "partitions of a second size" may refer to partitions of any size different from the first size, collectively including partitions of various sizes, e.g., 32x32 pixel partitions, 32x16 pixel partitions, 8x16 pixel partitions, and any other partitions recognized by the encoder specification.

[0055] At block 520, method 500 may continue to sequentially process the first plurality of partitions with one or more circuits to interpolate pixel blocks (PBs) of each of the first plurality of interpolated PBs (e.g., at Figure 3 A block of the initial prediction frame 316 in the image frame is associated with each of the first plurality of partitions. For example, for each partition of the image frame (e.g., 16×16 pixels), several interpolation modes (e.g., defined by the codec specification) may be evaluated. Figure 5AAs shown by the upper dashed arrows in , processing the first plurality of partitions may include identifying a plurality of candidate interpolated PBs. Respective interpolation modes of a plurality of interpolation modes may be used, and each of the plurality of candidate interpolated PBs may be generated based on a set of reference pixels for the partition. For example, each of a DC mode, a planar mode, a horizontal mode, a vertical mode, any angle mode, and the like may generate a different interpolated PB. As used herein, the term "interpolation" shall be understood to include any manner of inferring pixel values ​​of a PB based on a set of reference pixels (e.g., pixels of adjacent blocks located near a PB boundary), including interpolation, extrapolation, averaging, and / or using any suitable mathematical formula or scheme.

[0056] After generating one or more interpolated PBs, the one or more circuits may identify and select an optimal interpolation mode that generates an interpolated PB having interpolated pixel values ​​that are closest to source pixel values ​​for respective blocks of the same size and located at the same position in the image frame, as shown in block 524. The selection of the optimal mode may be based on a comparison of pixel values ​​of each of the plurality of candidate interpolated PBs with source pixel values ​​of the image frame, and the selection of the optimal mode may be facilitated by selecting an appropriate cost function, such as a SATD cost function, an SSD cost function, or the like.

[0057] At block 530, method 500 may continue by identifying, using one or more circuits, a reference pixel grid (RPG) that includes interpolated pixel values ​​for a subset of pixels of the first plurality of interpolated PBs. For example, the RPG may be a grid of pixels arranged along horizontal and vertical lines that are periodically spaced apart by a period corresponding to a minimum block size of a codec specification (e.g., 8x8 pixels). In some embodiments, the pixels of the RPG may be pixels adjacent to a boundary line that partitions the image frame into the minimum block size partitions (or any other pixels recognized as reference pixels by the codec standard).

[0058] At block 540, method 500 may continue to process a second plurality of partitions in parallel using the RPG using one or more circuits. The second plurality of partitions may be executed in parallel by multiple hardware processing threads. For example, one or more circuits may process partitions of all sizes in parallel using the RPG as a reference grid for further pixel interpolation. The parallel processing may be performed such that one or more interpolated PBs of the second plurality of interpolated PBs are associated with each partition in the second plurality of partitions. The second plurality of PBs may include multiple interpolated PBs for a given partition. For example, Figure 5AAs indicated by the bottom dashed arrow, the operations performed with block 540 can be similar to the operations performed with blocks 522 and 524 with block 520. Specifically, processing the second plurality of partitions can include identifying a plurality of candidate interpolated PBs. Each of the plurality of candidate interpolated PBs can be generated based on a set of reference pixels using a respective interpolation mode of a plurality of interpolation modes. The reference pixels used with block 540 can be reference pixels of the RPG identified at block 530. One or more circuits can then select one or more interpolated PBs (top candidates) based on a comparison of pixel values ​​of each of the plurality of candidate interpolated PBs with source pixel values ​​of the image frame. The selection of the top candidate can be facilitated by an appropriate choice of a cost function, such as a SATD cost function, an SSD cost function, and the like (which may be the same or different from the cost function used with block 520). In some embodiments, N top candidates are selected for each of the second plurality of partitions (e.g., for each 8x8 pixel block, each 32x16 pixel block, etc.). In some embodiments, additional candidates may be identified for each of the first plurality of partitions (eg, for each 16x16 pixel block or any other first size).

[0059] At block 550, method 500 may continue by selecting a prediction PB from a plurality of candidate PBs using one or more circuits, the candidate PBs may include at least a first plurality of interpolated PBs and a second plurality of interpolated PBs. For example, the top candidate for each block (partition) may be selected from the first plurality of interpolated PBs (e.g., 16×16 interpolated PBs) or the second plurality of interpolated PBs (e.g., PBs of all other sizes). In some embodiments, selecting the prediction PB may be based on a comparison of interpolated pixel values ​​of the corresponding plurality of interpolated PBs with source pixel values ​​of the image frame. Selecting the prediction PB may be performed using a second cost function different from the first cost function (for identifying the candidate interpolated PBs). In some embodiments, the prediction PB may be further selected from some additional plurality of candidate blocks. For example, the plurality of candidate PBs may further include a plurality of inter-frame reference PBs. Each of the plurality of inter-frame reference PBs may correspond to an inter-frame reference PB in one of the reference image frames (e.g., a previous frame or a next frame). In some embodiments, the inter-frame reference PBs may be additionally filtered using one or more motion compensation filters. In some embodiments, the inter-frame reference PB can be associated with multiple reference frames, for example, it can be the average of a first inter-frame reference PB of one reference frame (e.g., one of the previous reference frames) and a second inter-frame reference PB of another reference frame (e.g., one of the subsequent reference frames).

[0060] At block 560, method 500 may continue to create a representation of the image frame (e.g., generate a codec bitstream) using the interpolated pixel values ​​of the predicted PB using one or more circuits. In some embodiments, creating the representation of the image frame may include a reconstruction of the predicted PB, which may be related to the following Figure 5B Describe together.

[0061] Figure 5B is a flow chart of an example method 501 for reconstructing a predicted PB for efficient real-time compression encoding of a video file according to at least some embodiments. In some embodiments, the method 501 may be performed using multiple hardware processing threads that process various blocks of an image frame. In some embodiments, the method 501 may be implemented as described above. Figure 5A In some embodiments, method 501 may be performed as part of any other applicable method of video encoding, including but not limited to the following methods. Figure 6 Method 600 is described together with the above. At block 562, one or more circuits executing method 501 may select a first predicted PB. The term "first" should be understood merely as an identifier and does not imply a particular order. The first predicted PB may be associated with any partition of the image frame being reconstructed. Blocks 564-572 represent applying one or more transforms to the first predicted PB by one or more circuits to obtain a reconstructed first PB (associated with the same partition of the image frame).

[0062] At block 564, method 501 may continue to calculate, using one or more circuits, a residual difference PB (eg, between the first prediction PB (eg, final prediction block N 412) and the corresponding source PB (eg, source block 410). Figure 4 4. At block 566, method 501 may continue by applying a reversible discrete transform to the calculated residual difference values ​​PB (e.g., residual block 420) to obtain transformed residual difference values ​​PB (e.g., transformed block 424), and then applying an irreversible quantization transform to the transformed residual difference values ​​PB at block 568 to obtain quantized residual difference values ​​PB (e.g., quantized block 426).

[0063] At block 570, method 501 may continue to use one or more circuits to apply an inverse quantization transform to the quantized residual difference PB (e.g., quantized block 426) to obtain an inverse quantized residual difference PB (e.g., inverse quantization transform block 430), and then, at block 572, apply an inverse discrete transform to the inverse quantized residual difference PB to obtain a reconstructed first PB (e.g., reconstructed residual block 432). Then, at block 574, the reconstructed first PB obtained is used to obtain a reconstructed second PB (e.g., a reconstructed block obtained based on the final prediction block N+1 416). Blocks 562-574 may be repeated until all prediction blocks in the image frame are reconstructed.

[0064] At block 580, in addition to reconstructing the prediction block, one or more circuits may also identify parameters of one or more filters to be applied to the quantized residual difference PB by the image decoder (e.g., by providing Figure 4 The identified parameters of one or more filters may be included in a representation of the image (eg, encoded into a bitstream that a decoding system will use to decode the video file).

[0065] Figure 6 6 is a flow chart of another example method 600 for real-time compression encoding of a video file using hardware parallel processing, according to at least some embodiments. Method 600 may include parallel and sequential processing of blocks (partitions) of an image frame. Parallel processing may be performed by multiple hardware processing threads. The following description of the operations of method 500 uses an example image (or image frame) for illustration, but it should be understood that other images of a video file may be similarly processed. At block 610, one or more circuits executing method 600 may process a first plurality of partitions of the image to identify, for each of the first plurality of partitions, one or more initial interpolation modes from a plurality of interpolation modes for a video codec. In some embodiments, the first plurality of partitions may include partitions of one size (e.g., 16×16 pixels or 32×32 pixels). In some embodiments, the first plurality of partitions may include partitions of multiple sizes (e.g., 16×16 pixels and 32×32 pixels). Similarly, in some embodiments, an initial interpolation mode may be identified for each identified partition, e.g., a specified interpolation mode that most accurately approximates the source pixels of the image. In some embodiments, multiple initial interpolation modes (eg, N best modes) may be identified for each partition.

[0066] The processing operations of block 610 may be processed in parallel. In some embodiments, one or more initial interpolation modes may be applied to the source pixels of the first plurality of partitions. In these embodiments, the initial sequential processing 301 may be performed first. Figure 3In particular, the RPG for parallel processing may include source pixels as reference pixels. In other implementations, for example, as combined with Figure 1A As described in method 500 in FIG. 5 , the RPG may include reference pixels that are similar to source pixels but are pre-processed according to some suitable scheme.

[0067] At block 620, method 600 may continue by associating, using one or more circuits, a preferred interpolation mode with each of a second plurality of partitions of the image. The second plurality of partitions may be the same as or different from the first plurality of partitions. Associating the preferred interpolation mode may be performed using one or more initial interpolation modes identified for each of the first plurality of partitions. As shown in dashed block 622, associating each preferred interpolation mode with each of the second plurality of partitions of the image may include some or all of the following. The one or more circuits used to perform method 600 may select a size for each of the second plurality of partitions. The one or more circuits may further select a preferred interpolation mode for the selected size of the partition. The preferred interpolation mode and / or size may be selected based on a cost value representing a difference between source pixels of each partition and pixels of an interpolated PB generated using the respective preferred interpolation mode.

[0068] At block 630, the method 600 may continue to sequentially process the second plurality of partitions using one or more circuits to obtain a plurality of reconstructed pixel blocks (PBs), wherein each of the plurality of reconstructed PBs is obtained using a respective preferred interpolation mode associated with a respective partition in the second plurality of partitions. More specifically, similar to Figure 4 At block 640, method 600 may continue to create a compressed representation of the image using the multiple reconstructed PBs of the image using one or more circuits. Creating a compressed representation of the image may include combining Figure 5B Some or all of the operations of method 501 are described.

[0069] Images and videos generated by applying one or more of the techniques disclosed herein can be displayed on a monitor or other display device. In some embodiments, the display device can be directly coupled to a system or processor that generates or renders the image or video. In other embodiments, the display device can be indirectly coupled to the system or processor, such as via a network. Examples of such networks include the Internet, mobile telecommunications networks, WIFI networks, and any other wired and / or wireless network systems. When the display device is indirectly coupled, the image or video generated by the system or processor can be streamed to the display device via the network. For example, such streaming allows a video game or other application that renders an image or video to be executed on a server or in a data center and the rendered image and video to be transmitted and displayed on one or more user devices (such as computers, video game consoles, smartphones, other mobile devices, etc.) that are physically separated from the server or data center. Therefore, the techniques disclosed herein can be used to enhance streamed images or videos and enhance services for streaming images and videos, such as NVIDIA GeForce Now (GFN), Google Stadia, etc.

[0070] In addition, images and videos generated using one or more of the techniques disclosed herein can be used to train, test, or certify deep neural networks (DNNs) for recognizing objects and environments in the real world. These images and videos may include scenes of roads, factories, buildings, urban environments, rural environments, humans, animals, and any other physical objects or real-world environments. These images and videos can be used to train, test, or certify DNNs that are used in machines or robots to manipulate, process, or modify physical objects in the real world. In addition, these images and videos can be used to train, test, or certify DNNs that are used in self-driving cars to navigate and move vehicles in the real world. In addition, images and videos generated using one or more of the techniques described herein can be used to convey information to users of such machines, robots, and vehicles.

[0071] Other variations are within the spirit of the present disclosure. Thus, while the disclosed technology is susceptible to various modifications and alternative constructions, certain illustrative embodiments thereof are shown in the drawings and have been described in detail above. However, it should be understood that the present disclosure is not intended to limit the disclosure to one or more specific forms disclosed, but rather, is intended to cover all modifications, alternative constructions, and equivalents falling within the spirit and scope of the present disclosure as defined by the appended claims.

[0072] Unless otherwise noted or clearly contradicted by the context, the use of the terms "a" and "an" and "the" and similar references in the context of describing the disclosed embodiments (particularly in the context of the appended claims) should be interpreted as covering the singular and plural, rather than as definitions of terms. Unless otherwise noted, the terms "include," "have," "include," and "contain" should be interpreted as open-ended terms (meaning "including but not limited to"), unless otherwise noted. The term "connected" (when unmodified, refers to a physical connection) should be interpreted as partially or completely contained within, attached to, or connected together, even if there is some intervention. Unless otherwise noted herein, references to numerical ranges herein are intended only to be used as a shorthand method of referring to each individual value falling within the range, and each individual value is incorporated into the specification as if it were separately recited herein. In at least one embodiment, unless otherwise noted or contradicted by the context, the use of the term "set" (e.g., "item set") or "subset" should be interpreted as a non-empty set comprising one or more members. Furthermore, unless otherwise indicated or contradicted by context, the term "subset" of a corresponding set does not necessarily mean a proper subset of the corresponding set, but rather a subset and a corresponding set may be equivalent.

[0073] Unless expressly indicated otherwise or clearly contradicted by context, conjunctions such as phrases of the form "at least one of A, B, and C" or "at least one of A, B, and C" are understood in context to generally refer to an item, clause, or the like, which may be A or B or C, or any non-empty subset of the set of A, B, and C. For example, in the illustrative example of a set having three members, the conjunctions "at least one of A, B, and C" and "at least one of A, B, and C" refer to any of the following sets: {A}, {B}, {C}, {A, B}, {A, C}, {B, C}, {A, B, C}. Thus, such conjunctions are not generally intended to imply that certain embodiments require the presence of at least one of A, at least one of B, and at least one of C. Additionally, unless expressly indicated otherwise or contradicted by context, the term "plurality" refers to a plurality (e.g., "a plurality of items" refers to a plurality of items). In at least one embodiment, the number of items in the plurality of items is at least two, but may be more if expressly indicated or indicated by context. Further, unless stated otherwise or clear from context, the phrase "based on" means "based at least in part on" rather than "based solely on."

[0074] Unless otherwise indicated herein or clearly contradicted by the context, the operations of the processes described herein may be performed in any suitable order. In at least one embodiment, processes such as those described herein (or variations and / or combinations thereof) are performed under the control of one or more computer systems configured with executable instructions and are implemented as code (e.g., executable instructions, one or more computer programs, or one or more applications) that are collectively executed on one or more processors by hardware or a combination thereof. In at least one embodiment, the code is stored on a computer-readable storage medium in the form of, for example, a computer program that includes a plurality of instructions that can be executed by one or more processors. In at least one embodiment, the computer-readable storage medium is a non-transitory computer-readable storage medium that excludes transient signals (e.g., propagated transient electrical or electromagnetic transmissions) but includes non-transitory data storage circuits (e.g., buffers, caches, and queues). In at least one embodiment, code (e.g., executable code or source code) is stored on a set of one or more non-transitory computer-readable storage media (or other memory for storing executable instructions) having executable instructions stored thereon, which, when executed by one or more processors of a computer system (i.e., as a result of being executed), causes the computer system to perform the operations described herein. In at least one embodiment, the set of non-transitory computer-readable storage media includes a plurality of non-transitory computer-readable storage media, and one or more of the individual non-transitory storage media in the plurality of non-transitory computer-readable storage media lacks all of the code, but rather the plurality of non-transitory computer-readable storage media collectively store all of the code. In at least one embodiment, the executable instructions are executed so that different instructions are executed by different processors, for example, a non-transitory computer-readable storage medium stores instructions, and a main central processing unit ("CPU") executes some instructions, while a graphics processing unit ("GPU") executes other instructions. In at least one embodiment, different components of the computer system have separate processors, and different processors execute different subsets of instructions.

[0075] Thus, in at least one embodiment, a computer system is configured to implement one or more services that individually or collectively perform the operations of the processes described herein, and such a computer system is configured with applicable hardware and / or software that enables the implementation of the operations. Furthermore, the computer system implementing at least one embodiment of the present disclosure is a single device, and in another embodiment is a distributed computer system comprising multiple devices operating in different ways such that the distributed computer system performs the operations described herein and such that no single device performs all of the operations.

[0076] The use of any and all examples or exemplary language (e.g., "such as") provided herein is intended merely to better illuminate embodiments of the present disclosure and does not limit the scope of the disclosure unless otherwise required. No language in the specification should be construed as indicating any non-claimed element as essential to the practice of the disclosure.

[0077] All references, including publications, patent applications, and patents, cited herein are hereby incorporated by reference to the same extent as if each reference were individually and specifically indicated to be incorporated by reference and were set forth in its entirety herein.

[0078] In the description and claims, the terms "coupled" and "connected," along with their derivatives, may be used. It should be understood that these terms may not be intended as synonyms for each other. Rather, in specific examples, "connected" or "coupled" may be used to indicate that two or more elements are in direct or indirect physical or electrical contact with each other. "Coupled" may also mean that two or more elements are not in direct contact with each other, but still cooperate or interact with each other.

[0079] Unless otherwise expressly stated, it is understood that throughout the specification, terms such as "processing", "computing", "calculating", "determining", etc. refer to the actions and / or processes of a computer or computing system or similar electronic computing device that processes and / or converts data represented as physical quantities (e.g., electronic) in the registers and / or memories of the computing system into other data similarly represented as physical quantities in the memories, registers or other such information storage, transmission or display devices of the computing system.

[0080] In a similar manner, the term "processor" may refer to any device or portion of memory that processes electronic data from registers and / or memory and converts that electronic data into other electronic data that can be stored in registers and / or memory. As non-limiting examples, a "processor" may be a CPU or a GPU. A "computing platform" may include one or more processors. As used herein, a "software" process may include, for example, software and / or hardware entities that perform work over time, such as tasks, threads, and intelligent agents. Likewise, each process may refer to multiple processes to execute instructions continuously or intermittently, sequentially, or in parallel. In at least one embodiment, the terms "system" and "method" may be used interchangeably herein, as long as a system may embody one or more methods, and a method may be considered a system.

[0081] In this document, reference may be made to obtaining, acquiring, receiving, or inputting analog or digital data into a subsystem, computer system, or computer-implemented machine. In at least one embodiment, the process of obtaining, acquiring, receiving, or inputting analog and digital data can be accomplished in a variety of ways, such as by receiving data as parameters of a function call or a call to an application programming interface. In at least one embodiment, the process of obtaining, acquiring, receiving, or inputting analog or digital data can be accomplished by transmitting data via a serial or parallel interface. In at least one embodiment, the process of obtaining, acquiring, receiving, or inputting analog or digital data can be accomplished by transmitting data from a providing entity to an acquiring entity via a computer network. Reference may also be made to providing, outputting, transmitting, sending, or presenting analog or digital data. In various examples, the process of providing, outputting, transmitting, sending, or presenting analog or digital data can be accomplished by transmitting data as input or output parameters of a function call, an application programming interface, or an interprocess communication mechanism.

[0082] Although the above discussion sets forth example implementations of the described technology, other architectures may be used to implement the described functionality and are intended to fall within the scope of this disclosure. In addition, although specific responsibilities are defined above for discussion purposes, the various functions and responsibilities may be allocated and divided in different ways depending on the circumstances.

[0083] Furthermore, although the subject matter has been described in language specific to structural features and / or methodological acts, it is to be understood that the subject matter claimed in the appended claims is not necessarily limited to the specific features or acts described. Rather, the specific features and acts are disclosed as example forms of implementing the claims.

Claims

1. An image processing system comprising: a memory device for storing source pixel values ​​of an image frame; as well as one or more circuits communicatively coupled to the memory device, the one or more circuits configured to: dividing the image frame into a first plurality of partitions having a first size; associating a corresponding prediction pixel block PB of a first plurality of prediction pixel blocks PB with each of the first plurality of partitions; identifying a reference pixel grid RPG using the first plurality of prediction PBs, the reference pixel grid RPG comprising pixel values ​​of a subset of pixels of the first plurality of prediction PBs; dividing the image frame into a second plurality of partitions having a second size; obtaining an association of one or more predicted PBs of a second plurality of predicted PBs of a second size with each of the second plurality of partitions using the RPG identified by the first plurality of predicted PBs of the first size, wherein at least two of the associations are obtained in parallel; Selecting a final predicted PB from a plurality of candidate PBs, wherein the plurality of candidate PBs includes the first plurality of predicted PBs and the second plurality of predicted PBs; as well as A representation of the image frame is created using the pixel values ​​of the final prediction PB.

2. The system of claim 1, wherein: To select the final prediction PB, the one or more circuits are configured to use a comparison of the pixel values ​​of the corresponding plurality of prediction PBs with source pixel values ​​of the image frame.

3. The system of claim 1, wherein: To obtain the association, the one or more circuits are configured to: identifying a plurality of candidate predictions PB, wherein each of the plurality of candidate predictions PB is generated using a corresponding interpolation mode of a plurality of interpolation modes; as well as One or more prediction PBs are selected based on a comparison of pixel values ​​of each of the plurality of candidate prediction PBs with source pixel values ​​of the image frame and using a first cost function.

4. The system of claim 3, wherein: The final prediction PB is selected using a second cost function different from the first cost function.

5. The system of claim 1, wherein: The plurality of candidate PBs further include a plurality of inter-reference PBs, wherein each of the plurality of inter-reference PBs corresponds to a reference PB in one or more reference image frames, wherein the final prediction PB includes one or more inter-reference PBs selected from the plurality of inter-reference PBs, and wherein to create the representation of the image frame, the one or more circuits are configured to: A reversible discrete transform is selected for each of the selected one or more inter-frame references PB based on the pixel values ​​of the corresponding inter-frame reference PB.

6. The system of claim 1, wherein: To create the representation of the image frame using the pixel values ​​of the final prediction PB, the one or more circuits are configured to: Select the first final predicted PB; as well as Applying one or more transformations to the first final prediction PB to obtain a reconstructed first final PB; as well as Using the reconstructed first final PB, a reconstructed second final PB is obtained.

7. The system of claim 6, wherein: To apply the one or more transforms to the first final prediction PB, the one or more circuits are configured to: Calculating a residual difference PB between the first final prediction PB and the corresponding source PB; Applying a reversible discrete transformation to the calculated residual difference PB to obtain a transformed residual difference PB; as well as An irreversible quantization transform is applied to the transformed residual difference value PB to obtain a quantized residual difference value PB.

8. The system of claim 7, wherein: To create the representation of the image frame using the pixel values ​​of the final prediction PB, the one or more circuits are configured to: Applying an inverse quantization transform to the quantized residual difference value PB to obtain an inverse quantized residual difference value PB; and An inverse discrete transform is applied to the inversely quantized residual difference PB to obtain the reconstructed first final PB.

9. The system of claim 7, wherein: To create the representation of the image frame using the pixel values ​​of the final prediction PB, the one or more circuits are configured to: identifying parameters of one or more filters to be applied to the quantized residual difference value PB by an image decoder; and Parameters of the one or more identified filters are included in the representation of the image frame.

10. An image processing system comprising: a memory device for storing source pixel values ​​of the image; as well as One or more circuit groups communicatively coupled to the memory device, the one or more circuit groups comprising: The first circuit group is used for: identifying, for each of the first plurality of partitions, one or more initial interpolation modes of a plurality of interpolation modes of a video codec encoder; identifying a reference pixel grid (RPG) using the one or more initial interpolation patterns, the RPG comprising pixel values ​​of a subset of pixels of the first plurality of partitions; and using the RPG, associating a selected interpolation mode of the plurality of interpolation modes of the video codec encoder with each of a second plurality of partitions of the image; and a second circuit group communicatively coupled to the first circuit group, the second circuit group being configured to: processing the second plurality of partitions to obtain a plurality of reconstructed pixel blocks PB, wherein at least some of the plurality of reconstructed pixel blocks PB are obtained using respective selected interpolation modes associated with respective partitions of the second plurality of partitions; and A compressed representation of the image is created using the plurality of reconstructed PBs of the image.

11. The system of claim 10, wherein: Several of the first plurality of partitions have the same size, wherein the second plurality of partitions includes partitions of a plurality of sizes, wherein to associate the corresponding selected interpolation mode with each of the second plurality of partitions of the image, the first set of circuits is configured to: i) the selected interpolation mode and ii) the size of the respective partition of the second plurality of partitions are selected based on a cost value characterizing a difference between source pixels of the respective partition and pixels of a predicted PB generated using the respective selected interpolation mode.

12. The system of claim 10, wherein: The one or more circuit groups further include: a motion estimation ME circuit group, communicatively coupled to the first circuit group, the ME circuit group being configured to: For each of a third plurality of partitions of the image, an inter-frame reference PB corresponding to a reference PB in one or more reference image frames is identified; and wherein at least some of the multiple reconstructed PBs are obtained using the inter-frame reference PB for the corresponding partition in the third plurality of partitions.

13. An image processing method, comprising: partitioning the image frame into a first plurality of partitions having a first size; associating a corresponding prediction pixel block PB of the first plurality of prediction PBs with each of the first plurality of partitions; identifying a reference pixel grid RPG using the first plurality of prediction PBs, the reference pixel grid RPG comprising pixel values ​​of a subset of pixels of the first plurality of prediction PBs; dividing the image frame into a second plurality of partitions having a second size; obtaining an association of one or more predicted PBs in a second plurality of predicted PBs with each of the second plurality of partitions using the RPG identified by the first plurality of predicted PBs, wherein at least two of the associations are obtained in parallel; Selecting a final predicted PB from a plurality of candidate PBs, wherein the plurality of candidate PBs includes the first plurality of predicted PBs and the second plurality of predicted PBs; as well as A representation of the image frame is created using the pixel values ​​of the final prediction PB.

14. The method of claim 13, wherein: The selection of the final prediction PB is based on a comparison of the pixel values ​​of the corresponding plurality of prediction PBs with source pixel values ​​of the image frame.

15. The method of claim 13, wherein: Obtaining the association includes: identifying a plurality of candidate predictions PB, wherein each of the plurality of candidate predictions PB is generated using a corresponding interpolation mode of a plurality of interpolation modes; and One or more prediction PBs are selected based on a comparison of pixel values ​​of each of the plurality of candidate prediction PBs with source pixel values ​​of the image frame and using a first cost function.

16. The method of claim 15, wherein: The final prediction PB is selected using a second cost function different from the first cost function.

17. The method of claim 13, wherein: The plurality of candidate PBs further include a plurality of inter-reference PBs, wherein each of the plurality of inter-reference PBs corresponds to a reference PB in one or more reference image frames.

18. The method of claim 13, wherein: Creating the representation of the image frame using the pixel values ​​of the final prediction PB comprises: Selecting a first final predicted PB; and applying one or more transformations to the first final prediction PB to obtain a reconstructed first final PB; and Using the reconstructed first final PB, a reconstructed second final PB is obtained.

19. The method of claim 18, wherein: Applying the one or more transformations to the first final prediction PB comprises: Calculating a residual difference PB between the first final prediction PB and the corresponding source PB; Applying a reversible discrete transform to the calculated residual difference PB to obtain a transformed residual difference PB; and An irreversible quantization transform is applied to the transformed residual difference value PB to obtain a quantized residual difference value PB.

20. The method of claim 19, wherein creating the representation of the image frame using the pixel values ​​of the final prediction PB further comprises: Applying an inverse quantization transform to the quantized residual difference value PB to obtain an inverse quantized residual difference value PB; as well as An inverse discrete transform is applied to the inversely quantized residual difference PB to obtain the reconstructed first final PB.

21. The method of claim 19, wherein: Creating the representation of the image frame using the pixel values ​​of the final prediction PB further comprises: identifying parameters of one or more filters to be applied to the quantized residual difference value PB by an image decoder; and Parameters of the one or more identified filters are included in the representation of the image frame.

Images