Parallel processing of video frames during video encoding

Abstract

Techniques for compressing image data are disclosed. Intermediate pixels can be determined. Each position in the image can be associated with a block from a plurality of blocks of a first size and a block from a plurality of blocks of a second size. For each block of the first size and the second size, a first cost for a first mode and a second cost for a second mode can be determined in parallel using the intermediate pixels. For each position in the image, a final mode and a final block size can be selected using the first cost and the second cost for each block of the first size and the second size associated with the corresponding position. A final pixel can be determined, and a representation of the image can be obtained based on the final pixel.

Description

Technical Field

[0001] At least one embodiment relates to computational techniques for performing and facilitating efficient compression of video files. For example, at least one embodiment relates to operations used by video codec software to efficiently encode high-definition video files using modern video coding standards while providing numerous tools and options for improving video quality at the expense of increased computational complexity. Background Art

[0002] Video files in their original (source) pixel format can take up very large amounts of memory and require significant network bandwidth, making them impractical for storage and / or live streaming. For example, a typical HD video display has approximately 30 frames per second, which are typically very similar. Much of the information in these frames is inherently redundant, allowing for efficient compression. On the one hand, relatively minor changes occur between subsequent frames (temporal redundancy). On the other hand, regions within a given frame are often similar to each other (spatial redundancy); for example, an image of the sky can extend across a large portion of the frame. Therefore, instead of transmitting the actual pixel information (luminance and chrominance) for each pixel, the codec can identify a reference block similar to the block being encoded ("predicted") and provide the decoder with a suitable and compact mathematical representation of the difference ("delta") between the actual source block and the predicted block. The reference block can be a block from a different frame (e.g., the previous or even the next frame), a block from the same frame, or even a synthetic block generated according to some predetermined scheme (pattern) based on a small number of reference pixels. Then, instead of storing or streaming actual frames of pixels, the codec may output a bitstream of encoded data, which primarily contains instructions for the decoder on how to generate an approximation of a frame whose visual appearance is indistinguishable from or very similar to the source frame. BRIEF DESCRIPTION OF THE DRAWINGS

[0003] Figure 1 is a schematic block diagram of an example computing device that may implement the disclosed methods for efficient encoding operations, according to at least some embodiments.

[0004] Figure 2A is a diagram describing the operation of a sequential process of a codec system for efficient encoding operations according to at least some embodiments.

[0005] Figure 2B is a schematic diagram illustrating the operation of another possible sequential process of a codec system for efficient encoding operations according to at least some embodiments.

[0006] Figure 3A is a flow chart of an example method for processing video frames in parallel during video encoding to achieve efficient encoding operations, according to at least some embodiments.

[0007] Figure 3B is a flow chart of an example method of determining intermediate pixels for use in parallel processing of video frames during video encoding to achieve efficient encoding operations, in accordance with at least some embodiments.

[0008] Figure 4 An exemplary machine in the form of a computer system is shown within which a set of instructions, for causing the machine to perform any one or more of the methodologies discussed herein, may be executed. DETAILED DESCRIPTION

[0009] Video codecs allow digital video files to be compressed to reduce the amount of storage space and bandwidth required to store and transmit the file. A compressed video file may be very similar or indistinguishable from the original video file when uncompressed and displayed. To compress a digital video file, the codec divides each frame of the video into a series of blocks. For each block, the codec determines the optimal size and mode for compression. The codec can determine whether to use intra-frame prediction mode or inter-frame prediction mode. Then, for each block, after selecting the size and mode, the codec compares the predicted result with the source pixels to calculate the difference (or residual) and encodes the predicted size, prediction mode, and residual pixel value in the output bitstream. A single frame can be divided into blocks of various sizes, and each block can have a unique prediction mode. Subsequent frames can be divided into blocks of different sizes, and each block can have a different prediction mode than the blocks in the previous frame. The codec processes each frame individually, minimizing the cost of encoding the frame.

[0010] Each pixel value of a frame may include multiple components, such as a luminance component indicating the brightness or intensity of the pixel, and a chrominance component indicating one or more colors of the pixel (e.g., red, green, blue chrominance values, or any other suitable color values).

[0011] Intra-prediction blocks can use reference pixel values ​​from neighboring blocks to approximate the pixel values ​​of a source block. For example, the pixel values ​​of an intra-prediction block can be based on the pixel values ​​at the bottom boundary of the block above it, the pixel values ​​at the right boundary of the block to its left, the pixel values ​​at the bottom right corner of the block to its upper left, or a combination of these pixel values. Intra-prediction mode is effective in areas of a video frame that have similar colors (e.g., a blue sky) across many blocks.

[0012] An inter-frame prediction block can use pixel values ​​from a previous or subsequent frame (referred to herein as a reference frame) to approximate the pixel values ​​of a source block. This is effective for moving objects in a video. When an object moves, the object in the current frame may look similar or identical to the object in the reference frame, but the object may be in a new position. The codec can use the same pixel values ​​from the object in the reference frame to approximate the pixel values ​​of the object in the current frame. Because the codec uses the same pixel values ​​from the reference frame without having to create a new set of pixel values ​​for the inter-frame prediction block, the inter-frame prediction block has a higher compression rate than the intra-frame prediction block.

[0013] There are various video codec standards that have been defined, such as H.264 (Advanced Video Coding, or AVC), H.265 (High Efficiency Video Coding, or HEVC), H.266 (Versatile Video Coding, or VVC), VP9, ​​and AV1 (AO Media Video 1). Each standard defines a set of acceptable inter-frame prediction modes and a set of intra-frame prediction modes. Each standard also defines a set of acceptable block sizes. For example, AV1 allows 58 intra-frame prediction modes, multiple inter-frame prediction modes, and multiple block sizes, including square blocks of 8×8, 16×16, 32×32, 64×64, 128×128, and rectangular blocks of 64×32, 32×64, 32×16, 16×32, 16×8, and 8×16.

[0014] Standard video codecs start at the top-left corner of the frame, evaluate the cost of all possible combinations of block sizes and prediction modes (including intra and inter prediction modes), make a choice based on the computational cost, and then proceed to the next block (for example, the block to the right of the top-left block). Because intra prediction modes use pixel values ​​from neighboring blocks, each block needs to be processed sequentially to ensure that neighboring blocks have been processed and have reference pixels available for processing the current block. As a result, each block has dependencies on the previous block. Processing blocks sequentially slows down the encoding process because only a single block can be processed at a time, even if no additional computing resources are being used. Furthermore, because there are many possible combinations of block sizes and prediction modes that need to be considered, processing even a single block can take a significant amount of time.

[0015] Aspects and embodiments of the present disclosure address these and other technical challenges of real-time video encoding in modern advanced codecs by reducing the dependencies encountered during the encoding process. In certain embodiments, dependencies are reduced by performing initial sequential processing on blocks of a video frame to create a set of intermediate pixels, and then processing the blocks of the video frame in parallel. For example, the initial sequential processing can be performed on blocks of one size (e.g., 16×16 pixels), and intra-frame prediction mode evaluation is used when generating approximate values ​​for each block as described above, thereby reducing the number of block size and prediction mode combinations that need to be considered for each block. The generated intermediate pixels are then used as reference pixels, and the remaining combinations of block sizes and prediction modes can be evaluated in parallel for each block to select a final block size and a final prediction mode for each block of the video frame. After selecting the final block size and final prediction mode for each block of the video frame, the codec sequentially calculates and encodes the final residual, final prediction size, and final prediction mode for each block. Advantages of the embodiments of the present disclosure over the prior art include a significant acceleration of the encoding process. The initial and final sequential processing stages can be executed much faster than conventional techniques: the initial sequential processing is restricted to a much smaller number of fixed block sizes (e.g., one), while the final sequential processing is restricted to the final size selected for each block and the final mode selected.

[0016] System Architecture

[0017] Figure 1 1 is a schematic block diagram of an example computing device 100 that can implement the disclosed methods for efficient encoding operations, according to at least some embodiments. The computing device 100 may include one or more devices, modules, and / or applications that interact with a video encoder 105, such as a camera 101, video editing software 102, an internet browser 103, and / or media playback software 104, among others. The video encoder 105 may perform at least some aspects of the present disclosure. The video encoder 105 may include a reference pixel module 106, a block prediction module 107, a cost function module 110, a compression module 111, and a memory 112. The block prediction module 107 may include an intra-frame prediction submodule 108 and an inter-frame prediction submodule 109.

[0018] Reference pixel module 106 may store predicted block pixel values ​​in memory 112 as they are created. Reference pixel module 106 may also retrieve from memory 112 reference pixel values ​​currently used to approximate the block being processed by video encoder 105. For example, reference pixel module 106 may retrieve from memory 112 a subset of calculated pixel values ​​for blocks located to the top, left, and top-left corner of the currently processed block for use in calculating costs for one or more intra-prediction modes. Reference pixel module 106 may also retrieve from memory 112 calculated pixel values ​​for blocks in previously processed frames or subsequent frames corresponding to the currently processed block or blocks for use in calculating costs for one or more inter-prediction modes. Because inter-prediction modes are often used for moving objects, reference pixel module 106 may retrieve pixel values ​​from previously processed frames, for example, from blocks adjacent to the current block. For example, reference pixel module 106 may retrieve from memory 112 calculated pixel values ​​for blocks located to the right of the currently processed block in a previously processed frame.

[0019] When calculating the prediction for each block, block prediction module 107 can use reference pixel values ​​from reference pixel module 106. Intra-prediction submodule 108 can be used to perform intra-prediction, while inter-prediction submodule 109 can be used to perform inter-prediction. Block prediction module 107 can be used during the initial sequential processing, parallel processing, and final sequential processing described above. In some cases, block prediction module 107 can use one of the submodules to create the block prediction. For example, during initial sequential processing, block prediction module 107 can calculate the intra prediction and use intra-prediction submodule 108 instead of inter-prediction submodule 109, whereas during parallel processing, block prediction module 107 can use both intra-prediction submodule 108 and inter-prediction submodule 109.

[0020] When creating an intra-prediction block, the intra-prediction submodule 108 can use reference pixel values ​​from the reference pixel module 106, as described above. The reference pixel values ​​can come from one or more blocks adjacent to the current block. Depending on the intra-prediction mode used, the intra-prediction block created by the intra-prediction submodule 108 may have different pixel values. For example, the intra-prediction mode can be a vertical mode, so that the predicted block has pixel values ​​obtained from the top neighboring block. Alternatively, the intra-prediction mode can be a horizontal mode, so that the predicted block has pixel values ​​obtained from the left neighboring block. As another example, the intra-prediction mode can use different directions (e.g., at a certain angle to the horizontal direction) so that the predicted block has a combination of pixel values ​​from multiple neighboring blocks.

[0021] When creating an inter-prediction block, the inter-prediction submodule 109 can use reference pixel values ​​from the reference pixel module 106, as described above. The reference pixel values ​​can be from the same block in a previously processed frame, or from a block adjacent to the current block in a previously processed frame. In some cases, the reference pixel values ​​can be from a block that is shifted relative to (but partially overlaps with) the current block. Thus, similar to an intra-prediction block, the inter-prediction block created by the inter-prediction submodule 109 can have different pixel values ​​depending on the particular intra-prediction mode used.

[0022] The block prediction module 107 can generate multiple predictions for a given block. Each prediction can be based on a unique combination of block size and prediction mode for the current block. The number of predictions generated for each block can vary between the initial sequential processing, the parallel processing, and the final sequential processing. For example, during the initial sequential processing, each prediction can have the same block size but a different intra prediction mode, while during the parallel processing, at least some or even most predictions can have different sizes and different prediction modes (whether intra or inter prediction mode). Each prediction is passed to the cost function module 110 to calculate the corresponding cost.

[0023] Cost function module 110 may receive predictions for one or more blocks from block prediction module 107. For each prediction, the pixel values ​​of the predicted block are compared to the pixel values ​​of the corresponding source block of the original video frame being encoded. Various cost functions may be used, including the sum of absolute transformed differences (SATD), the sum of squared differences (SSD), or any other suitable cost function. In some cases, cost function module 110 may calculate the cost using one component of the pixel value during initial sequential processing and multiple components during parallel processing. For example, cost function module 110 may calculate the cost of the intra prediction for the initial sequential processing using the luma component of each pixel value. Then, during parallel processing, cost function module 110 may calculate the cost of the block prediction using both the luma component and the chroma component of each pixel value. In another embodiment, cost function module 110 may calculate the cost of the prediction for both the initial sequential processing and the parallel processing using the luma component of each pixel value and ignoring the chroma component of each pixel value. In another embodiment, cost function module 110 may calculate the cost of the prediction for both the initial sequential processing and the parallel processing using both the luma component and the chroma component of each pixel value. In some embodiments, the cost function module 110 may weight the errors in the representations of the luma and chroma components differently, e.g., using weights selected empirically. Furthermore, errors in different colors of the chroma components may be weighted differently from one another, e.g., errors in red may be weighted more heavily than errors in blue (or vice versa).

[0024] If the predicted block is significantly different from the corresponding source block, the cost may be high. If the predicted block is substantially similar to the corresponding source block, the cost may be low. Cost function module 110 may select a prediction for a given block based on a comparison of the costs of each prediction for the block. In one embodiment, cost function module 110 may select the prediction with the lowest cost. The prediction selected for a given block may include the block size and the prediction mode.

[0025] After selecting a prediction for a given block, the cost function module 110 can pass the selection to the block prediction module 107. The block prediction module 107 can then pass the prediction block corresponding to the selected prediction to the reference pixel module 106 for storage in the memory 112 for use as a reference block for future blocks and / or frames. The block prediction module 107 can also pass the prediction block corresponding to the selected prediction to the compression module 111.

[0026] The compression module 111 may calculate a residual, which may be a block of residual pixel values ​​representing the difference between the pixel values ​​of the prediction block and the pixel values ​​of the corresponding source block. The compression module 111 may then encode the residual pixel values, the prediction block size, and the prediction block mode in the output bitstream according to specifications defined in a video codec standard, such as AV1 or any other suitable standard. In some embodiments, the compression module 111 may transform and compress the residual pixel values ​​according to specifications defined in the video codec standard before encoding the values ​​in the output bitstream.

[0027] Figure 2A is a diagram illustrating the operation of a sequential process 200 of a codec system for efficient encoding operations according to at least some embodiments. In some embodiments, the codec system may be Figure 1 1. As described above, sequential processing 200 can be used to generate intermediate pixels 204 for subsequent parallel processing. A video frame 202 can be divided into blocks of a first size (e.g., 16x16 pixel blocks or any other suitable blocks). A first target region indicator 206 (depicted by a dashed box and a first magnified target region 208) shows a current block 210 (block N) being processed and portions of blocks adjacent to the current block 210. Blocks above and to the left of the current block 210 in the video frame 202 may have already been processed sequentially, and corresponding blocks of intermediate pixels may have already been generated, as shown by the line-shaded blocks in the video frame 202. Each block of the video frame 202 corresponds to (and approximates) a similarly positioned block of source pixels in the original video frame.

[0028] During processing of the current block 210, a first set of reference pixels 212 may be selected from the intermediate pixels of the generated blocks adjacent to the current block 210. Specifically, the first set of reference pixels 212 may be selected from the generated intermediate pixel blocks to the left, top, and upper left corner of the current block 210 (or from any other set of blocks specified by the codec standard). This set of reference pixels 212 may be combined with each of a plurality of patterns 214A-M (Mode 1, Mode 2, ..., Mode M) to create prediction blocks 216A-M (P1, P2, ..., PM). Each of the prediction blocks 216A-M may then be processed using a cost function 218 to determine a cost 220A-M (Cost 1, Cost 2, ..., Cost M) for each of the prediction blocks 216A-M. The cost function 218 may compare the prediction blocks to corresponding source pixel blocks in the video frame 202. Based on the comparison of the costs 220A-M, a selected mode 222 may be determined for the current block 210. The intermediate pixel generator 224 may generate the intermediate pixel block 204 for the current block 210. The intermediate pixel block 204 may be added to the video frame 202 at the location of the current block 210.

[0029] Sequential processing can continue with the next block 230 (e.g., the block to the right of the current block 210), as indicated by the second target region indicator 226 and the second magnified target region 228. During processing of the next block 230, a second set of reference pixels 232 can be selected from the intermediate pixels of the generated blocks adjacent to the next block 230. For example, a portion of the intermediate pixel block 204 created during processing of the current block 210 can be used as part of the second set of reference pixels 232. The intermediate pixel block can be generated for the next block 230 in a manner similar to that described above. This process can continue until each block of the first size in the video frame 202 has a corresponding intermediate pixel block.

[0030] Figure 2BFIG2 is a schematic diagram illustrating the operation of another possible sequential process 250 of a codec system for efficient encoding operations, according to at least some embodiments. In some embodiments, the codec system may be the video encoder 105. Sequential process 250 may be used to generate intermediate pixels for subsequent parallel processing in a manner similar to sequential process 200. As described above, sequential process 250 may also use reference frame 282, reference block N 284, and inter-frame mode 280. Video frame 202 may be divided into blocks of a first size (e.g., 16x16 pixels). A first target region indicator 206 illustrates a current block 210 (block N) being processed and portions of blocks adjacent to current block 210. Next to video frame 202 is a first magnified target region 208. Blocks above and to the left of current block 210 in video frame 202 may have already been sequentially processed, and corresponding intermediate pixel blocks may have already been generated, as indicated by the line-shaded blocks in video frame 202.

[0031] During processing of the current block in sequential processing 250, a first set of reference pixels 212 can be selected from generated intermediate pixels of blocks adjacent to the current block 210. Specifically, the first set of reference pixels 212 can be selected from a generated intermediate pixel block located to the left, top, and upper left corner of the current block 210. Furthermore, a reference block 284 (reference block N) can be selected from generated intermediate pixels of a reference frame 282. A third target region indicator 286 displays the reference block 284 in the reference frame 282. Below the reference frame 282 is a third magnified target region 288 displaying the reference block 284. The set of reference pixels 212 can be combined with each of the multiple modes 214A-M to create a prediction block 216A-M. The reference block 284 can also be combined with the inter mode 280 to create a prediction block 290. Each of the prediction blocks 216A-M and the prediction block 290 may then be processed using a cost function 218 to determine costs 220A-M and 292 for each of the prediction blocks 216A-M and the prediction block 290, respectively. The cost function 218 may compare the prediction blocks with corresponding source pixel blocks from the video frame 202. A selected mode 294 may then be determined for the current block 210 based on the comparison of the costs 220A-M and 292. An intermediate pixel block may be generated for the current block 210 using the selected mode 294 and the reference block 284 by the intermediate pixel generator 224.

[0032] In another embodiment, the initial sequential processing may divide the video frame into blocks of a first size and select corresponding source pixel blocks from the video frame as intermediate pixels of each block.

[0033] After each block of the video frame has a corresponding intermediate pixel block (e.g., after initial sequential processing), the codec system can continue with parallel processing. The codec system can divide the video frame into blocks of a second size, blocks of a third size, and so on. The codec system can then process each block in parallel in a manner similar to the initial sequential processing. In some embodiments, instead of using the values ​​of neighboring blocks to calculate the predicted block in parallel processing, the values ​​of the neighboring blocks from the generated intermediate blocks are used. This eliminates the dependency between the current block and the previous block, allowing the codec system to process the blocks in parallel. For example, the codec system can begin processing the second and third size blocks at the top of the video frame using a first thread, and can simultaneously begin processing the second and third size blocks at the bottom of the video frame using a second thread.

[0034] In some embodiments, one difference between the initial sequential processing and the parallel processing is that for each block in the parallel processing, a prediction size and prediction mode are selected, but no intermediate pixel blocks are generated. The generation of the prediction blocks using the prediction size and prediction mode selected during the parallel processing is deferred until the final sequential processing.

[0035] Final sequential processing can be performed similarly to initial sequential processing. In some embodiments, instead of dividing the video frame into blocks of equal size, the video frame is divided into blocks based on a selected prediction size for each block. Then, block by block, the codec system generates a final prediction block using the prediction mode for the selected block and the pixel values ​​from the adjacent, previously processed final prediction block. During the final sequential processing, no cost is calculated for each block. As described above, once the final prediction block is created, the residual is calculated, and the codec system creates a representation (e.g., an image) of the video frame by encoding the residual, the prediction mode, and the prediction size in the output bitstream.

[0036] Figure 3A is a flow chart of an example method 300 for processing video frames in parallel during video encoding for efficient encoding operations according to at least some embodiments. Figure 1The video encoder 105 performs method 300. In some embodiments, method 300 may be performed by a dedicated codec accelerator (e.g., a coprocessor) that communicates with one or more storage devices. In some embodiments, method 300 may be performed by a general-purpose processor (e.g., a central processing unit or a graphics processing unit). In some embodiments, at least some operations of method 300 may be performed by multiple threads (e.g., in parallel), each thread executing one or more separate functions, routines, subroutines, or operations of the method. In some embodiments, the processing threads implementing method 300 may be synchronized (e.g., using semaphores, critical sections, and / or other thread synchronization mechanisms). Alternatively, the threads implementing method 300 may execute asynchronously with each other. The various operations of method 300 may be performed in accordance with Figure 3A Some operations of method 300 may be performed concurrently with other operations. In some embodiments, the operations of method 300 may not be performed concurrently with other operations. Figure 3A One or more operations as shown.

[0037] For the sake of brevity, the following description of method 300 refers to images. It should be understood that in some embodiments, the images may be video frames, such as frames from a sequence of frames in a motion picture, a video advertisement, a video game, or any other video file (including composite video files) generated by a camera or any suitable video application. In other embodiments, the images may be individual images (including composite images), such as still images, images generated by a camera, a scanner, a graphics application, or any other suitable image being compressed.

[0038] At block 310, processing logic executing method 300 may determine an intermediate pixel (e.g., Figure 2A In one embodiment, the middle pixel can be a combination of the above Figure 2A The result of the initial sequential processing described. For example, the following Figure 3B The intermediate pixels are determined by the method 370 shown in FIG. In another embodiment, the intermediate pixels may be pixels of the source video frame.

[0039] At block 320, processing logic may associate each position of the image with a block of a plurality of blocks of a first size and a block of a plurality of blocks of a second size. In some embodiments, each position may be further associated with a block of a third size, a fourth size, and so on. The first block size and the second block size (and other block sizes) may each be a block size defined by a standard video codec such as AV1. For example, each superblock of 64x64 pixels may be partitioned into blocks of a first size (32x32 pixels), a second size (16x8 pixels), a third size (8x8 pixels), a fourth size (32x16 pixels), and so on.

[0040] At block 330, processing logic may determine, in parallel, a first cost for the first mode and a second cost for the second mode for each block of the first size and each block of the second size (and other block sizes) using the intermediate pixels. Processing logic may further determine additional costs for additional modes (e.g., third / fourth / etc. costs for third / fourth / etc. modes), for example, as associated with modes 214A-M and costs 220A-M. Figure 2B As shown. Block 330 may be performed as part of a parallel process. In some cases, the first mode and the second mode may be an intra-prediction mode and an inter-prediction mode, respectively, and a cost for each mode may be calculated by the cost function module 110. For example, the first mode may be intra-prediction mode 214A (associated with cost 220A) and the second mode may be inter-prediction mode 280 (associated with cost 292). In another example, the first mode and the second mode may both be intra-prediction modes. In some embodiments, for each block of a given size (first / second / etc.), a corresponding cost may be determined for all modes or at least a subset of the modes identified by the codec standard.

[0041] In certain embodiments, as Figure 3A As shown in the upper labeled portion of FIG, at block 332, for each of the first mode and the second mode in block 330, the processing logic may generate a temporary block prediction based on the respective mode and a set of intermediate pixels. Thus, a temporary block prediction may be generated for each block of a given size.

[0042] At block 334, for each of the first and second modes of block 330, processing logic may calculate a cost for the respective mode based on a comparison of the temporary block prediction and a corresponding source pixel block of the image. More specifically, for each block of a given size, multiple temporary block predictions may be obtained, each prediction being associated with a respective cost.

[0043] At block 340, processing logic may, for each location of the image, select a final mode from at least the first mode and the second mode using the first cost and the second cost for each of the corresponding blocks of the first size and the corresponding blocks of the second size associated with the corresponding location; and select a final block size from at least the first size and the second size. For example, processing logic may determine how to divide each 64x64 pixel superblock into smaller blocks and what mode to use to represent the source pixels of the respective blocks, e.g., based on the costs calculated for each block / mode combination. For example, the cost of representing the entire top left square 32x32 pixel block using inter prediction mode may be compared to the cost of representing the same square 32x32 pixel block by further dividing it into two rectangular 32x16 pixel blocks, where the top rectangular 32x16 pixel block is represented using horizontal intra prediction mode and the bottom rectangular 32x16 pixel block is represented using oblique prediction mode.

[0044] At block 350, processing logic may determine the final pixel based on the final mode selected and the final block size selected for each location of the image. Figure 3A As described in the subscripted portion of , at block 352, processing logic may generate an initial prediction block for each position of the image based on the selected final mode, the selected final block size, and a set of previously determined final pixels associated with one or more neighboring positions. For example, the initial prediction block may be a winning block selected from the temporary block predictions based on a cost function.

[0045] At block 354, processing logic may derive a final pixel block for each location of the image by transforming the initial prediction block using a compression algorithm. For example, compression module 111 may determine a residual block for each selected winning block, transform the residual block using one of the discrete transforms identified by the codec standard (e.g., a discrete Fourier transform), quantize the transformed block, and perform an inverse discrete transform to obtain the final pixel block. Processing logic may further determine parameters for various filters that may be applied to the final pixel block to improve the visual quality of the block and remove various artifacts of block reconstruction. For example, a deblocking filter may remove visible block boundaries between adjacent blocks, a constrained directional enhancement filter may remove artifacts in sharp edge descriptions, a loop recovery filter may reduce noise and improve edge quality, and so on.

[0046] The processing logic may obtain a representation of the image based on the final pixels at block 360. For example, the representation of the image may be an encoded output bitstream of the image that includes identification of selected block sizes for portions of the image, a selected mode for each block, a representation of the residual block, parameters of filters selected for each block, and the like.

[0047] Figure 3Bis a flow chart of an example method 370 for determining intermediate pixels for use in parallel processing of video frames during video encoding to achieve efficient encoding operations in accordance with at least some embodiments. In one embodiment, the method 370 may be performed as part of block 310. The method 370 may correspond to the above-described method in conjunction with Figure 2A The initial order of processing discussed and can be Figure 1 The video encoder 105 is executed.

[0048] At block 312, processing logic may divide the image into a plurality of blocks of a predetermined size, referred to herein as a third size. For example, the image may be divided into blocks of size 16x16 pixels, 32x32 pixels, 8x8 pixels, or any other size (including non-square block sizes, such as 16x8 pixels).

[0049] At block 314, processing logic may include sequentially processing each block of the third size to determine a third cost for the first mode and a fourth cost for the second mode, wherein the third and fourth costs are determined using a set of intermediate pixels obtained from a previously processed block. In one embodiment, the first and second modes may be intra-prediction modes and inter-prediction modes, respectively, and the cost of each mode may be calculated by cost function module 110. In another embodiment, the first and second modes may both be intra-prediction modes. At block 316, processing logic may select an intermediate mode for each block of the third size based on a comparison of the third cost of the first mode and the fourth cost of the second mode. At block 318, processing logic may use the selected intermediate mode to obtain a set of intermediate pixels for the corresponding block of the third size.

[0050] Figure 4 An example machine 400 in the form of a computer system is shown. The computer system 400 executes one or more sets of instructions 426 that cause the machine to perform any one or more of the methods discussed herein. The machine can operate as a server or client machine in a client-server network environment, or as a peer machine in a peer-to-peer (or distributed) network environment. The machine can be a personal computer (PC), a tablet computer, a set-top box (STB), a personal digital assistant (PDA), a mobile phone, a network appliance, a server, a network router, a switch or a bridge, or any machine capable of executing a set of instructions (sequential or otherwise) that specify the actions to be taken by the machine. In addition, although only a single machine is shown, the term "machine" should also be considered to include any machine collection that executes the instruction set 426 individually or in combination to perform any one or more of the methods discussed herein.

[0051] The computer system 400 includes a processor 402, a main memory 404 (e.g., read-only memory (ROM), flash memory, dynamic random access memory (DRAM) such as synchronous DRAM (SDRAM) or Rambus DRAM (RDRAM), etc.), a static memory 406 (e.g., flash memory, static random access memory (SRAM), etc.), and a data storage device 416, which communicate with each other via a bus 408.

[0052] Processor 402 represents one or more general-purpose processing devices, such as a microprocessor, a central processing unit, or similar devices. More specifically, processor 402 can be a complex instruction set computing (CISC) microprocessor, a reduced instruction set computing (RISC) microprocessor, a very long instruction word (VLIW) microprocessor, or a processor that implements other instruction sets or a processor that implements a combination of instruction sets. Processor 402 can also be one or more special-purpose processing devices, such as an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA), a digital signal processor (DSP), a network processor, or similar devices. Processor 402 is configured to execute instructions of the network node to perform the operations and steps discussed herein.

[0053] The computer system 400 may also include a network interface device 422 that provides communication with other machines via a network 418, such as a local area network (LAN), an intranet, an extranet, or the Internet. The computer system 400 may also include a display device 410, such as a liquid crystal display (LCD) or a cathode ray tube (CRT), an alphanumeric input device 412, such as a keyboard, a cursor control device 414, such as a mouse, and a signal generating device 420, such as a speaker.

[0054] The data storage device 416 may include a computer-readable storage medium 424 on which is stored a network node instruction set 426 embodying any one or more of the methods or functions described herein. During execution of the network node instruction set 426 by the computer system 400, the network node instruction set 426 may also reside completely or at least partially within the main memory 404 and / or the processor 402, which also constitute computer-readable storage media. The instruction set 426 may further be transmitted or received via the network 418 via the network interface device 422.

[0055] Although the example of computer-readable storage medium 424 is shown as a single medium, the term "computer-readable storage medium" may include a single medium or multiple media (e.g., a centralized or distributed database, and / or associated caches and servers) that store the instruction set 426. The term "computer-readable storage medium" may include any medium that can store, encode, or carry a set of instructions for execution by a machine and cause the machine to perform any one or more of the methods of the present disclosure. The term "computer-readable storage medium" may include, but is not limited to, solid-state memory, optical media, and magnetic media.

[0056] Images and videos generated using one or more of the techniques disclosed herein can be displayed on a monitor or other display device. In some embodiments, the display device can be directly coupled to the system or processor that generates or presents the images or videos. In other embodiments, the display device can be indirectly coupled to the system or processor, such as via a network. Examples of such networks include the Internet, mobile communication networks, Wi-Fi networks, and any other wired and / or wireless networked systems. When the display device is indirectly coupled, the images or videos generated by the system or processor can be streamed to the display device via the network. This streaming allows, for example, video games or other applications that render images or videos to be executed on a server or in a data center, and the presented images and videos to be transmitted and displayed on one or more user devices (such as computers, video game consoles, smartphones, other mobile devices, etc.) that are physically separated from the server or data center. Therefore, the techniques disclosed herein can be used to enhance streamed images or videos and enhance services that stream images and videos, such as NVIDIA GeForce Now (GFN) and Google Stadia.

[0057] In addition, images and videos generated using one or more of the techniques disclosed herein can be used to train, test, or validate deep neural networks (DNNs) that are used to recognize objects and environments in the real world. Such images and videos can include scenes of roads, factories, buildings, urban environments, rural environments, humans, animals, and any other physical objects or real-world settings. Such images and videos can be used to train, test, or certify DNNs used in machines or robots to manipulate, process, or modify physical objects in the real world. In addition, such images and videos can be used to train, test, or certify DNNs used in autonomous vehicles to navigate and move vehicles through the real world. In addition, images and videos generated using one or more of the techniques disclosed herein can be used to convey information to users of these machines, robots, and vehicles.

[0058] Other variations are within the spirit of the present disclosure. Therefore, while the disclosed technology is susceptible to various modifications and alternative constructions, certain illustrated embodiments thereof are shown in the drawings and have been described above in detail. However, it should be understood that there is no intention to limit the disclosure to one or more specific forms disclosed, but on the contrary, the intention is to cover all modifications, alternative constructions, and equivalents falling within the spirit and scope of the present disclosure as defined by the appended claims.

[0059] Unless otherwise indicated or clearly contradicted by context, the use of the terms "a," "an," "the," and similar references in the context of describing the disclosed embodiments (particularly in the context of the claims below) should be interpreted as covering the singular and the plural, and not as definitions of terms. Unless otherwise indicated, the terms "comprising," "having," "inlcuding," and "containing" should be interpreted as open-ended terms (meaning "including, but not limited to"), unless otherwise indicated. The term "connected" (when unmodified, referring to a physical connection) should be interpreted as partially or completely contained within, attached to, or connected together, even if there are some intervening elements. Unless otherwise indicated herein, references to numerical ranges herein are intended merely to serve as a shorthand method of referring individually to each individual value falling within the range, and each individual value is incorporated into the specification as if it were individually recited herein. In at least one embodiment, the use of the term "set" (e.g., "set of items") or "subset" should be interpreted as a non-empty set comprising one or more members, unless otherwise indicated or contradicted by context. Furthermore, unless otherwise indicated or contradicted by context, the term "subset" of a corresponding set does not necessarily mean a proper subset of the corresponding set, but rather a subset and a corresponding set may be equivalent.

[0060] Unless expressly indicated otherwise or clearly contradicted by context, conjunctions such as phrases of the form "at least one of A, B, and C" or "at least one of A, B, and C" are understood in context to generally refer to an item, clause, or the like, which may be A or B or C, or any non-empty subset of the set of A, B, and C. For example, in the illustrative example of a set having three members, the conjunctions "at least one of A, B, and C" and "at least one of A, B, and C" refer to any of the following sets: {A}, {B}, {C}, {A, B}, {A, C}, {B, C}, {A, B, C}. Thus, such conjunctions are not generally intended to imply that certain embodiments require the presence of at least one of A, at least one of B, and at least one of C. Additionally, unless expressly indicated otherwise or contradicted by context, the term "plurality" refers to plurality (e.g., "a plurality of items" refers to a plurality of items). In at least one embodiment, the number of items in the plurality is at least two, but may be more if expressly indicated or indicated by context. Further, unless stated otherwise or clear from context, the phrase "based on" means "based at least in part on" rather than "based solely on."

[0061] Unless otherwise indicated herein or clearly contradicted by the context, the operations of the processes described herein may be performed in any suitable order. In at least one embodiment, processes such as those described herein (or variations and / or combinations thereof) are performed under the control of one or more computer systems configured with executable instructions and are implemented as code (e.g., executable instructions, one or more computer programs, or one or more applications) that are collectively executed on one or more processors by hardware or a combination thereof. In at least one embodiment, the code is stored on a computer-readable storage medium in the form of, for example, a computer program that includes a plurality of instructions that can be executed by one or more processors. In at least one embodiment, the computer-readable storage medium is a non-transitory computer-readable storage medium that excludes transient signals (e.g., propagated transient electrical or electromagnetic transmissions) but includes non-transitory data storage circuits (e.g., buffers, caches, and queues) within a transceiver for transient signals. In at least one embodiment, code (e.g., executable code or source code) is stored on a set of one or more non-transitory computer-readable storage media (or other memory for storing executable instructions) having executable instructions stored thereon, which, when executed by one or more processors of a computer system (i.e., as a result of being executed), causes the computer system to perform the operations described herein. In at least one embodiment, the set of non-transitory computer-readable storage media includes a plurality of non-transitory computer-readable storage media, and one or more of the individual non-transitory storage media in the plurality of non-transitory computer-readable storage media lacks all of the code, while the plurality of non-transitory computer-readable storage media collectively store all of the code. In at least one embodiment, the executable instructions are executed so that different instructions are executed by different processors, for example, a non-transitory computer-readable storage medium stores instructions, and a main central processing unit ("CPU") executes some instructions, while a graphics processing unit ("GPU") executes other instructions. In at least one embodiment, different components of the computer system have separate processors, and different processors execute different subsets of instructions.

[0062] Thus, in at least one embodiment, a computer system is configured to implement one or more services that individually or collectively perform the operations of the processes described herein, and such a computer system is configured with applicable hardware and / or software that enables the execution of the operations. Furthermore, the computer system that implements at least one embodiment of the present disclosure is a single device, and in another embodiment is a distributed computer system that includes multiple devices that operate in different ways, such that the distributed computer system performs the operations described herein, and such that no single device performs all of the operations.

[0063] The use of any and all examples or exemplary language (e.g., "such as") provided herein is intended merely to better illuminate embodiments of the present disclosure and does not limit the scope of the disclosure unless otherwise required. No language in the specification should be construed as indicating any non-claimed element as essential to the practice of the disclosure.

[0064] All references, including publications, patent applications, and patents, cited herein are hereby incorporated by reference to the same extent as if each reference were individually and specifically indicated to be incorporated by reference and were set forth in its entirety herein.

[0065] In the description and claims, the terms "coupled" and "connected," along with their derivatives, may be used. It should be understood that these terms may not be intended as synonyms for each other. Rather, in specific examples, "connected" or "coupled" may be used to indicate that two or more elements are in direct or indirect physical or electrical contact with each other. "Coupled" may also mean that two or more elements are not in direct contact with each other, but still cooperate or interact with each other.

[0066] Unless otherwise expressly stated, it is understood that throughout the specification, terms such as “processing,” “computing,” “calculating,” “determining,” etc. refer to the actions and / or processes of a computer or computing system or similar electronic computing device that processes and / or transforms data represented as physical quantities (e.g., electronic) in the registers and / or memories of the computing system into other data similarly represented as physical quantities in the memories, registers, or other such information storage, transmission, or display devices of the computing system.

[0067] In a similar manner, the term "processor" may refer to any device or portion of a device that processes electronic data from registers and / or memory and transforms that electronic data into other electronic data that can be stored in registers and / or memory. As non-limiting examples, a "processor" may be a CPU or a GPU. A "computing platform" may include one or more processors. As used herein, a "software" process may include, for example, software and / or hardware entities that perform work over time, such as tasks, threads, and intelligent agents. Likewise, each process may refer to multiple processes to execute instructions sequentially or in parallel, continuously or intermittently. In at least one embodiment, the terms "system" and "method" may be used interchangeably herein, as long as a system may embody one or more methods, and a method may be considered a system.

[0068] In this document, reference may be made to obtaining, acquiring, receiving, or inputting analog or digital data into a subsystem, computer system, or computer-implemented machine. In at least one embodiment, the process of obtaining, acquiring, receiving, or inputting analog and digital data can be accomplished in a variety of ways, such as by receiving data as parameters of a function call or a call to an application programming interface. In at least one embodiment, the process of obtaining, acquiring, receiving, or inputting analog or digital data can be accomplished by transmitting data via a serial or parallel interface. In at least one embodiment, the process of obtaining, acquiring, receiving, or inputting analog or digital data can be accomplished by transmitting data from a providing entity to an acquiring entity via a computer network. In at least one embodiment, reference may also be made to providing, outputting, transmitting, sending, or presenting analog or digital data. In various examples, the process of providing, outputting, transmitting, sending, or presenting analog or digital data can be accomplished by transmitting data as input or output parameters of a function call, an application programming interface, or an interprocess communication mechanism.

[0069] Although the above discussion sets forth example embodiments of the described technology, other architectures may be used to implement the described functionality and are intended to fall within the scope of this disclosure. In addition, although specific responsibilities are defined above for descriptive purposes, the various functions and responsibilities may be allocated and divided in different ways depending on the circumstances.

[0070] Furthermore, although the subject matter has been described in language specific to structural features and / or methodological acts, it is to be understood that the subject matter claimed in the appended claims is not necessarily limited to the specific features or acts described. Rather, the specific features and acts are disclosed as example forms of implementing the claims.

Claims

1. A method for compressing image data, the method comprising: Determine the middle pixel; associating each location of the image with a block in a plurality of blocks of a first size and a block in a plurality of blocks of a second size; determining, in parallel, for each block of the first size and for each block of the second size, a first cost for the first mode and a second cost for the second mode using the intermediate pixels; for each position of the image, using the first cost and the second cost for each of the corresponding blocks of the first size and the corresponding blocks of the second size associated with the corresponding position, i) selecting a final mode from at least the first mode and the second mode, and ii) selecting a final block size from at least the first size and the second size; determining a final pixel based on the selected final mode and the selected final block size for each location of the image; as well as Based on the final pixels, a representation of the image is obtained.

2. The method of claim 1 , wherein determining the intermediate pixel comprises: Divide the image into a plurality of blocks of a third size: sequentially processing each block of the third size to determine a third cost for the first mode and a fourth cost for the second mode, wherein determining the third cost and the fourth cost uses a set of intermediate pixels obtained for a previously processed block; selecting, for each block of the third size, an intermediate mode based on a comparison of the third cost of the first mode and a fourth cost of the second mode; as well as Using the selected intermediate mode, a set of intermediate pixels is obtained for the corresponding block of the third size.

3. The method of claim 1 , wherein determining the intermediate pixel comprises: dividing the image into a plurality of blocks of a third size; and For each block of the third size, a corresponding block of source pixels is selected from the image.

4. The method of claim 1 , wherein determining the first cost of the first mode and the second cost of the second mode comprises, for each of the first mode and the second mode: generating a temporary block prediction based on the corresponding pattern and a set of intermediate pixels; and Based on a comparison of the temporary block prediction and a corresponding source pixel block of the image, a cost of the corresponding mode is calculated.

5. The method of claim 1 , wherein for each location of the image, determining the final pixel comprises: generating an initial prediction block based on the selected final mode, the selected final block size, and a set of previously determined final pixels associated with one or more neighboring locations; as well as The final pixel block is derived by transforming the initial prediction block using a compression algorithm.

6. The method of claim 5, wherein the compression algorithm is based on one of the AV1, HEVC, VVC or VP9 coding standards. 7 . The method of claim 1 , wherein the first mode is one of a plurality of intra-prediction modes, and the second mode is an inter-prediction mode.

8. The method according to claim 7, wherein the inter prediction mode: approximating each block of the first size with a first reference block of the first size in a first reference image; and Each block of the second size is approximated with a second reference block of the second size in a second reference image.

9. A system for compressing image data, comprising: Memory; as well as A processor coupled to the memory, configured to perform the following operations, including: Determine the middle pixel of the image; associating each location of the image with a block in a plurality of blocks of a first size and a block in a plurality of blocks of a second size; determining, in parallel, for each block of the first size and for each block of the second size, a first cost for the first mode and a second cost for the second mode using the intermediate pixels; for each position of the image, using the first cost and the second cost for each of the corresponding blocks of the first size and the corresponding blocks of the second size associated with the corresponding position, i) selecting a final mode from at least the first mode and the second mode, and ii) selecting a final block size from at least the first size and the second size; determining a final pixel based on the selected final mode and the selected final block size for each location of the image; as well as Based on the final pixels, a representation of the image is obtained.

10. The system of claim 9, wherein determining the intermediate pixel comprises: Divide the image into a plurality of blocks of a third size: sequentially processing each block of the third size to determine a third cost for the first mode and a fourth cost for the second mode, wherein determining the third cost and the fourth cost uses a set of intermediate pixels obtained for a previously processed block; selecting, for each block of the third size, an intermediate mode based on a comparison of the third cost of the first mode and a fourth cost of the second mode; as well as Using the selected intermediate mode, a set of intermediate pixels is obtained for the corresponding block of the third size.

11. The system of claim 9, wherein determining the first cost of the first mode and the second cost of the second mode comprises, for each of the first mode and the second mode: generating a temporary block prediction based on the corresponding pattern and a set of intermediate pixels; and Based on a comparison of the temporary block prediction and a corresponding source pixel block of the image, a cost of the corresponding mode is calculated.

12. The system of claim 9, wherein for each location of the image, determining the final pixel comprises: generating an initial prediction block based on the selected final mode, the selected final block size, and a set of previously determined final pixels associated with one or more neighboring locations; as well as The final pixel block is derived by transforming the initial prediction block using a compression algorithm.

13. The system of claim 12, wherein the compression algorithm is based on one of the AV1, HEVC, VVC, or VP9 coding standards.

14. The system of claim 9, wherein the first mode is one of a plurality of intra-prediction modes and the second mode is an inter-prediction mode.

15. A non-transitory computer-readable medium comprising instructions that, when executed by a processor, cause the processor to perform the following operations, comprising: Determine the middle pixel of the image; associating each location of the image with a block in a plurality of blocks of a first size and a block in a plurality of blocks of a second size; determining, in parallel, for each block of the first size and for each block of the second size, a first cost for the first mode and a second cost for the second mode using the intermediate pixels; for each position of the image, using the first cost and the second cost for each of the corresponding blocks of the first size and the corresponding blocks of the second size associated with the corresponding position, i) selecting a final mode from at least the first mode and the second mode, and ii) selecting a final block size from at least the first size and the second size; determining a final pixel based on the selected final mode and the selected final block size for each location of the image; as well as Based on the final pixels, a representation of the image is obtained.

16. The non-transitory computer-readable medium of claim 15, wherein determining the intermediate pixel comprises: Divide the image into a plurality of blocks of a third size: sequentially processing each block of the third size to determine a third cost for the first mode and a fourth cost for the second mode, wherein determining the third cost and the fourth cost uses a set of intermediate pixels obtained for a previously processed block; selecting, for each block of the third size, an intermediate mode based on a comparison of the third cost of the first mode and a fourth cost of the second mode; as well as Using the selected intermediate mode, a set of intermediate pixels is obtained for the corresponding block of the third size.

17. The non-transitory computer-readable medium of claim 15, wherein determining the first cost of the first mode and the second cost of the second mode comprises, for each of the first mode and the second mode: generating a temporary block prediction based on the corresponding pattern and a set of intermediate pixels; and Based on a comparison of the temporary block prediction and a corresponding source pixel block of the image, a cost of the corresponding mode is calculated.

18. The non-transitory computer-readable medium of claim 15 , wherein for each location of the image, determining the final pixel comprises: generating an initial prediction block based on the selected final mode, the selected final block size, and a set of previously determined final pixels associated with one or more neighboring locations; and The final pixel block is derived by transforming the initial prediction block using a compression algorithm.

19. The non-transitory computer-readable medium of claim 18, wherein the compression algorithm is based on one of AV1, HEVC, VVC, or VP9 coding standards.

20. The non-transitory computer-readable medium of claim 15, wherein the first mode is one of a plurality of intra-prediction modes, and the second mode is an inter-prediction mode.

Images