Parallel coding of video frames without filter dependency

Abstract

A technique for compressing image data using multiple processing cores is disclosed. The technique includes: obtaining a first (second, etc.) plurality of reconstructed blocks of source pixels approximating a first (second, etc.) portion of an image using a first (second, etc.) processing core; and filtering the first plurality of reconstructed blocks using the first processing core. The filtering includes enabling one or more filters for a first plurality of regions of pixels including the first plurality of reconstructed blocks but excluding a second plurality of reconstructed blocks. The filtering also includes disabling the one or more filters for a second plurality of regions of pixels including both the first plurality of reconstructed blocks and the second plurality of reconstructed blocks.

Description

Technical Field

[0001] At least one embodiment relates to computational techniques for performing and facilitating efficient compression of video files. For example, at least one embodiment relates to operations by which a video codec efficiently encodes high-definition video files using modern video coding standards, providing a wealth of tools and options for enhancing video quality at the cost of increased computational complexity. Background Technology

[0002] Raw (source) pixel format video files can occupy a very large amount of storage space and may require significant network bandwidth and / or time for their network transmission. This often makes raw pixel format impractical for data storage and / or live streaming. For example, typical high-definition video displays approximately 30 frames per second, which are generally quite similar to each other. Much information in such frames is necessarily redundant, thus allowing for efficient compression. On the one hand, relatively small variations occur between subsequent frames (temporal redundancy). On the other hand, regions within a given frame are often similar to each other (spatial redundancy); for example, an image of the sky can extend across most of the frame. Therefore, instead of transmitting the actual pixel information (luminance and chrominance) for each pixel, the codec can identify reference blocks that resemble the blocks being encoded (“predicted”) and provide the decoder with a suitable and compact mathematical representation of the difference (“increment” or “residual”) between the actual source block and the predicted block. The reference block can be a block from different (e.g., previous or even subsequent) frames, a block from the same frame, or even a composite block generated according to a predetermined scheme (pattern) based on a small number of reference pixels. Subsequently, instead of storing or live streaming the actual frames of pixels, the codec can output a bitstream of encoded data that mainly contains approximate instructions to the decoder on how to generate frames that are visually indistinguishable from or very similar to the source frames. Attached Figure Description

[0003] Figure 1A This is a schematic block diagram of an example computing device that can implement the disclosed techniques for efficient coding operations according to at least some embodiments.

[0004] Figure 1B It is a schematic diagram depicting the filtering operation of a codec system for efficient encoding operations according to at least some embodiments.

[0005] Figure 2A It is a schematic diagram depicting the sequential processing operations of a codec system for efficient encoding operations according to at least some embodiments.

[0006] Figure 2B This is a schematic diagram depicting another possible sequential processing operation of a codec system for efficient encoding operations according to at least some embodiments.

[0007] Figure 3 This is a flowchart of an example method for filtering reconstructed blocks during multi-core processing of video frames during video encoding for efficient encoding operations, according to at least some embodiments.

[0008] Figure 4A This is a flowchart of an example method for parallel processing of video frames during video encoding for efficient encoding operations, according to at least some embodiments.

[0009] Figure 4B This is a flowchart of an example method for determining intermediate pixels in parallel processing of video frames during video encoding for efficient encoding operations, according to at least some embodiments.

[0010] Figure 5 An example machine in the form of a computer system is shown, within which a set of instructions can be executed to cause the machine to perform any or more of the methods discussed herein. Detailed Implementation

[0011] Video codecs allow digital video files to be compressed to reduce the amount of storage space and bandwidth required for storing and transmitting the files. When the source video file is uncompressed and displayed to a viewer, a compressed video file may look very similar to or be indistinguishable from the source video file. To compress a digital video file, the codec divides each frame of the video into a series of blocks. For each block, the codec determines the optimal size and optimal mode for compression. The codec may determine whether to use an intra-frame prediction mode or an inter-frame prediction mode. Then, for each block, after size and mode selection, the codec compares the pixels of the predicted block with the corresponding blocks of source pixels, calculates the difference (residual), and encodes the prediction size, prediction mode, and residual pixel values ​​in the output bitstream. A single frame can be divided into blocks of multiple sizes, and each block can have its unique prediction mode. Subsequent frames can be divided into blocks of different sizes, and each block can have a different prediction mode than the blocks of previous frames. The codec processes each frame individually and attempts to minimize the cost of encoding the frame.

[0012] Each pixel value in a frame may include multiple components, such as a luminance component that indicates the brightness or intensity of the pixel and a chromaticity component that indicates one or more colors of the pixel (e.g., red, green, blue chromaticity values ​​or any other suitable color value).

[0013] Intra-prediction blocks can approximate the pixel values ​​of a source block using reference pixel values ​​from neighboring blocks. For example, the pixel values ​​of an intra-prediction block can be based on pixel values ​​from the lower boundary of the block above it, pixel values ​​from the right boundary of the block to its left, pixel values ​​from the lower right corner of the block to its upper left, or a combination of these pixel values. Intra-prediction mode is effective in regions of video frames with similar colors that span many blocks (e.g., a blue sky).

[0014] Inter-frame prediction blocks can approximate the pixel values ​​of a source block using pixel values ​​from a previous or subsequent frame (referred to here as a reference frame). Inter-frame prediction is particularly effective for moving objects in video. When an object moves, the object in the current frame can look similar to or identical to the object in the reference frame, even if the object is in a new position. The codec can use the same pixel values ​​from the object in the reference frame to approximate the pixel values ​​of the object in the current frame. Because the codec uses the same pixel values ​​from the reference frame and does not need to generate a new set of pixel values ​​for the inter-frame prediction block, inter-frame prediction blocks often result in higher compression ratios than intra-frame prediction blocks.

[0015] Various video codec standards exist, such as H.264 (Advanced Video Coding or AVC), H.265 (High Efficiency Video Coding or HEVC), H.266 (Various Universal Video Coding or VVC), VP9, ​​and AV1 (AO Media Video 1). Each standard defines a set of identified inter-frame prediction modes and a set of identified intra-frame prediction modes. Each standard also defines a set of acceptable block sizes. For example, AV1 allows 58 intra-frame prediction modes, multiple inter-frame prediction modes, and various block sizes, including square blocks of the following sizes (in pixels): 8x8, 16x16, 32x32, 64x64, 128x128, and rectangular blocks of 64x32, 32x64, 32x16, 16x32, 16x8, and 8x16.

[0016] A standard video codec begins at the top left corner of a frame, evaluates the cost of all possible combinations of block size and prediction mode (both intra-frame and inter-frame prediction modes), selects a block based on the calculated cost, and moves on to the next block (e.g., the block to the right of the top-left block). Because intra-frame prediction mode uses pixel values ​​from neighboring blocks, each block needs to be processed sequentially to ensure that neighboring blocks have been processed and have reference pixels available for processing the current block.

[0017] Multiple processing cores (physical or virtual) can be used to accelerate block processing. Each core can process one or more blocks of a video frame in parallel with other cores that process one or more different blocks of the same frame. For example, core 1 can process a first plurality of blocks including the upper half of the frame, while core 2 can process a second plurality of blocks including the lower half of the frame. Each core obtains a prediction block within the corresponding plurality of blocks (e.g., using inter-frame prediction mode or intra-frame prediction mode) and computes a residual block representing the difference between the source pixels of the corresponding block and the pixels of the prediction block. The residual block is then processed using a suitable discrete transform (e.g., discrete Fourier transform, discrete cosine transform, etc.), and quantization is performed to eliminate the short wavelength (high frequency) ends of the transformed representation that are difficult for the human eye to discern. The remaining (quantized) coefficients of the transform are encoded in a bitstream, which is provided to the receiving device (e.g., a decoder) via a live stream or storage in memory.

[0018] Additionally, regarding encoder size, a decoding process is performed to determine the parameters of the filters to be applied to the decoded blocks in order to improve the image quality of the video frames. Specifically, an inverse residual block is obtained by inverting the discrete transform using quantized coefficients. The inverse residual block is then added to the prediction block to obtain the reconstructed block. However, the resulting reconstructed video frame (composed of reconstructed blocks) may include various artifacts from the compression process performed. For example, the reconstructed video frame may have discontinuities in intensity / color along the boundaries of the individual blocks, the quantized representation of the discrete transform may cause blurring of sharp boundaries of objects depicted in the video frame (ringing artifacts), some blocks may have acquired noise, and so on.

[0019] To reduce compression artifacts, each processing core can apply one or more filters to the reconstructed block. Some of these filters can be applied across the boundaries of different blocks (e.g., blocks reconstructed using different modes and / or reference pixels). For example, an 8×8 pixel constrained orientation enhancement filter (CDEF) or a 7×7 pixel Winer filter can be applied to a region that spans at least one block boundary, where some pixels belong to the first block and some to the second block (and possibly the third, and possibly the fourth block, if the filter is applied near the intersection of three or four block boundaries). Here, the region comprising pixels from multiple blocks is called a boundary region. Boundary regions can be of two types. The first type of boundary region comprises pixels from blocks predicted and reconstructed by a single processing core. The second type of boundary region comprises pixels from blocks predicted and reconstructed by multiple processing cores. For example, a filter can be applied to an 8×8 pixel boundary region comprised of the top 8×4 pixel portion processed by the first core and the bottom 8×4 pixel portion processed by the second core. Filtering of the second type of boundary region introduces interdependencies into the operation of different cores. More specifically, if filtering of the boundary regions is to be performed by the first core, then the first core needs access to the pixels reconstructed by the second core (and vice versa). This may require cross-coordination of processing and memory operations performed by independent cores. Furthermore, pixels in the second type of boundary region cannot be filtered until all pixels in that region have been predicted and reconstructed by the corresponding core. These and other complexities of multi-core processing result in a performance degradation of up to 20% (or more) in terms of increased encoding time.

[0020] The aspects and embodiments of the present invention address these and other technical challenges of real-time video coding in modern advanced codecs by reducing filter dependencies encountered during multi-core coding. In some embodiments, dependencies are removed by tracking boundary regions and preventing the application of filters to boundary regions that include pixels processed by different cores (e.g., prediction and reconstruction). As a result, filtering is suppressed along the boundaries of blocks and chunks assigned to different cores and enabled along the boundaries of blocks and chunks assigned to the same one or more cores.

[0021] The advantages of the disclosed embodiments over the prior art include a significant acceleration of the encoding process. Removing filter dependencies eliminates the need for processing coordinated (e.g., synchronized) by different cores, and improves live streaming compatibility by increasing the encoding bit rate at the cost of a negligible decrease in the quality of compressed video frames.

[0022] System Architecture

[0023] Figure 1AThis is a schematic block diagram of an example computing device 100 that can implement the disclosed techniques for efficient encoding operations according to at least some embodiments. The computing device 100 may include one or more devices, modules, and / or applications that interact with the video encoder 105, such as a camera 101, video editing software 102, an internet browser 103, and / or media playback software 104, etc. The video encoder 105 can perform at least some aspects of the present invention. The video encoder 105 may include a reference pixel module 106, a block prediction module 107, a filtering module 108, a cost function module 110, a compression module 111, and a memory 112. The filtering module 108 may have a boundary region tracking submodule 109 for identifying filtered regions that cross the boundaries of blocks processed by different cores.

[0024] The reference pixel module 106 may store these values ​​in memory 112 when generating the predicted block pixel values. Memory 112 may also store source pixel values ​​of video frames. The reference pixel module 106 may also retrieve reference pixel values ​​from memory 112 for approximating the block currently being processed by video encoder 105. For example, the reference pixel module 106 may retrieve a subset of calculated pixel values ​​from memory 112 for blocks above, to the left, and to the upper left of the currently processed block to calculate the cost of one or more intra-frame prediction modes. The reference pixel module 106 may also retrieve calculated pixel values ​​from memory 112 for blocks corresponding to the currently processed block from previously processed or subsequent frames to calculate the cost of one or more inter-frame prediction modes. Since inter-frame prediction modes are often used for moving objects, the reference pixel module 106 may retrieve pixel values ​​from previously processed frames (e.g., from blocks adjacent to the current block). For example, the reference pixel module 106 may retrieve calculated pixel values ​​from memory 112 for previously processed frames from blocks to the right of the currently processed block.

[0025] Block prediction module 107 may use reference pixel values ​​from reference pixel module 106 when calculating predictions for each block. Block prediction module 107 may perform intra-frame prediction and inter-frame prediction. In some embodiments, block prediction module 107 may perform sequential processing of blocks and chunks assigned to a particular processing core (e.g., in the current frame and reference frame) independently of blocks assigned to other processing cores. In some embodiments, as described in more detail below, a combination of sequential and parallel processing may be used to process chunks assigned to a particular core. For example, block prediction module 107 may perform initial (fast) sequential prediction of a chunk, subsequent (more detailed) parallel prediction of the same chunk, and final sequential reconstruction of the chunk. Multiple variations of chunk / block processing in each core may be performed. For example, during initial sequential processing (fast prediction), block prediction module 107 may calculate intra-frame predictions instead of inter-frame predictions, while during parallel processing (detailed prediction), block prediction module 107 may use both intra-frame and inter-frame predictions.

[0026] When creating an intra-prediction block, block prediction module 107 can use reference pixel values ​​from reference pixel module 106, as described above. The reference pixel values ​​can come from one or more adjacent blocks. Based on the intra-prediction mode used, the intra-prediction block can have different pixel values. For example, the intra-prediction mode can be a vertical mode, such that the predicted block has pixel values ​​obtained from its top neighboring block. Alternatively, the intra-prediction mode can be a horizontal mode, such that the predicted block has pixel values ​​obtained from its left neighboring block. As another example, the intra-prediction mode can use different orientations (e.g., at a tilt angle relative to the horizontal direction), such that the predicted block has a combination of pixel values ​​from multiple neighboring blocks.

[0027] When creating an inter-frame prediction block, the block prediction module 107 can use reference pixel values ​​from the reference pixel module 106, as described above. The reference pixel values ​​can come from the same block in a previously processed frame or from a block in a previously processed frame adjacent to the current block. In some instances, the reference pixel values ​​can come from a block that is shifted (but partially overlaps) relative to the current block. As a result, similar to intra-frame prediction blocks, inter-frame prediction blocks can have different pixel values ​​depending on the specific inter-frame prediction mode used.

[0028] Block prediction module 107 can generate multiple predictions for a given block. Each prediction can be based on a unique combination of the current block size and prediction mode. If a combination of sequential and parallel processing is being used, the number of predictions generated for each block can vary between initial sequential processing, parallel processing, and final sequential processing. For example, during initial sequential processing, each prediction can have the same block size but can have different intra-frame prediction modes, while during parallel processing, at least some or even most predictions can have different sizes and different prediction modes (intra-frame prediction or inter-frame prediction modes). Each prediction is passed to cost function module 110 to calculate the corresponding cost.

[0029] Cost function module 110 may receive one or more predictions for a block from block prediction module 107. For each prediction, the pixel values ​​of the predicted block are compared with the pixel values ​​of the corresponding source block of the original video frame being encoded. Different cost functions may be used, including sum of absolute transform differences (SATD), sum of squared differences (SSD), or any other suitable cost function. In some instances, cost function module 110 may use one component of the pixel value during initial sequential processing and multiple components during parallel processing to calculate the cost. For example, cost function module 110 may use the luma component of each pixel value to calculate the cost of intra-frame prediction during initial sequential processing. Then, during parallel processing, cost function module 110 may use both the luma and chroma components of each pixel value to calculate the cost of block prediction. In another embodiment, cost function module 110 may use the luma component of each pixel value and ignore the chroma component of each pixel value to calculate the cost of predictions for both initial sequential processing and parallel processing. In yet another embodiment, cost function module 110 may use both the luma and chroma components of each pixel value to calculate the cost of predictions for both initial sequential processing and parallel processing. In some embodiments, the cost function module 110 may, for example, use empirically selected weights to weight the errors in the representations of the luminance and chrominance components differently. Furthermore, the errors in different colors of the chrominance component may be weighted differently from each other; for example, the error in red may be weighted more than the error in blue (or vice versa).

[0030] The cost may be high if the predicted block is substantially different from the corresponding source block. The cost may be low if the predicted block is substantially similar to the corresponding source block. The cost function module 110 may select a prediction for a given block based on a comparison of the costs of each prediction in the block's predictions. In one embodiment, the cost function module 110 may select the prediction with the lowest cost. The prediction selected for a given block may include the block size and prediction mode.

[0031] After a prediction has been selected for a given block, the cost function module 110 can transmit the selection to the block prediction module 107. The block prediction module 107 can then transmit the predicted block corresponding to the selected prediction to the reference pixel module 106 for storage in the memory 112 as a reference block for future blocks and / or frames. The block prediction module 107 can also transmit the predicted block corresponding to the selected prediction to the compression module 111.

[0032] Compression module 111 can calculate a residual, which may be a residual pixel value block representing the difference between the pixel values ​​of the predicted block and the pixel values ​​of the corresponding source block. Compression module 111 can then encode the residual pixel values, prediction block size, and prediction block mode in the output bitstream according to specifications defined in a video codec standard (e.g., AV1 or any other suitable standard). In some embodiments, before encoding the values ​​in the output bitstream, compression module 111 can transform and compress the residual pixel values ​​according to specifications defined in a video codec standard. Compression performed by compression module 111 may include applying a discrete transform to the prediction block, quantizing the resulting transform, applying the discrete transform to obtain a reconstructed residual, and adding the reconstructed residual to the prediction block to obtain a reconstructed block.

[0033] The reconstructed blocks of a video frame may not yet be the blocks intended to be displayed to the viewer (on the decoder side). The reconstructed blocks may be further processed by the filtering module 108 to determine the parameters of filters to be applied to enhance the visual quality of the reconstructed blocks. The filtering module 108 may select from a plurality of filters identified by the codec standard used, and may further determine different filter parameters to enhance the visual quality of the reconstructed blocks, including removing boundaries and other artifacts generated during block prediction and reconstruction. In some embodiments, the available filters may include a deblocking filter (loop filter) that removes visible block boundaries between adjacent blocks. For example, the deblocking filter may identify values ​​of discontinuity in luminance and / or chrominance values ​​across boundaries and extend this value across multiple pixels. The filtering module 108 may determine the degree of extension as one of the parameters to be included in the bitstream and provided to the decoder on the receiving side. Additionally, the filtering module 108 may apply a constrained directional enhancement filter (CDEF) to remove ringing artifacts near the depiction of sharp edges of different objects. More specifically, CDEF can compensate for the loss of such harmonics because quantization transform irreversibly reduces or eliminates some short-wavelength harmonics (which are important for imaging sharp edges). CDEF can identify the most probable direction of an edge, for example, by identifying lines of constant intensity and lines with the maximum intensity gradient. In some embodiments, the identification of such lines can be performed by identifying and minimizing the SSD of the direction pattern for the best-matching block. After identifying the direction of the edges depicted in the block, CDEF can sharpen the depiction of the edges in the block by selecting filter intensities along and across this direction (or at a 45-degree angle to this direction). The filtering module 108 can also apply a loop recovery (LR) filter to the reconstructed block. The LR filter uses multiple available filters, such as self-guided filters, Wiener filters, or a combination of self-guided filters and Wiener filters, to reduce blur and noise. The identified filter parameters (for deblocking, CTEF, LR filters, or other suitable filters or any combination thereof) are then included in the codec output, which may be stored in memory 112 and / or streamed in real time via the Internet or any other suitable network (including LAN, WAN, PAN, public network, private network, etc.).

[0034] Figure 1B This is a schematic diagram illustrating a filtering operation 150 of a codec system for efficient encoding operations according to at least some embodiments. The filtering operation 150 can be performed by... Figure 1AThe filtering module 108 and the boundary region tracking submodule 109 are executed. The image shown is a portion of a video frame comprising multiple blocks, including blocks 1 through 4, the boundaries of which are indicated by thick solid lines. Multiple processing cores can process (predict, reconstruct, and filter) the individual blocks of the video frame. For example, as depicted, core-1 151 can process blocks-1 161 and block-2 162, while core-2 152 can process blocks-3 163 and block-4 164. Each core can also process any number of additional blocks; for simplicity, Figure 1B Not shown in the image. Any number of additional processing cores can process other chunks of the video frame. Figure 1B Not shown. The core used herein can be a physical core of a processing device (also referred to herein as a processor) or a virtual core supported by a virtualized processing environment executed on the processing device. The processing device may include a central processing unit (CPU), a graphics processing unit (GPU), a field-programmable gate array (FPGA), an application-specific integrated circuit (ASIC), or any other suitable processing unit or combination thereof. In some embodiments, the processing device may be implemented as one or more circuits. In some embodiments, each circuit may perform a function of a codec. For example, a first circuit may perform initial (e.g., sequential) block size and mode selection, a second circuit may perform subsequent parallel block size and mode selection, a third circuit may perform block reconstruction, a fourth circuit may perform discrete transformation and quantization, a fifth circuit may perform filtering, and so on. Each of the foregoing circuits may support multiple processing cores for processing blocks assigned to a corresponding core.

[0035] Each block can include any number of superblocks (coding tree blocks), such as 64×64 pixel superblocks, 128×128 pixel superblocks, etc., with their boundaries at... Figure 1B A thin solid line is used to indicate the superblock (unless the block boundary is also the block boundary). For example, block-1 161 and block-3 163 each have six superblocks, while block-2 162 and block-4 164 each have eight superblocks. A block can have any number of superblocks. Each superblock can be divided into any number of smaller blocks (coding blocks, prediction blocks) for intra-frame prediction or inter-frame prediction. The prediction blocks of block-1 161 are... Figure 1BThe diagram shows the boundaries indicated by dashed lines (unless the predicted block boundary is also the boundary of a superblock or subblock). For example, the top-left superblock of block-1 161 is not subdivided into smaller blocks and is predicted as a complete 64×64 pixel block; the top-middle superblock of block-1 161 is subdivided into sixteen 16×16 pixel blocks; the top-right superblock of block-1 161 is subdivided into two 64×32 pixel blocks; the bottom-left superblock of block-1 161 is subdivided into eight 32×16 pixel blocks; the bottom-middle superblock of block-1 161 is subdivided into four 32×32 pixel blocks; and the bottom-right superblock of block-1 161 is subdivided into four 16×64 pixel blocks. Discrete Transform ( Figure 1B (Not shown in the diagram) can be applied to the prediction block; typically, the dimension of the discrete transformation may differ from the dimension of the prediction block.

[0036] Individual filters can be applied to prediction and reconstruction (using discrete transform, quantization transform, and inverse discrete transform, as described above) blocks. The dimension of the applied filter may differ from the dimension of the prediction block and the dimension of the applied discrete transform. In some embodiments, the dimension of the filter can be set in the codec specification; for example, the dimension of the Wiener LR filter can be 7×7 pixels. In some instances, filters (e.g., CDEF and / or LR filters) can be applied to pixels entirely within a particular prediction block. Such filters can be enabled because they operate on pixels processed (prediction and reconstruction) by a single core. In some instances, filters (depicted by shaded squares) can be applied across the boundaries of prediction blocks, superblock boundaries, boundaries of blocks processed by the same core, and boundaries of blocks processed by multiple cores. For example, filter 153-1 is applied to pixels in two prediction blocks within a single superblock, filter 153-2 is applied to pixels in two superblocks, and filters 153-3 and 153-4 are applied to pixels in different blocks processed by the same core (e.g., core-1 151 and core-2 152, respectively). Although filters 153-1…153-4 are described using squares of the same size, the various filters may have different sizes. Figure 1B The annotations in the diagram schematically depict that filters 153-3 (or any other filters) may include one or more of the following: a deblocking filter 154 for smoothing block boundary artifacts, a CDEF 156 for removing ringing artifacts associated with sharp edges of objects, and / or an LR filter 158 for reducing blur and noise. Depending on the specific codec standard, some filters may not be applied; similarly, other suitable filters (not shown in the diagram) may be applied. Figure 1B (Listed in).

[0037] In conventional encoding methods, one or more of the deblocking filter 154, CDEF 156, and / or LR filter 158 can be applied to pixels processed by different cores. For example, filter 153-5 is applied to a combination of pixels in block-1 161 and block-3 163, and filter 153-6 is applied to a combination of pixels in block-2 162 and block-4 164. In some embodiments of the invention, filters that cross the boundaries of blocks processed by different cores (such as bridging filters 153-5 and 153-6, schematically depicted) can be deactivated to remove inter-core filtering dependencies. More specifically, the boundary region tracking submodule 109 can track the coordinates of pixels belonging to the region to which one of the filters will be applied, and if the corresponding region contains pixels processed by different cores, then the corresponding filter can be deactivated. Specifically, the boundary region tracking submodule 109 can know the number of processing cores used to process video frames and the distribution of various blocks / superblocks among different cores. When a filter of size (l1+1)×(l2+1) is applied to the region [x, x+l1; y, y+l2] corresponding to a rectangle having a top-left corner at pixels x, y and a bottom-right corner at pixels x+l1, y+l2, the boundary region tracking submodule 109 can determine whether any pixels in the rectangle (or any other suitable region) belong to blocks processed by different cores. If all pixels in the rectangle are processed by the same core, then the filtering module 108 can apply one or more filters in a conventional manner. If at least one pixel in the rectangle (or any other suitable region) is processed by a core different from the core that is processing other pixels, then the filtering module 108 can disable the corresponding filter. For example, in an AV1 encoder, the filtering module 108 can disable the deblocking filter by implementing the following configuration and setting the loop filter level to zero:

[0038] seg_feature_active_idx(segment, 1)=1FeatureData[segment][1]=-63

[0039] seg_feature_active_idx(segment, 2)=1FeatureData[segment][2]=-63

[0040] seg_feature_active_idx(segment, 3)=1FeatureData[segment][3]=-63

[0041] seg_feature_active_idx(segment, 4)=1FeatureData[segment][4]=-63

[0042] Similarly, filter module 108 can set parameter idx to -1 in the parameters of CDEF:

[0043] cdef_block(r, c, idx),

[0044] For a specific block that spans block boundaries and is identified by row index r and column index c, the filtering module 108 can also disable the LR filter by making the following settings.

[0045] lr_type = RESTORE_NONE.

[0046] Similar commands and operations can be used with other video encoders, but the specific syntax of the operations may differ for different encoders.

[0047] In some embodiments, such as regarding Figure 2A and Figure 2B In more detail, prior to filtering, each processing core can perform a combination of fast initial sequential predictions of blocks of video frames, followed by more detailed sequential processing of the same video frame.

[0048] Figure 2A This is a schematic diagram depicting the operation of sequential processing 200 of a codec system for efficient encoding operations according to at least some embodiments. In some embodiments, the codec system may be the video encoder 105 of FIG1. ​​Sequential processing 200 may be performed individually by each processing core and applied to a portion of a video frame assigned to the respective core. As described above, sequential processing 200 may be used to generate intermediate pixels 204 for use in subsequent parallel processing. The video frame 202 may be divided into blocks of a first size (e.g., 16x16 pixel blocks or any other suitable blocks). A first target region indicator 206 (depicted with a dashed box and a first magnified target region 208) indicates portions of the current block 210 (block N) being processed and the blocks adjacent to the current block 210. Blocks above and to the left of the current block 210 in the video frame 202 may have already been processed sequentially and may have already generated corresponding blocks of intermediate pixels, as shown by the line-shaded blocks in the video frame 202. Each block of the video frame 202 corresponds to (and approximates) a similarly positioned source pixel block in the original video frame.

[0049] During the processing of the current block 210, a first set of reference pixels 212 can be selected from the generated intermediate pixels of the blocks adjacent to the current block 210. Specifically, the first set of reference pixels 212 can be selected from the generated intermediate pixel blocks to the left, top, and upper left of the current block 210 (or from any other set of blocks, as may be specified by the codec standard). The set of reference pixels 212 can be combined with each of the multiple modes 214A-M (mode 1, mode 2, ..., mode M) to produce prediction blocks 216A-M (P1, P2, ..., PM). Each of the prediction blocks 216A-M can then be processed using a cost function 218 to determine the cost 220A-M (cost 1, cost 2, ..., cost M) of each of the prediction blocks 216A-M. The cost function 218 can compare the predicted block with the corresponding block of source pixels from video frame 202. The selected mode 222 of the current block 210 can then be determined based on the comparison of costs 220A-M. Intermediate pixel generator 224 can generate intermediate pixel block 204 for the current block 210. Intermediate pixel block 204 can be added to video frame 202 at the position of the current block 210.

[0050] Sequential processing can continue to the next block 230 (e.g., the block to the right of the current block 210), as indicated by the second target region indicator 226 and the second magnified target region 228. During the processing of the next block 230, a second set of reference pixels 232 can be selected from the generated intermediate pixels of the blocks adjacent to the next block 230. For example, a portion of the intermediate pixel block 204 created during the processing of the current block 210 can be used as part of the second set of reference pixels 232. Intermediate pixel blocks can be generated for the next block 230 in a similar manner to that described above. This process can continue until each block of the first size in the video frame 202 has a corresponding intermediate pixel block.

[0051] Figure 2BThis is a schematic diagram depicting the operation of another possible sequential processing 250 of a codec system for efficient encoding operations according to at least some embodiments. In some embodiments, the codec system may be a video encoder 105. Similar to sequential processing 200, sequential processing 250 may be executed individually by each processing core and applied to the portion of a video frame assigned to the respective core. Sequential processing 250 may be used to generate intermediate pixels for subsequent parallel processing in a manner similar to sequential processing 200. As depicted, sequential processing 250 may also utilize a reference frame 282, a reference block N 284, and an inter-frame mode 280. Video frame 202 may be divided into blocks having a first size (e.g., 16×16 pixels). A first target region indicator 206 indicates the current block 210 (block N) being processed and portions of blocks adjacent to the current block 210. Immediately adjacent to video frame 202 is a first magnified target region 208. In video frame 202, the blocks above and to the left of the current block 210 may have been processed sequentially, and the corresponding blocks for the middle pixels may have been generated, as shown by the line shadow block in video frame 202.

[0052] During the processing of the current block in sequential processing 250, a first set of reference pixels 212 can be selected from the generated intermediate pixels of the blocks adjacent to the current block 210. Specifically, the first set of reference pixels 212 can be selected from the generated intermediate pixel blocks to the left, top, and upper left of the current block 210. Additionally, reference block 284 (reference block N) can be selected from the generated intermediate pixels of reference frame 282. A third target region indicator 286 shows reference block 284 in reference frame 282. Below reference frame 282 is a third magnified target region 288, showing reference block 284. This set of reference pixels 212 can be combined with each of the multiple modes 214A-M to produce prediction blocks 216A-M. Reference block 284 can also be combined with inter-frame mode 280 to create prediction block 290. Each of prediction blocks 216A-M and prediction block 290 can then be processed using cost function 218 to determine costs 220A-M and 292 for each of prediction blocks 216A-M and prediction block 290, respectively. Cost function 218 compares the prediction block with corresponding blocks of source pixels from video frame 202. Then, based on the comparison of costs 220A-M and 292, a selected mode 294 is determined for the current block 210. Intermediate pixel generator 224 can use the selected mode 294 and reference block 284 to generate intermediate pixel blocks for the current block 210.

[0053] In another embodiment, the initial sequential processing may divide the video frame into blocks of a first size and select the corresponding source pixel blocks from the video frame as the intermediate pixels for each block.

[0054] After each block of a video frame has a corresponding intermediate pixel block (e.g., after initial sequential processing), the codec system can continue processing in parallel. The codec system can divide the video frame into blocks of a second size, a third size, and so on. Then, the codec system can process each block in parallel, similar to the initial sequential processing. In some embodiments, instead of using values ​​from neighboring blocks to compute predicted blocks in parallel processing, values ​​from neighboring blocks derived from the generated intermediate blocks are used. This eliminates the correlation between the current block and previous blocks, thereby allowing the codec system to process blocks in parallel. For example, the codec system can use a first thread to begin processing the second-size and third-size blocks at the top of the video frame, and can use a second thread to begin processing the second-size and third-size blocks simultaneously at the bottom of the video frame.

[0055] In some embodiments, one difference between the initial sequential processing and parallel processing is that, for each block in parallel processing, a prediction size and prediction mode are selected, but intermediate pixel blocks are not generated. The generation of prediction blocks using the prediction size and prediction mode selected during parallel processing is deferred until the final sequential processing.

[0056] Final sequential processing can be performed similarly to the initial sequential processing. In some embodiments, instead of segmenting the video frame into blocks of equal size, the video frame is segmented into blocks based on a prediction size selected for each block. Then, block by block, the codec system generates final prediction blocks using the selected prediction mode for the block and pixel values ​​from adjacent, previously processed final prediction blocks. The cost of each block is not calculated during final sequential processing. Once the final prediction blocks are created, residuals are calculated as described above, and the codec system creates a representation of the video frame (e.g., an image) by encoding the residuals, prediction mode, and prediction size in the output bitstream.

[0057] Figure 3This is a flowchart of an example method 300 for filtering reconstructed blocks during multi-core processing of video frames during video encoding for efficient encoding operations, according to at least some embodiments. In some embodiments, method 300 may be executed by the video encoder 105 of FIG1. ​​In some embodiments, method 300 may be executed by a dedicated codec accelerator (e.g., a coprocessor) communicating with one or more memory devices. In some embodiments, method 300 may be executed by a general-purpose processor (e.g., a central processing unit or a graphics processing unit). In some embodiments, at least some operations of method 300 may be executed by multiple (e.g., in parallel) threads, each executing one or more separate functions, routines, subroutines, or operations of the method. In some embodiments, the processing threads implementing method 300 may be synchronized (e.g., using semaphores, critical sections, and / or other thread synchronization mechanisms). Alternatively, the threads implementing method 300 may execute asynchronously relative to each other. Various operations of method 300 may be performed with... Figure 3 The order shown is different from the order in which they are executed. Some operations of method 300 can be executed concurrently with other operations. In some implementations, they are not executed. Figure 3 One or more operations are shown in the diagram.

[0058] For the sake of brevity and conciseness, the description of method 300 (and hereinafter methods 400 and 470) refers to images. It should be understood that in some embodiments, an image may be a video frame, such as a moving picture, a video advertisement, a video game, or a frame from a sequence of frames in any other video file (including synthetic video files) generated by a camera or any suitable video application. In another embodiment, an image may be a single image (including synthetic images), such as a still image, an image generated by a camera, scanner, graphics application, etc., or any other suitable image that has been compressed.

[0059] Method 300 can be executed by processing logic having multiple (physical or virtual) processing cores. At block 310, method 300 may include a first plurality of reconstructed blocks of source pixels that approximate a first portion of an image using a first processing core. In some embodiments, it can be used in... Figure 2A and Figure 2B The operation shown in the figure and the following combination Figure 4A Method 400 and Figure 4B Method 470 describes the operations used to obtain reconstructed blocks. For example, for different locations in the image, block size and prediction mode can be selected based on a comparison of computational costs, and the obtained predicted pixels can be reconstructed by computing residual blocks, performing discrete transformations, quantization transformations, etc. Similarly, at box 320, method 300 can continue to use a second processing core to obtain a second plurality of reconstructed blocks of source pixels approximating a second portion of the image. For example, referring to... Figure 1BThe reconstructed blocks of block-1 161 and block-2 162 can be obtained using the first processing core (e.g., core-1 151), while the reconstructed blocks of block-3 163 and block-4 164 can be obtained using the first processing core (e.g., core-2 152). In some embodiments, a compression algorithm based on one of the AV1, HEVC, VVC, or VP9 video coding standards is used, for example, using block size, prediction mode (inter-frame and intra-frame), discrete transform, quantization transform, etc., identified by the corresponding coding standard, to obtain the first plurality of reconstructed blocks and the second plurality of reconstructed blocks. Operations similar to those in boxes 310 and 320 can be performed for any additional cores (e.g., a third processing core, a fourth processing core, etc.) deployed together with the first and second processing cores.

[0060] At boxes 330-350, method 300 can continue to perform filtering on the obtained reconstructed blocks. More specifically, the first processing core can filter the first plurality of reconstructed blocks. The filtering may include enabling the application of one or more filters to the first plurality of regions at box 330, wherein each of the first plurality of regions includes pixels of the first plurality of reconstructed blocks but does not include pixels of a second plurality of reconstructed blocks. For example, Figure 1B The regions where filters 153-1, 153-2, and 153-3 are applied can belong to a first plurality of regions. It is possible to enable the application of one or more filters to such regions. The one or more filters may include at least one of a deblocking filter, a constrained directional enhancement filter, a Wiener filter, or a self-guided filter. At block 340, the first processing core can disable the application of one or more filters to a second plurality of regions, wherein each region in the second plurality of regions includes pixels of the first plurality of reconstructed blocks and pixels of the second plurality of reconstructed blocks. For example, in Figure 1B The regions where filters 153-5 and 153-6 are applied can belong to a second set of regions, including pixels processed by the first processing core (core-1 151) and pixels processed by the second processing core (core-2 152). The application of one or more filters to such regions can be disabled. More specifically, in some embodiments, such as those processed by... Figure 3 As depicted in the labeled section, disabling one or more filters may include: at box 342, identifying each region in the second plurality of regions that intersects with the boundary separating the first plurality of reconstructed blocks from the second plurality of reconstructed blocks. At box 344, in response to identifying the regions intersecting with the boundary, the first processing core may set one or more region-specific filter configuration parameters for each of the second plurality of regions to indicate that one or more filters will not be applied to the corresponding region. In some embodiments, the configuration parameters may have a format defined in the specification of a video encoder for compressing image data.

[0061] Operations similar to those described in combined boxes 330, 340, 342, and 344 above can be performed by other processing cores (e.g., independently and in parallel). Specifically, as depicted using box 350, method 300 may include the application of enabling one or more filters to a third plurality of regions using a second processing core, wherein each of the third plurality of regions includes pixels of a second plurality of reconstructed blocks and does not include pixels of a first plurality of reconstructed blocks. For example, Figure 1B The region where filter 153-4 is applied can belong to a third or more regions. It is possible to enable the application of one or more filters to such regions.

[0062] Figure 4A This is a flowchart of an example method 400 for parallel processing of video frames during video encoding for efficient encoding operations, according to at least some embodiments. In some embodiments, method 400 may be executed by the video encoder 105 of FIG1. ​​In some embodiments, method 400 may be executed by a dedicated codec accelerator (e.g., a coprocessor) communicating with one or more memory devices. In some embodiments, method 400 may be executed by a general-purpose processor (e.g., a central processing unit or a graphics processing unit). In some embodiments, at least some operations of method 400 may be executed by multiple (e.g., in parallel) threads, each executing one or more separate functions, routines, subroutines, or operations of the method. In some embodiments, the processing threads implementing method 400 may be synchronized (e.g., using semaphores, critical sections, and / or other thread synchronization mechanisms). Alternatively, the threads implementing method 400 may execute asynchronously relative to each other. Figure 4A Compared to the order shown, the various operations of method 400 can be executed in a different order. Some operations of method 400 can be executed concurrently with other operations. In some embodiments, no operations are executed. Figure 4A One or more operations are shown.

[0063] At box 410, the processing logic of method 400 can determine the intermediate pixel (e.g., Figure 2A (intermediate pixel 204). In one embodiment, the intermediate pixel can be a combination of the above. Figure 2A The results of the initial order of processing are described. For example, the following can be used: Figure 4B The method 470 shown in the figure determines the intermediate pixel. In another embodiment, the intermediate pixel may be a pixel of the source video frame.

[0064] At box 420, processing logic (e.g., a first processing core, a second processing core, etc.) can associate each location of the image with a block of a plurality of blocks of a first size and a block of a plurality of blocks of a second size. In some embodiments, each location can be further associated with blocks of a third size, a fourth size, etc. The first block size and the second block size (and other block sizes) can each be a block size defined by a standardized video codec such as AV1. For example, each 64x64 pixel superblock can be divided into blocks of a first size (32x32 pixels), a second size (16x8 pixels), a third size (8x8 pixels), a fourth size (32x16 pixels), etc.

[0065] At box 430, the processing logic can determine, in parallel, the first cost of the first mode and the second cost of the second mode for each block of the first size and for each block of the second size (and other sizes of blocks) using intermediate pixels. The processing logic can further determine additional costs for additional modes (e.g., the costs of third / fourth / etc. modes), for example, as... Figure 2B The diagram shows the relative modes 214A-M and costs 220A-M. Box 430 can be executed as part of parallel processing. In some instances, the first mode and the second mode can be intra-prediction mode and inter-prediction mode, respectively, and the cost of each mode can be calculated by cost function module 110. For example, the first mode can be intra-prediction mode 214A (associated with cost 220A), and the second mode can be inter-prediction mode 280 (associated with cost 292). In another instance, both the first mode and the second mode can be intra-prediction modes. In some embodiments, for each block of a given size (first / second / etc.), the corresponding cost can be determined for all or at least a subset of the modes identified by the codec standard.

[0066] In some embodiments, such as by Figure 4A As depicted in the upper annotation of the diagram, at box 432, for each of the first and second modes in box 430, the processing logic can generate a temporary block prediction based on the corresponding mode and a set of intermediate pixels. Therefore, a temporary block prediction can be generated for each block of a given size.

[0067] At box 434, for each of the first and second modes in box 430, the processing logic can calculate the cost of the corresponding mode based on a comparison of the temporary block prediction with the corresponding source pixel block of the image. More specifically, for each block of a given size, multiple temporary block predictions can be obtained, each prediction associated with a corresponding cost.

[0068] At box 440, the processing logic may select a final mode from at least a first mode and a second mode for each location of the image using a first cost and a second cost for each of a corresponding block of a first size and a corresponding block of a second size associated with the corresponding location; and select a final block size from at least the first size and the second size. For example, the processing logic may determine how to partition each 64x64 pixel superblock into smaller blocks and what mode to use for the representation of the source pixels of the corresponding block, for example, based on the cost calculated for each block / mode combination. For example, by further dividing the cost of representing the entire upper left square 32x32 pixel block using inter-frame prediction modes into two rectangular 32x16 pixel blocks, the cost of representing the entire upper left square 32x32 pixel block using inter-frame prediction modes can be compared with the cost of representing the same square 32x32 pixel block, wherein the top rectangular 32x16 pixel block is represented using a horizontal intra-frame prediction mode and the bottom rectangular 32x16 pixel block is represented using a tilt angle prediction mode.

[0069] At box 450, the processing logic can determine the final pixel based on the selected final mode and the selected final block size for each location of the image. In some embodiments, such as Figure 4A As depicted in the lower labeled section of the image, at box 452, the processing logic can generate an initial prediction block for each location in the image, based on the selected final mode, the selected final block size, and a previously determined set of final pixels associated with one or more neighboring locations. For example, the initial prediction block could be the winner block selected from the temporary block prediction based on a cost function.

[0070] At box 454, the processing logic can derive a final pixel block for each location in the image by transforming the initial predicted block using a compression algorithm. For example, compression module 111 can determine the residual block for each selected winning block, transform the residual block using one of the discrete transforms recognized by the codec standard (e.g., Discrete Fourier Transform), quantize the transformed block, and perform an inverse discrete transform to obtain the final pixel block. The processing logic can further determine the parameters of different filters that can be applied to the final pixel block to improve the visual quality of the block and remove different artifacts in the block reconstruction. For example, a deblocking filter can remove visible block boundaries between adjacent blocks, a constrained orientation enhancement filter can remove artifacts in the depiction of sharp edges, a loop recovery filter can reduce noise and improve edge quality, and so on.

[0071] At box 460, the processing logic can obtain an image representation based on the final pixels. For example, the image representation can be an encoded output bitstream of the image, which includes identifiers of selected block sizes for each part of the image, the selection mode for each block, the representation of residual blocks, and the parameters of the filters selected for different blocks, etc.

[0072] Figure 4B This is a flowchart of an example method 470 for determining intermediate pixels according to at least some embodiments in order to process video frames in parallel during video encoding for efficient encoding operations. In one embodiment, method 470 may be performed as part of block 410. Method 470 may correspond to the above-described combination. Figure 2A The initial sequential processing of the discussion can be performed by the video encoder 105 of Figure 1.

[0073] At box 412, the processing logic can segment the image into multiple blocks of a predetermined size (referred to herein as the third size). For example, the image can be segmented into blocks of 16x16 pixels, 32x32 pixels, 8x8 pixels, or any other size (including non-square block sizes, such as 16x8 pixels).

[0074] At block 414, the processing logic can sequentially process each block of the third size to determine a third cost for the first mode and a fourth cost for the second mode, wherein the determination of the third and fourth costs uses a set of intermediate pixels obtained for the previously processed blocks. In one embodiment, the first mode and the second mode can be an intra-prediction mode and an inter-prediction mode, respectively, and the cost of each mode can be calculated by the cost function module 110. In another embodiment, both the first mode and the second mode can be intra-prediction modes. At block 416, the processing logic can select an intermediate mode for each block of the third size based on a comparison of the third cost of the first mode and the fourth cost of the second mode. At block 418, the processing logic can use the selected intermediate mode to obtain a set of intermediate pixels for the corresponding block of the third size.

[0075] Figure 5 An example machine in the form of a computer system 500 is shown. The computer system 500 executes one or more sets of instructions 526, which cause the machine to perform any one or more of the methods discussed herein. The machine may operate as a server or client machine in a client-server network environment, or as a peer machine in a peer-to-peer (or distributed) network environment. The machine may be a personal computer (PC), tablet PC, set-top box (STB), personal digital assistant (PDA), mobile phone, network device, server, network router, switch, or bridge, or any machine capable of (sequentially or otherwise) executing a set of instructions specifying the actions to be taken by that machine. Further, although only a single machine is shown, the term "machine" should also be considered as any collection of machines that individually or jointly execute the set of instructions 526 to perform any one or more of the methods discussed herein.

[0076] Computer system 500 includes processor 502, main memory 504 (e.g., read-only memory (ROM), flash memory, dynamic random access memory (DRAM) (such as synchronous DRAM (SDRAM) or Rambus DRAM (RDRAM)), static memory 506 (e.g., flash memory, static random access memory (SRAM), etc.), and data storage device 516, which communicate with each other via bus 508.

[0077] Processor 502 represents one or more general-purpose processing devices, such as microprocessors, central processing units, etc. More specifically, processor 502 may be a Complex Instruction Set Computing (CISC) microprocessor, a Reduced Instruction Set Computing (RISC) microprocessor, a Very Long Instruction Word (VLIW) microprocessor, or a processor that implements other instruction sets or combinations of instruction sets. Processor 502 may also be one or more special-purpose processing devices, such as application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), digital signal processors (DSPs), network processors, etc. Processor 502 is configured to execute instructions for network nodes used to perform the operations and steps discussed herein.

[0078] Computer system 500 may also include a network interface device 522 that provides communication with other machines via a network 518 (such as a local area network (LAN), intranet, extranet, or Internet). Computer system 500 may also include a display device 510 (e.g., a liquid crystal display (LCD) or cathode ray tube (CRT)), an alphanumeric input device 512 (e.g., a keyboard), a cursor control device 514 (e.g., a mouse), and a signal generation device 520 (e.g., a speaker).

[0079] Data storage device 516 may include computer-readable storage medium 524 on which the instruction set 526 of the network node embodying any one or more methods or functions described herein is stored. During execution by computer system 500, the instruction set 526 of the network node may also reside wholly or at least partially in main memory 504 and / or processor 502, which also constitute computer-readable storage media. The instruction set 526 may further be transmitted or received via network 518 via network interface device 522.

[0080] While an example of computer-readable storage medium 524 is shown as a single medium, the term "computer-readable storage medium" can include a single medium or multiple media (e.g., a centralized or distributed database, and / or associated caches and servers) storing instruction set 526. The term "computer-readable storage medium" can include any medium capable of storing, encoding, or carrying an instruction set for execution by a machine and causing the machine to perform any one or more of the methods of this disclosure. The term "computer-readable storage medium" can include, but is not limited to, solid-state memory, optical media, and magnetic media.

[0081] Images and videos generated using one or more of the techniques disclosed herein can be displayed on a monitor or other display device. In some embodiments, the display device may be directly coupled to a system or processor that generates or renders the images or videos. In other embodiments, the display device may be indirectly coupled to a system or processor, such as via a network. Examples of such networks include the Internet, mobile telecommunications networks, Wi-Fi networks, and any other wired and / or wireless network systems. When the display device is indirectly coupled, images or videos generated by the system or processor can be streamed to the display device via the network. This streaming allows, for example, video games or other applications that render images or videos to execute on a server or in a data center, and the rendered images and videos to be transmitted and displayed on one or more user devices (such as computers, video game consoles, smartphones, other mobile devices, etc.) that are physically separate from the server or data center. Therefore, the techniques disclosed herein can be applied to enhance streamed images or videos and services that enhance streamed images and videos, such as NVIDIA GeForce Now (GFN), Google Stadia, etc.

[0082] Furthermore, images and videos generated using one or more of the techniques disclosed herein can be used to train, test, or certify deep neural networks (DNNs) for recognizing objects and environments in the real world. Such images and videos can include scenes of roads, factories, buildings, urban environments, rural environments, humans, animals, and any other physical objects or real-world environments. Such images and videos can be used to train, test, or certify DNNs used in machines or robots for manipulating, handling, or modifying physical objects in the real world. Additionally, such images and videos can be used to train, test, or certify DNNs used in autonomous vehicles to navigate and move vehicles in the real world. Furthermore, images and videos generated using one or more of the techniques disclosed herein can be used to communicate information to users of such machines, robots, and vehicles.

[0083] Other variations are also within the spirit of this disclosure. Therefore, while the disclosed technology is susceptible to various modifications and alternative constructions, certain illustrated embodiments are shown in the accompanying drawings and have been described in detail above. However, it should be understood that this disclosure is not intended to be limited to the one or more specific forms disclosed, but rather, it is intended to cover all modifications, alternative constructions, and equivalents falling within the spirit and scope of this disclosure as defined by the appended claims.

[0084] In the context of describing the disclosed embodiments (especially in the context of the following claims), the use of the terms “a” and “an”, as well as “the” and similar indicators, should be interpreted to cover both the singular and plural, unless otherwise indicated herein or obviously contradicted by the context, and is not as a definition of the terms. Unless otherwise stated, the terms “comprising,” “having,” “including,” and “containing” should be interpreted as open-ended terms (meaning “including, but not limited to”). When unmodified and referring to a physical connection, “connection” should be interpreted as partially or completely contained in, attached to, or joined together, even in the presence of intervening elements. Unless otherwise indicated herein, the enumeration of ranges of values ​​herein is intended only as a shorthand method for individually referring to each individual value falling within the range, and each individual value is incorporated into the specification as if it were individually enumerated herein. In at least one embodiment, unless otherwise indicated or contradicted by the context, the use of the terms “set” (e.g., “set of items”) or “subset” will be interpreted as a non-empty set comprising one or more members. Furthermore, unless otherwise stated or contradicted by the context, the term “subset” of the corresponding set does not necessarily mean a suitable subset of the corresponding set, but rather that the subset and the corresponding set can be equal.

[0085] Unless otherwise explicitly stated or otherwise clearly contradicted by the context, connective language (such as phrases of the form "at least one of A, B, and C" or "at least one of A, B, and C") is also understood, along with the generally used context, to represent any non-empty subset of items, terms, etc., that can be A or B or C, or A and B and C. For example, in an exemplary instance of a group with three members, the connective phrases "at least one of A, B, and C" and "at least one of A, B, and C" refer to any one of the following groups: {A}, {B}, {C}, {A, B}, {A, C}, {B, C}, {A, B, C}. Therefore, such connective language is generally not intended to imply that certain embodiments require the existence of at least one of A, at least one of B, and at least one of C, respectively. Furthermore, unless otherwise stated or contradicted by the context, the term "multiple" means multiple states (e.g., "multiple items" means multiple items). In at least one embodiment, the number of items in the multiple is at least two, but may be more when explicitly indicated or indicated by the context. Furthermore, unless otherwise stated or clearly indicated from the context, the phrase “based on” means “at least partially based on” rather than “based on alone.”

[0086] The operations of the processes described herein may be performed in any suitable order unless otherwise indicated herein or otherwise obviously contradicted by the context. In at least one embodiment, processes such as those described herein (or variations and / or combinations thereof) are executed under the control of one or more computer systems configured with executable instructions and implemented by hardware or a combination thereof as code (e.g., executable instructions, one or more computer programs, or one or more applications) that executes jointly on one or more processors. In at least one embodiment, the code is stored on a computer-readable storage medium, for example, in the form of a computer program comprising a plurality of instructions executable by one or more processors. In at least one embodiment, the computer-readable storage medium is a non-transient computer-readable storage medium that does not include transient signals (e.g., propagation of transient electrical or electromagnetic transmissions) but includes non-transient data storage circuitry (e.g., buffers, caches, and queues) within a transceiver that includes transient signals. In at least one embodiment, code (e.g., executable code or source code) is stored on a collection of one or more non-transitory computer-readable storage media having executable instructions (or other memory for storing executable instructions) stored thereon, which, when executed by one or more processors of a computer system (i.e., due to being executed), cause the computer system to perform the operations described herein. In at least one embodiment, the set of non-transitory computer-readable storage media comprises a plurality of non-transitory computer-readable storage media, and one or more of the individual non-transitory storage media among the plurality of non-transitory computer-readable storage media lack all code, while the plurality of non-transitory computer-readable storage media collectively store all code. In at least one embodiment, the executable instructions are executed such that different instructions are executed by different processors—for example, a non-transitory computer-readable storage medium stores instructions and the main central processing unit (“CPU”) executes some instructions while the graphics processing unit (“GPU”) executes other instructions. In at least one embodiment, different components of the computer system have separate processors and different processors execute different subsets of instructions.

[0087] Therefore, in at least one embodiment, the computer system is configured to implement one or more services that individually or jointly perform the operations of the processes described herein, and such a computer system is configured with applicable hardware and / or software capable of performing the operations. Furthermore, the computer system implementing at least one embodiment of this disclosure is a single device, and in another embodiment, it is a distributed computer system comprising multiple devices operating differently, such that the distributed computer system performs the operations described herein, and that a single device does not perform all the operations.

[0088] The use of any and all instances or exemplary language (e.g., “such”) provided herein is intended only to better illustrate embodiments of this disclosure and is not intended to limit the scope of this disclosure unless otherwise required. The language in the specification should not be construed as indicating that any unclaimed element is essential to the practice of this disclosure.

[0089] All references cited herein (including publications, patent applications, and patents) are incorporated herein by reference to the extent that each reference is individually and specifically indicated to be incorporated herein by reference and is listed in its full text.

[0090] The terms “coupled” and “connected” and their derivatives may be used in the specification and claims. It should be understood that these terms are not intended to be synonyms with each other. Rather, in specific examples, “connected” or “coupled” can be used to indicate that two or more elements are in direct or indirect physical or electrical contact with each other. “Coupled” can also mean that two or more elements are not in direct contact with each other, but still cooperate or interact with each other.

[0091] Unless otherwise explicitly stated, it should be understood that throughout this specification, terms such as “processing,” “calculation,” “operation,” and “determine” refer to the actions and / or processes of a computer or computing system or similar electronic computing device that manipulate and / or convert data represented as physical (e.g., electronic) quantities in the registers and / or memory of the computing system into other data similarly represented as physical quantities in the memory, registers, or other such information storage, transmission, or display devices of the computing system.

[0092] Similarly, the term "processor" can refer to any device or part of a device that processes electronic data from registers and / or memory and converts that electronic data into other electronic data that can be stored in registers and / or memory. As a non-limiting example, a "processor" can be a CPU or a GPU. A "computing platform" can include one or more processors. As used herein, a "software" process can include, for example, software and / or hardware entities that perform work over time, such as tasks, threads, and intelligent agents. Moreover, each process can refer to multiple processes for executing instructions sequentially or in parallel, continuously or intermittently. In at least one embodiment, the terms "system" and "method" are used interchangeably herein, provided that a system can embody one or more methods and a method can be considered a system.

[0093] In this document, reference may be made to obtaining, acquiring, receiving, or inputting analog or digital data into a subsystem, computer system, or computer-implemented machine. In at least one embodiment, the process of obtaining, acquiring, receiving, or inputting analog and digital data can be accomplished in various ways, such as by receiving data as parameters of a function call or a call to an application programming interface. In at least one embodiment, the process of obtaining, acquiring, receiving, or inputting analog or digital data can be accomplished by transmitting data via a serial or parallel interface. In at least one embodiment, the process of obtaining, acquiring, receiving, or inputting analog or digital data can be accomplished by transmitting data from a providing entity to an acquiring entity via a computer network. In at least one embodiment, reference may also be made to providing, outputting, transmitting, sending, or presenting analog or digital data. In various examples, the process of providing, outputting, transmitting, sending, or presenting analog or digital data can be implemented by transmitting data as input or output parameters of a function call, an application programming interface, or an inter-process communication mechanism.

[0094] While the description herein illustrates exemplary embodiments of the described technologies, other architectures may be used to implement the described functionality and are intended to fall within the scope of this disclosure. Furthermore, although a specific distribution of responsibilities may be defined above for descriptive purposes, different functions and responsibilities may be distributed and divided in different ways depending on the environment.

[0095] Furthermore, although the subject matter has been described using language specific to structural features and / or methodological actions, it is to be understood that the subject matter claimed in the appended claims is not necessarily limited to the specific features or actions described. Rather, specific features and actions are disclosed as exemplary forms for implementing the claims.

Claims

1. A method for compressing image data, the method comprising: Using a first processing core, a first plurality of reconstructed blocks of source pixels approximating a first portion of the image are obtained, wherein obtaining the first plurality of reconstructed blocks includes: The source pixels of the first portion of the image are processed sequentially to determine intermediate pixels for the first portion of the image, wherein the intermediate pixels are associated with an initial pattern and an initial block size; Associate multiple modes and multiple block sizes with pixels; The intermediate pixels are used to determine the cost for each of the plurality of patterns and each of the plurality of block sizes, wherein the cost for at least a subset of the plurality of patterns and the plurality of block sizes is determined in parallel; Based on the determined cost, a final mode is selected from the plurality of modes for the pixel, and a final block size is selected from the plurality of block sizes; and The pixel is obtained using the selected final mode and final block size; The second processing core is used to obtain a second plurality of reconstructed blocks of source pixels that approximate a second portion of the image; and The first processing core is used to filter the first plurality of reconstruction blocks, the filtering including: An application that enables one or more filters to a first plurality of regions, each of which includes pixels of the first plurality of reconstructed blocks and excludes pixels of the second plurality of reconstructed blocks; and Disabling the application of one or more filters to a second plurality of regions, each of which includes pixels of the first plurality of reconstruction blocks and pixels of the second plurality of reconstruction blocks.

2. The method according to claim 1, wherein, The one or more filters include at least one of a deblocking filter, a constrained directional enhancement filter, a Wiener filter, or a self-guided filter.

3. The method according to claim 1, wherein, The first plurality of reconstructed blocks and the second plurality of reconstructed blocks are obtained using a compression algorithm based on one of the AV1, HEVC, VVC or VP9 video coding standards.

4. The method according to claim 1, further comprising: The second processing core is used to enable the application of the one or more filters to a third plurality of regions, each of which includes pixels of the second plurality of reconstruction blocks but does not include pixels of the first plurality of reconstruction blocks.

5. The method according to claim 1, wherein, The application of disabling one or more filters to the second plurality of regions includes: Identify each of the second plurality of regions that intersects with the boundary separating the first plurality of reconstructed blocks and the second plurality of reconstructed blocks; and One or more region-specific filter configuration parameters are set for each of the second plurality of regions to indicate that the one or more filters are not applied to the corresponding regions of the second plurality of regions, wherein the one or more region-specific filter configuration parameters have a format defined in the specification of the video encoder for compressing the data of the image.

6. The method according to claim 1, wherein, Obtaining the first plurality of reconstructed blocks includes: The first processing core is used to associate the pixel with a first block size and a second block size among the plurality of block sizes; Using the first processing core, for each of the first block size and the second block size, the intermediate pixels are used in parallel to determine a first cost for the first mode and a second cost for the second mode among the plurality of modes; Using the first processing core, for the pixel, using the first cost and the second cost, i) select a final mode from at least the first mode or the second mode and ii) select a final block size from at least the first block size and the second block size; and The first processing core is used to determine the pixel based on the selected final mode and the selected final block size.

7. The method according to claim 6, wherein, Determining the intermediate pixel includes: The first processing core is used to divide the first portion of the image into multiple blocks of the third size; Each block of the third block size is processed sequentially to determine a third cost of the first mode and a fourth cost of the second mode, wherein the third cost and the fourth cost are determined using a set of intermediate pixels obtained for the previously processed blocks; Based on a comparison of the third cost of the first mode and the fourth cost of the second mode, an intermediate mode is selected for each block of the third block size; and Use the selected intermediate mode to obtain a set of intermediate pixels for the corresponding block of the third block size.

8. A system for compressing image data, comprising: A memory device used to store the source pixels of an image; as well as One or more circuits communicatively coupled to the memory device, said one or more circuits being configured to: A first plurality of reconstructed blocks of the source pixels approximating a first portion of the image are obtained using a first processing core, wherein, in order to obtain the first plurality of reconstructed blocks, the one or more circuits are configured to: The source pixels of the first portion of the image are processed sequentially to determine intermediate pixels for the image, wherein the intermediate pixels are associated with an initial pattern and an initial block size; Associate multiple modes and multiple block sizes with pixels; The intermediate pixels are used to determine the cost for each of the plurality of patterns and each of the plurality of block sizes, wherein the cost for at least a subset of the plurality of patterns and the plurality of block sizes is determined in parallel; Based on the determined cost, a final mode is selected from the plurality of modes and a final block size is selected from the plurality of block sizes for the pixel; as well as The pixel is obtained using the selected final mode and final block size; The second processing core is used to obtain a second plurality of reconstructed blocks of the source pixels that approximate a second portion of the image; as well as The first processing core is used to filter the first plurality of reconstruction blocks, wherein, in order to filter the first plurality of reconstruction blocks, the one or more circuits are configured to: Enabling one or more filters in an application to a first plurality of regions, each of the first plurality of regions including pixels of the first plurality of reconstruction blocks and excluding pixels of the second plurality of reconstruction blocks; as well as Disabling the application of one or more filters to a second plurality of regions, each of the second plurality of regions including pixels of the first plurality of reconstruction blocks and pixels of the second plurality of reconstruction blocks.

9. The system according to claim 8, wherein, The one or more filters include at least one of a deblocking filter, a constrained directional enhancement filter, a Wiener filter, or a self-guided filter.

10. The system according to claim 8, wherein, The first plurality of reconstructed blocks and the second plurality of reconstructed blocks are obtained using a compression algorithm based on one of the AV1, HEVC, VVC or VP9 video coding standards.

11. The system according to claim 8, wherein, The one or more circuits are further configured to: The second processing core is used to enable the application of the one or more filters to a third plurality of regions, each of which includes pixels of the second plurality of reconstruction blocks but does not include pixels of the first plurality of reconstruction blocks.

12. The system according to claim 8, wherein, To disable the application of the one or more filters to the second plurality of regions, the one or more circuits are further configured to: Identify each of the second plurality of regions that intersects with the boundary that separates the first plurality of reconstruction blocks and the second plurality of reconstruction blocks; as well as One or more region-specific filter configuration parameters are set for each of the second plurality of regions to indicate that the one or more filters are not applied to the corresponding regions of the second plurality of regions, wherein the one or more region-specific filter configuration parameters have a format defined in the specification of a video encoder for compressing the image.

13. The system according to claim 8, wherein, In order to obtain the first plurality of reconfiguration blocks, the one or more circuits are further configured to: The first processing core is used to associate the pixel with a first block size and a second block size among a plurality of block sizes; Using the first processing core, for each of the first block size and the second block size, the intermediate pixels are used in parallel to determine a first cost for the first mode and a second cost for the second mode among the plurality of modes; Using the first processing core, using the first cost and the second cost, i) select a final mode from at least the first mode or the second mode and ii) select a final block size from at least the first block size or the second block size; as well as Using the first processing core, the pixel is determined based on the selected final mode and the selected final block size.

14. The system according to claim 13, wherein, To determine the intermediate pixel, the one or more circuits are configured to: The first processing core is used to divide the first portion of the image into multiple blocks of the third size; Each block of the third block size is processed sequentially to determine a third cost of the first mode and a fourth cost of the second mode, wherein the third cost and the fourth cost are determined using a set of intermediate pixels obtained for the previously processed blocks; Based on the comparison between the third cost of the first mode and the fourth cost of the second mode, an intermediate mode is selected for each block of the third block size; as well as Use the selected intermediate mode to obtain a set of intermediate pixels for the corresponding block of the third block size.

15. A non-transitory computer-readable medium comprising instructions that, when executed by a processor, cause the processor to perform the following operations: Using the first processing core of the processor, a first plurality of reconstructed blocks of source pixels of a first portion of an approximate image are obtained, wherein obtaining the first plurality of reconstructed blocks includes: The source pixels of the first portion of the image are processed sequentially to determine intermediate pixels for the image, wherein the intermediate pixels are associated with an initial pattern and an initial block size; Associate multiple modes and multiple block sizes with pixels; The intermediate pixels are used to determine the cost for each of the plurality of patterns and each of the plurality of block sizes, wherein the cost for at least a subset of the plurality of patterns and the plurality of block sizes is determined in parallel; Based on the determined cost, a final mode is selected from the plurality of modes for the pixel, and a final block size is selected from the plurality of block sizes; and The pixel is obtained using the selected final mode and final block size; The second processing core of the processor is used to obtain a second plurality of reconstructed blocks of source pixels that approximate a second portion of the image; and The first processing core is used to filter the first plurality of reconstruction blocks, the filtering including: An application that enables one or more filters to a first plurality of regions, each of which includes pixels of the first plurality of reconstructed blocks and excludes pixels of the second plurality of reconstructed blocks; and Disabling the application of one or more filters to a second plurality of regions, each of which includes pixels of the first plurality of reconstruction blocks and pixels of the second plurality of reconstruction blocks.

16. The non-transitory computer-readable medium according to claim 15, wherein, The one or more filters include at least one of a deblocking filter, a constrained directional enhancement filter, a Wiener filter, or a self-guided filter.

17. The non-transitory computer-readable medium according to claim 15, wherein, The first plurality of reconstructed blocks and the second plurality of reconstructed blocks are obtained using a compression algorithm based on one of the AV1, HEVC, VVC or VP9 video coding standards.

18. The non-transitory computer-readable medium according to claim 15, wherein, The instructions further cause the processor to perform the following operations: The second processing core is used to enable the application of the one or more filters to a third plurality of regions, each of which includes pixels of the second plurality of reconstruction blocks but does not include pixels of the first plurality of reconstruction blocks.

19. The non-transitory computer-readable medium according to claim 15, wherein, The application of disabling one or more filters to the second plurality of regions includes: Identify each of the second plurality of regions that intersects with the boundary separating the first plurality of reconstructed blocks and the second plurality of reconstructed blocks; and One or more region-specific filter configuration parameters are set for each of the second plurality of regions to indicate that the one or more filters are not applied to the corresponding regions of the second plurality of regions, wherein the configuration parameters have a format defined in the specification of a video encoder for compressing the image.

20. The non-transitory computer-readable medium according to claim 15, wherein, Obtaining the first plurality of reconstructed blocks includes: The first processing core is used to associate the pixel with a first block size and a second block size among a plurality of block sizes; Using the first processing core, for each of the first block size and the second block size, the intermediate pixels are used in parallel to determine a first cost for the first mode and a second cost for the second mode among the plurality of modes; Using the first processing core, using the first cost and the second cost, i) select a final mode from at least the first mode or the second mode and ii) select a final block size from at least the first block size or the second block size; The first processing core is used to determine the pixel based on the selected final mode and the selected final block size.

Images