Method and apparatus for encoding and decoding pictures - Patents.com
Adaptive scan order schemes for video compression systems optimize entropy encoding and decoding by considering block shape, prediction tools, and transform types, enhancing compression efficiency and reducing coefficient interlacing.
Patent Information
- Application Number
- JP2021532334
- Authority / Receiving Office
- JP · JP
- Patent Type
- Patents
- Current Assignee / Owner
- Priority Date
- 2018-12-05
- Filing Date
- 2019-12-02
- Publication Date
- 2024-11-06
- Estimated Expiration
- 2039-12-02
AI Technical Summary
Existing video compression systems lack an adaptive scan order for quantized transform coefficients, leading to inefficient entropy encoding and decoding, particularly for blocks with varying shapes and prediction tools, which affects compression efficiency.
Adaptive scan order schemes are introduced that depend on block shape, prediction tools, and transform types, using methods like exhaustive rate-distortion search to select the optimal scan pattern for entropy encoding and decoding.
Improves compression efficiency by aligning the scan order with block characteristics, reducing interlacing of low and high-frequency coefficients, and enhancing entropy coding performance.
Smart Images

Figure 0007578596000012 
Figure 0007578596000013 
Figure 0007578596000014
Abstract
Description
[Technical field]
[0001] At least one of the present embodiments relates generally to methods and apparatus for encoding and decoding pictures, and more specifically to methods and apparatus for encoding and decoding pictures in which quantized transform coefficients are scanned according to a scan pattern for entropy coding and decoding. [Background technology]
[0002] To achieve high compression efficiency, image and video coding schemes typically use prediction and transformation to exploit spatial and temporal redundancy in video content. In general, intra- or inter-prediction is used to exploit intra- or inter-frame correlation, and the difference between the original and predicted image blocks, often denoted as prediction error, prediction residual, or predictor, is transformed, quantized, and entropy coded. During encoding, the original image blocks are typically partitioned / divided into sub-blocks, possibly by using quadtree partitioning. To reconstruct the video, the compressed data is decoded by an inverse process corresponding to prediction, transformation, quantization, and entropy coding. Summary of the Invention
[0003] According to a general aspect of at least one embodiment, a decoding method is presented that includes: entropy decoding transform coefficients of a block scanned according to a scan pattern, where the scan pattern is determined according to a shape of the block.
[0004] According to a general aspect of at least one embodiment, a decoding method is presented that includes: entropy decoding transform coefficients of a block scanned according to a scan pattern, where, when the block is a multi-hypothesis intra-inter predicted block, vertical scanning is applied if the intra prediction mode is horizontal, and horizontal scanning is applied if the intra prediction mode is vertical.
[0005] According to a general aspect of at least one embodiment, a decoding method is presented that includes: entropy decoding transform coefficients of a block scanned according to a scan pattern, where if the block is a triangular predicted block, the coefficients are scanned according to a scan pattern selected by an exhaustive rate-distortion search among a set of patterns, and the selected scan pattern is coded.
[0006] According to a general aspect of at least one embodiment, a decoding method is presented that includes: entropy decoding transform coefficients of a block scanned according to a scan pattern, where the scan pattern is determined depending on a type of inverse transform used to inverse transform said coefficients, the type of transform being defined by its low-frequency basis functions being constant, increasing or decreasing.
[0007] According to a general aspect of at least one embodiment, a method of encoding is presented that includes: - obtaining a transformation coefficient for the block; - scanning said coefficients for entropy coding according to a scan pattern; Here, the scan pattern is determined according to the shape of the block.
[0008] According to a general aspect of at least one embodiment, a method of encoding is presented that includes: - obtaining a transformation coefficient for the block; - scanning the coefficients according to a scan pattern for entropy coding; Here, when the block is a multi-hypothesis intra / inter predicted block, if the intra prediction mode is horizontal, vertical scanning is applied, and if the intra prediction mode is vertical, horizontal scanning is applied.
[0009] According to a general aspect of at least one embodiment, a method of encoding is presented that includes: - obtaining a transformation coefficient for the block; - scanning the coefficients according to a scan pattern for entropy coding; Here, if the block is a triangular predicted block, the coefficients are scanned according to a scan pattern selected by an exhaustive rate-distortion search among a set of patterns, and the selected scan pattern is coded.
[0010] According to a general aspect of at least one embodiment, a method of encoding is presented that includes: - transforming the residuals of the blocks using at least one transform to obtain transform coefficients; - scanning the coefficients according to a scan pattern for entropy coding; Here, the scan pattern is determined according to the type of transformation, which is defined by its low-frequency basis functions being constant, increasing or decreasing.
[0011] According to a general aspect of at least one embodiment, a decoding device is presented that includes one or more processors configured to implement: The transform coefficients of a block scanned according to a scan pattern are entropy decoded, where the scan pattern is determined according to a shape of the block.
[0012] According to a general aspect of at least one embodiment, a decoding device is presented that includes one or more processors configured to implement: Entropy decode transform coefficients of a block scanned according to a scan pattern, where, when the block is a multi-hypothesis intra / inter predicted block, vertical scanning is applied if the intra prediction mode is horizontal, and horizontal scanning is applied if the intra prediction mode is vertical.
[0013] According to a general aspect of at least one embodiment, a decoding device is presented that includes one or more processors configured to implement: The transform coefficients of a block scanned according to a scan pattern are entropy decoded, and if the block is a triangular predicted block, the coefficients are scanned according to a scan pattern selected by an exhaustive rate-distortion search among a set of patterns, and the selected scan pattern is coded.
[0014] According to a general aspect of at least one embodiment, a decoding device is presented that includes one or more processors configured to implement: Entropy decode transform coefficients of a block scanned according to a scan pattern, where the scan pattern is determined according to a type of inverse transform used to inverse transform said coefficients, the type of transform being defined by its low-frequency basis functions being constant, increasing or decreasing.
[0015] According to a general aspect of at least one embodiment, an encoding device is presented that includes one or more processors configured to implement: - obtaining a transformation coefficient for the block; - scanning said coefficients for entropy coding according to a scan pattern; Here, the scan pattern is determined according to the shape of the block.
[0016] According to a general aspect of at least one embodiment, an encoding device is presented that includes one or more processors configured to implement: - obtaining a transformation coefficient for the block; - scanning the coefficients according to a scan pattern for entropy coding; Here, when the block is a multi-hypothesis intra / inter predicted block, if the intra prediction mode is horizontal, vertical scanning is applied, and if the intra prediction mode is vertical, horizontal scanning is applied.
[0017] According to a general aspect of at least one embodiment, an encoding device is presented that includes one or more processors configured to implement: - obtaining a transformation coefficient for the block; - scanning the coefficients according to a scan pattern for entropy coding; If the block is a triangular predicted block, the coefficients are scanned according to a scan pattern selected by an exhaustive rate-distortion search among a set of patterns, and the selected scan pattern is coded.
[0018] According to a general aspect of at least one embodiment, an encoding device is presented that includes one or more processors configured to implement: - transforming the residuals of the blocks using at least one transform to obtain transform coefficients; - scanning the coefficients according to a scan pattern for entropy coding; Here, the scan pattern is determined according to the type of transformation, which is defined by its low-frequency basis functions being constant, increasing or decreasing.
[0019] According to another general aspect of at least one embodiment, a bitstream is formatted to include a signal generated according to the encoding method described above.
[0020] One or more of the present embodiments also provide a computer-readable storage medium having stored thereon instructions for encoding or decoding video data according to at least a portion of any of the above methods. One or more embodiments also provide a computer-readable storage medium having stored thereon a bitstream generated according to the above encoding methods. One or more embodiments also provide methods and apparatus for transmitting or receiving a bitstream generated according to the above encoding methods. One or more embodiments also provide a computer program product including instructions for performing at least a portion of any of the above methods. [Brief description of the drawings]
[0021] [Figure 1] A diagram showing several scanning patterns for 8x8 blocks used in HEVC. [Diagram 2] FIG. 2 illustrates a decision table that associates scan patterns with the size of each block. [Diagram 3] FIG. 2 illustrates a decision table that associates scan patterns with the size of each block. [Figure 4] 4 is a flowchart of a decoding method according to one embodiment. [Diagram 5] 4 is a flowchart of an encoding method according to one embodiment. [Figure 6A] 1 illustrates a decision table that associates scan patterns with the size of each block, according to one embodiment. [Figure 6B] 1 illustrates a decision table that associates scan patterns with the size of each block, according to one embodiment. [Figure 7] FIG. 1 illustrates conventional zigzag scanning in a 4x4 block. [Figure 8] 1A-1C are diagrams illustrating various scanning patterns of a rectangular block according to one embodiment. [Figure 9] FIG. 13 is a diagram showing a multi-hypothesis prediction block for a combination of inter and intra modes. [Figure 10] FIG. 13 illustrates various weightings of multi-hypothesis intra-inter predicted blocks for different intra prediction directions. [Figure 11] FIG. 2 illustrates blocks divided by a diagonal split line and the weights associated with the diagonal edges. [Figure 12] FIG. 2 illustrates a block diagram of a video encoder according to one embodiment. [Figure 13] FIG. 2 illustrates a block diagram of a video decoder according to one embodiment. [Figure 14]FIG. 1 illustrates a block diagram of an example of a system in which various aspects and embodiments may be implemented. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS
[0022] In HEVC coding, a picture is divided into square CTUs of configurable size, usually 64x64. In other video coding standards, the size is 128x128 or 256x256. A CTU is the root of a quadtree partition into four square coding units (CUs) of the same size, i.e., half the parent block size in width and height. A quadtree is a tree in which a parent node can be divided into four child nodes, and each node may be a parent node of another node that is divided into four child nodes. In HEVC, a coding block (CB) forms the root of a quadtree in which a coding block (CB) is divided into one or more predictive blocks (PBs) and transform blocks (TBs). Corresponding to coding blocks, predictive blocks, and transform blocks, a coding unit (CU) contains a set of predictive units (PUs) and tree-structured transform units (TUs), where a PU contains prediction information for all color components, and a TU contains the residual coding syntax structure for each color component. The sizes of the CB, PB, and TB of the luma component are applied to the corresponding CU, PU, and TU.
[0023] In modern coding systems, a CTU is the root of a coding tree that is divided into coding units (CUs). A coding tree is a tree in which a parent node (usually corresponding to a block) can be divided into child nodes (e.g., 2, 3 or 4 child nodes), and each node can be a parent node of another node that is divided into child nodes. In addition to the quadtree partitioning mode, new partitioning modes are also defined that increase the total number of possible partitioning modes: binary tree symmetric partitioning mode, binary tree asymmetric partitioning mode and ternary tree partitioning mode. A coding tree has a unique root node, such as a CTU. The leaves of the coding tree are the end nodes of the tree. Each node of the coding tree represents a block that can be further divided into smaller blocks, also called subblocks. Once the partitioning of the CTU into CUs is determined, the CUs corresponding to the leaves of the coding tree are encoded. The partitioning of the CTU into CUs, and the coding parameters used to code each CU (corresponding to the leaves of the coding tree), can be determined at the encoder side via a rate-distortion optimization procedure.
[0024] In this application, the term "block" or "picture block" may be used to refer to any one of the CTU, CU, PU, TU, CB, PB, and TB. Additionally, the term "block" or "picture block" may be used to refer to macroblocks, partitions, and sub-blocks specified in H.264 / AVC or other video coding standards, and more generally to an array of samples of any size.
[0025] In this application, the terms "reconstructed" and "decoded" can be used interchangeably, the terms "pixel" and "sample" can be used interchangeably, and the terms "image", "picture" and "frame" can be used interchangeably. Typically, but not necessarily, the term "reconstructed" is used on the encoder side and the term "decoded" is used on the decoder side.
[0026] In this application, the terms "scanning order" and "scan pattern" may be used interchangeably.
[0027] In HEVC, a picture is divided into so-called coding tree units (CTUs), each of which is represented by a coding unit (CU) in the compressed domain. Each CU is given some intra or inter prediction parameters. To do so, it is spatially divided into one or more prediction units (PUs), each of which is assigned some prediction information (e.g. intra prediction mode, reference index, motion vectors). The intra or inter coding mode is assigned at the CU level.
[0028] After partitioning, intra- or inter-prediction is used to exploit intra- or inter-frame correlation, and the difference between the original block and the predicted block, often denoted as the prediction error or prediction residual, is transformed, quantized, and entropy coded. To reconstruct the video, the compressed data is decoded by an inverse process corresponding to the entropy coding, quantization, transformation, and prediction.
[0029] In HEVC, the quantized transform coefficients of a coding block are coded using non-overlapping coefficient groups (CGs). A CG contains the coefficients of a 4x4 block of the coding block. As an example, the CGs contained in an 8x8 block are shown in Figure 1. The CGs in the coding block, and the 16 transform coefficients in the CG, are scanned and coded according to a scan pattern selected from three scan orders: diagonal, horizontal, and vertical. Between blocks, the diagonal scan on the left side of Figure 1 is always used. Figure 2 shows the type of scan pattern used for each size of intra block. In this figure, the first vertical line indicates the block height, the first horizontal line indicates the block width, D indicates that a diagonal scan pattern is used, and M indicates that a mode-dependent coefficient scan (MDCS) is used. In HEVC, this mode is only used for 4x4 and 8x8 intra blocks. Therefore, for 4x4 and 8x8 intra blocks, the scan order depends on the intra prediction mode active for that block. MDCS is not currently implemented in VTM-3, the software model for VVC (Versatile Video Coding). However, the standards body proposal document JVET-L0379 proposes to use MDCS in VVC and extend it to several rectangular block sizes introduced in VTM-3 by the quadtree (QT) / binary tree (BT) / ternary tree (TT) partitioning. QT is the partitioning of a block into four identical sub-blocks. BT is the partitioning of a block into two identical rectangles by either a vertical or horizontal line. TT is the partitioning of a block into three identical rectangles by either two vertical or two horizontal lines.
[0030] When using MDCS for HEVC 4x4 or 8x8 blocks, the scan pattern for intra blocks is determined by the intra prediction mode using a fixed look-up table (LUT). The LUT for HEVC luma blocks is shown in Table 1: vertical modes 6-14 use horizontal scan, horizontal modes 22-30 use vertical scan, and other modes use diagonal scan.
[0031] [Table 1]
[0032] In JVET-L0379, the solution in Figure 2 is modified. A different decision table is used to switch between MDCS and diagonal scan. Figure 3 shows this different decision table. This table is based on the ratio of the block height to width. If the ratio of the block's larger dimension to its smaller dimension is 4 or more, MDCS is applied. MDCS also applies to blocks of size 4x4, 4x8, 8x4 and 8x8. Note that the maximum size of a transform block is 64x64 for luma and 32x32 for chroma. One drawback of this solution is that the scan order of coefficients depends only on the intra prediction mode of intra blocks. Furthermore, there is no adaptive scan order for inter blocks, i.e., inter blocks always use a diagonal scan pattern. The entropy coding / decoding of transform coefficients consists of up to five scan paths, each of which covers the coding of syntax elements from the following list: sig_flag or important-coeff-flag: coefficient importance indicating whether the value of the coefficient is equal to zero; gt1_flag or coeff-abs-level-greater1-flag: indicates whether the absolute value of the coefficient level is greater than 1; gt2_flag or coeff-abs-level-greater2-flag: indicates whether the absolute value of the coefficient level is greater than 2; par_flag: indicates the parity of the coefficients if Dependent Quantization is used; coeff-abs-level-remaining: remaining value of coefficient level absolute value (if the coefficient's value is greater than the value coded in the previous pass); coeff-sign-flag: The sign of the significant coefficient (0: positive, 1: negative).
[0033] Each scan path follows a scan pattern. In each scan path, syntax is coded only if necessary as determined by the previous scan path. For example, if a coefficient's sig_flag is zero, the remaining scan paths are not needed to code the absolute value of that coefficient. The same is true for sign coding. The coeff-sign-flag is sent only for non-zero coefficients.
[0034] All scan paths are performed for a given CG until all quantized coefficients in that CG are reconstructed before processing the next CG. The scan starts from the last significant coefficient of the coding block, sweeps from high frequency coefficients to low frequency coefficients, and processes all coefficients up to the DC coefficient.
[0035] The quantized transform coefficients are entropy coded to output a bitstream. As a non-limiting example, context-based adaptive binary arithmetic coding (CABAC) can be used to code the syntax elements into the bitstream. For coding with CABAC, non-binary syntax element values are mapped to binary sequences called bin strings. For a bin, a context model is selected. The context model stores the probability that each bin is a "1" or a "0", and the context model can be adaptive or static. The static model triggers the coding engine with equal probability for bins "0" and "1". In an adaptive coding engine, the context model is updated based on the actual coding value of the bin. The operating modes corresponding to the adaptive and static models are called normal and bypass modes, respectively.
[0036] At least one embodiment aims to improve compression efficiency compared to existing video compression systems.
[0037] Therefore, at least one embodiment proposes to efficiently adapt the scanning order of quantized transform coefficients in a manner that provides good compression efficiency (rate-distortion performance) with minimal increase in the complexity of the coding design.
[0038] Therefore, at least one embodiment proposes several schemes for adapting the scan order of the quantized transform coefficients to one or more of the block shape, prediction tool, and transform type.
[0039] In the following, for simplicity, the term "transform coefficients" is used instead of "quantized transform coefficients".
[0040] The energy of the residual information is highly compressed in the upper left corner of the coding block after transformation, which indicates that the DC coefficients and low-frequency transform coefficients usually have more information and higher absolute value compared to the transform coefficients located in the high-frequency domain.
[0041] A scan pattern that follows the descending order of the likelihood that the coefficients are non-zero provides more accurate CABAC probability modeling. Diagonal scanning arranges the 2D transform coefficients in a 1D array in descending order according to their frequency. However, this single scan pattern does not adapt to the coefficient values, and can be improved by employing an adaptive scan order.
[0042] 4 shows a flowchart of an encoding method according to one general embodiment. In S100, a scan pattern is determined according to the shape of a block to be decoded. In S110, the transform coefficients of the block scanned according to the determined scan pattern are entropy decoded.
[0043] 5 shows a flowchart of a decoding method according to one general embodiment. In S200, a scan pattern is determined according to the shape of the block to be coded. In S210, the transform coefficients of the block scanned according to the determined scan pattern are entropy coded.
[0044] The first embodiment proposes a general adaptive scan order scheme that is directly related to the block shape.
[0045] Intra prediction modes are correlated with the shape of the block. Horizontal blocks are more likely to select horizontal intra prediction modes. The frequency domain is also correlated with the block shape. In fact, if the block is rectangular, there will be many zero-valued coefficients in the high frequency region along the long dimension compared to the short dimension. Therefore, using diagonal scans can result in too many interlaced low and high frequency coefficients. Take a vertical block for example, along the vertical direction, which is the longer dimension, more coefficients tend to be zero in the high frequency region. By using horizontally oriented coefficient scans instead of diagonal scans, the non-zero coefficients of the 2D matrix can be further compressed towards the beginning of the 1D vector. This improves the efficiency of entropy coding.
[0046] According to the basic idea of the first embodiment, the selection of the scanning order depends on the block shape, which can also be called "block-shape dependent coefficients scanning". For square blocks, diagonal scanning or MDCS can be applied. For rectangular blocks, an adaptive scanning order is used. The shape of a rectangular block is defined by its height and width values and its orientation. Thus, a block of size 8x16 (vertical block) and a block of size 16x8 (horizontal block) have different shapes.
[0047] In the first variant, vertical scanning is used in horizontal blocks and horizontal scanning is used in vertical blocks. Otherwise, only diagonal scanning or MDCS is applied. The second variant is shown in Figure 6A.
[0048] In a second variant, if the ratio of the dimensions of the larger block to the smaller block is equal to or greater than one value T, then vertical scanning is used for the horizontal blocks and horizontal scanning is used for the vertical blocks. Otherwise, only diagonal scanning or MDCS is applied. This second variant is shown in FIG. 6B for the case where the value T is equal to 4.
[0049] In a third variant, for rectangular blocks, the best scan order is selected by exhaustive search among a set of scan patterns, e.g., conventional diagonal, horizontal, and vertical scan patterns. The exhaustive search can also be performed among another set of scan patterns and is not limited to the above list. The optimal scan order for rectangular blocks is selected based on rate-distortion (RD) estimation and is signaled to the decoder by encoding an additional syntax element tu_coeffscan_type. Thus, the scan pattern offering the best rate-distortion compromise is selected among the set of scan patterns. Otherwise, only diagonal scanning or MDCS is applied.
[0050] In a fourth variant, if the ratio of the dimensions of the larger blocks to the dimensions of the smaller blocks is equal to or greater than one value T, the best scan order is selected by exhaustive search among a set of scan patterns, e.g., the conventional diagonal, horizontal, and vertical scan patterns. The exhaustive search can also be performed among a different set of scan patterns and is not limited to the above list. The optimal scan order of the blocks is selected based on the RD estimate and is signaled to the decoder by encoding an additional syntax element tu_coeffscan_type, as shown in the syntax table. Otherwise, only diagonal scanning or MDCS is applied.
[0051] Additional syntax elements are coded / decoded as shown for the third and fourth variants. Details are shown in the syntax table below of how to signal the optimal scan order type selected by exhaustive search among diagonal, horizontal, and vertical scan patterns based on RD estimation. Specific signaling is shown in italics.
[0052] [Table 2]
[0053] [Table 3-1]
[0054] [Table 3-2]
[0055] [Table 3-3]
[0056] [Table 3-4]
[0057] tu_coeffscan_type specifies a coefficient scan type selected from, for example, diagonal, horizontal and vertical scans of a transform block. The value of tu_coeffscan_type can be 0, 1 or 2, representing diagonal (0), horizontal (1) and vertical (2), respectively. In a variant, tu_coeffscan_type specifies a coefficient scan type / pattern selected from another set of scan patterns, which may include scan patterns different from the diagonal, horizontal and vertical scan patterns.
[0058] ScanOrder specifies the scan order array that produces the positions of the coefficients in the scan order. For each scan type, the ScanOrder array is initialized differently and is obtained by calling the function getScanOrder().
[0059] In another variant, an additional adaptive scan order is defined that differs from the conventional diagonal, horizontal, and vertical scan orders. For this purpose, a specific vertical and horizontal scan order is designed for a rectangular block. The design of the adaptive scan order is based on a zigzag scan order, as shown in Figure 7. The specific vertical and horizontal scan order is designed such that more coefficients along the longest dimension of the scan order are compressed together than along the shortest dimension. According to this variant, the scan step size along the longest dimension, Step L is the scan step size along the shortest dimension. S (Usually, the default is Step S = 1). The scan step size is the distance interval from the first coefficient to the last coefficient along one dimension before scanning coefficients along another dimension, as shown in Figure 8. This multiple is the maximum length of the longest dimension of the coding block, Size, as defined below. L and the shortest dimension Size S is related to the ratio of
[0060]
number
[0061] Some examples of the proposed adaptive zigzag scan orders for rectangular blocks of size 8x4, 4x8 and 16x4 are shown in FIG.
[0062] As mentioned above, an intra block can select a scan pattern based on the intra prediction mode of this block if MDCS is activated. One limitation of MDCS or current algorithms is that no adaptive scan order scheme is defined for inter blocks. Diagonal scan patterns are the only scan patterns used for inter blocks. The second embodiment proposes a general adaptive scan order scheme for inter blocks related to the prediction tool used.
[0063] The new video codec VTM-3 implements an increased number of inter-coding tools, e.g., multi-hypothesis prediction (MH), to improve coding efficiency. The general concept of MH is to combine the inter prediction performed in merge mode with intra prediction mode or another inter prediction mode (e.g., unipredictive AMVP, skip and merge). In merge mode, a list of merge candidates (reference index, motion value) is created and a merge index is signaled that identifies one candidate. Then, the motion information of the block in merge mode is derived from the candidate identified by the merge index and used for motion compensated prediction. In inter-intra MH, the final prediction is a weighted average of the predictions generated by the merge indexed prediction and the intra prediction mode, with different weights applied depending on the combination, as shown in Figure 9. The current block is divided into four equal area regions. The weights gradually decrease as the regions get further from the reference intra sample. Each set of weights, denoted as (w_intrai, w_interi), is applied to the corresponding region. where i goes from 1 to 4, (w_intra1, w_inter1) = (6, 2), (w_intra2, w_inter2) = (5, 3), (w_intra3, w_inter3) = (3, 5), and (w_intra4, w_inter4) = (2, 6). An example of intra-vertical / horizontal prediction and DC / intra-plane prediction used in multiple hypotheses is shown in Figure 10.
[0064] According to a second embodiment, the adaptive scan order is extended to inter blocks depending on the prediction tool (e.g., whether to use a specific inter prediction tool such as MH, OBMC, triangular prediction, etc.). For multi-hypothesis prediction blocks, an adaptive scan order is used.
[0065] In the first variant, for multi-hypothesis inter-intra predicted blocks, the MDCS is reused regardless of the block size. If the intra prediction mode is horizontal, vertical scanning is applied. Conversely, for vertical intra prediction modes, horizontal scanning is used.
[0066] In a second variant, for multi-hypothesis prediction blocks (i.e., inter-intra or inter-inter), the best scan order is selected by exhaustive search among a set of scan patterns, e.g., conventional diagonal, horizontal, and vertical scan patterns. The exhaustive search can also be performed among a different set of scan patterns and is not limited to the above list. The optimal scan order for the block is selected based on the RD estimation and is signaled to the decoder by encoding an additional syntax element, tu_coeffscan_type, as shown in the syntax table.
[0067] Geometric (e.g. triangular) modes are defined that give flexibility in dividing the picture into blocks before coding. Various triangular divisions are shown in the top part of Figure 11. The blocks are divided using a diagonal division line. Additional coding performance can be obtained by performing weighting of the two predictors P1 and P2 at the frontier (gray area at the bottom of Figure 11). Triangular prediction unit modes are only applied to CUs with block size 8x8 or larger, whose coding prediction modes are either skip or merge modes. Triangular prediction modes have many advantages in scenarios where objects have different movements within the same block or where a diagonal line divides a block into two partitions. Therefore, triangular prediction modes are often applied to non-flat blocks. After transformation and quantization, low-frequency and high-frequency transform coefficients may still be too interlaced if only diagonal scanning is used.
[0068] In a third variant, for triangular prediction blocks, the best scan order is selected by exhaustive search among a set of scan patterns, e.g., conventional diagonal, horizontal, and vertical scan patterns. The exhaustive search can also be performed among a different set of scan patterns and is not limited to the above list. The optimal scan order for a block is selected based on the RD estimate and is signaled to the decoder by encoding an additional syntax element, tu_coeffscan_type, as shown in the syntax table.
[0069] The third embodiment proposes a general adaptive scan order scheme that takes into account some relationships that exist between the activation of the adaptive scan order and the type of transformation applied to a block.
[0070] Multiple transform selection (MTS) is currently implemented in VTM-3. According to MTS, DST-7 and DCT-8 are used in addition to DCT-2. There is a separate MTS to enable intra and inter flags at the SPS level. When MTS is enabled, a single flag is encoded / decoded to signal that DCT-2 is applied in both directions. When MTS is enabled and DCT-2 is not used in both directions, a 2-bit index (MTS_Idx) is coded / decoded to signal 4 pairs of horizontal and vertical transforms. Thus, DCT8 and DST7 are the two possible transforms for horizontal and vertical directions.
[0071] Some of the DCT / DST transform families can be classified as increasing, decreasing, and constant linear basis functions, as shown in Table 2. As explained in Table 2, DCT-2 has constant linear basis functions, DST-7 has increasing linear basis functions, and DCT-8 has decreasing linear basis functions. The linear basis functions represent the basis functions of the transform considered at the lowest frequencies. As an example, the DCT2 default function T i (j) (i,j=0,1,…,N-1) is defined as follows:
[0072]
number
[0073] The first-order basis function is constant for DCT2 since it is for i=0.
[0074] As another example, the DCT8 basis function T i (j) (i,j=0,1,…,N-1) is defined as follows:
[0075]
number
[0076] The first order basis function is for i=0, so it is reduced in DCT8.
[0077] As another example, the DST7 basis function T i (j) (i,j=0,1,…,N-1) is defined as follows:
[0078]
number
[0079] Since the first-order basis function is for i=0, the increase is seen in the case of DST7.
[0080] By implementing the 2-D DCT-2 separately, by performing the 1-D DCT-2 twice, horizontally and vertically, the signal correlation along either the horizontal or vertical direction can be captured by this separable 2D transform. However, this 2-D DCT-2 may not be optimal in the presence of directional edges. Therefore, various combinations of DST-7 and DCT-8 applied separately to the horizontal and vertical directions are complementary for these cases.
[0081] After an adaptive transform such as MTS, the distribution statistics of the transform coefficients become less mode-dependent, but there is still room in the adaptive scan order to avoid too many interlaced low-frequency and high-frequency transform coefficients.
[0082] [Table 4]
[0083] In a third embodiment, the selection of the scan order depends on the transform type, which may also be referred to as "transform-type dependent coefficients scanning."
[0084] In a first variant, if a separable transformation scheme is used, the following applies: - if a transformation with constant linear basis functions is applied to this block in both the horizontal and vertical directions, a diagonal scan is applied; - Otherwise, if neither the horizontal nor the vertical direction uses a transform with constant linear basis functions, an adaptive scan order is active for this block. For this purpose, the optimal scan order is selected by an exhaustive search among a set of scan patterns, e.g., the conventional diagonal, horizontal, and vertical scan patterns. The exhaustive search can also be performed among a different set of scan patterns and is not limited to the above list. The optimal scan order for a block is selected based on the RD estimation and is signaled to the decoder by encoding the additional syntax element tu_coeffscan_type, as shown in the syntax table.
[0085] In a second variant, if MTS is used, the following applies: - if a separable 2-D DCT-2 transform is applied to this block, a diagonal scan is applied; - Otherwise, if DST-7 and DCT-8 are applied separately for this block in the horizontal and vertical directions, an adaptive scan order is active. For this purpose, the optimal scan order is selected by an exhaustive search among a set of scan patterns, e.g., the conventional diagonal, horizontal, and vertical scan patterns. The exhaustive search can also be performed among a different set of scan patterns and is not limited to the above list. The optimal scan order for a block is selected based on the RD estimation and is signaled to the decoder by encoding the additional syntax element tu_coeffscan_type, as shown in the syntax table.
[0086] The first, second and third embodiments can be used completely independently or in combination. In one example, the second and third embodiments can use the new scan order defined in the first embodiment, for example the one shown in FIG. 8. In order to select the new scan order to use for a block, the shape of the block is required. Finally, for rectangular blocks, the final scan pattern can be selected from D, H, V and additional new scan patterns based on the prediction tool or transformation type, as in the second and third embodiments.
[0087] If the additional syntax tu_coeffscan_type is applied, indicating a scan order selected from a set of scan patterns, such as diagonal, horizontal, vertical scan patterns, the context used for entropy encoding / decoding can be selected based on the block shape, prediction tool and transform type.
[0088] This application describes various aspects, including tools, features, embodiments, models, approaches, and the like. Many of these aspects are described specifically and often in a manner that may sound restrictive, at least to illustrate their individual characteristics. However, this is for clarity of description and does not limit the application or scope of these aspects. Indeed, all of the various aspects can be combined and interchanged to provide further aspects. Additionally, these aspects can also be combined and interchanged with aspects described in previous applications.
[0089] The aspects described and discussed in this application can be implemented in a variety of forms. Figures 12, 13, and 14 below provide some embodiments, but other embodiments are contemplated, and the descriptions of Figures 12, 13, and 14 are not intended to limit the breadth of implementations. At least one aspect generally relates to encoding and decoding video, and at least one other aspect generally relates to transmitting a generated or encoded bitstream. These and other aspects can be implemented as a method, an apparatus, a computer-readable storage medium having stored thereon instructions for encoding or decoding video data according to any of the described methods, and / or a computer-readable storage medium having stored thereon a bitstream generated according to any of the described methods.
[0090] Various methods are described herein, each method including one or more steps or actions for achieving the described method. Unless a specific order of steps or actions is required for proper operation of the method, the order and / or use of specific steps and / or actions may be varied or combined.
[0091] As shown in Figures 12 and 13, various methods and other aspects described herein can be used to modify modules of video encoder 100 and decoder 200, such as entropy encoding and / or decoding modules (145, 230). Additionally, the present aspects are not limited to VVC or HEVC, and can be applied to other standards and recommendations, whether existing or developed in the future, and extensions of such standards and recommendations (including VVC and HEVC). Unless otherwise specified or technically precluded, aspects described in this application can be used individually or in combination.
[0092] In this application, various numerical values are used, such as the value T. The specific values are for illustrative purposes, and the described aspects are not limited to these specific values.
[0093] 12 illustrates an encoder 100. Although variations of this encoder 100 are contemplated, the encoder 100 is described below for clarity without describing all anticipated variations.
[0094] Before being encoded, a video sequence may undergo pre-encoding processing (101), for example to apply a color transformation to the input color picture (e.g., from RGB 4:4:4 to YCbCr 4:2:0) or to perform a remapping of the input picture components to make the signal distribution more resilient to compression (e.g., using histogram equalization of one of the color components). Metadata associated with the pre-processing may be attached to the bitstream.
[0095] In the encoder 100, a picture is coded by the encoder elements as described below. The picture to be coded is divided (102) and processed, for example, in units of CUs. Each unit is coded, for example, using either intra mode or inter mode. If the unit is coded in intra mode, intra prediction is performed (160). In inter mode, motion estimation (175) and motion compensation (170) are performed. The encoder decides (105) whether to use intra mode or inter mode to code the unit and indicates the intra mode / inter mode decision, for example, by a prediction mode flag. A prediction residual is calculated, for example, by subtracting (110) the prediction block from the original image block.
[0096] The prediction residual is transformed (125) and quantized (130). The quantized transform coefficients, as well as motion vectors and other syntax elements, are entropy coded (145) to output a bitstream. The encoder can skip the transform and apply quantization directly to the untransformed residual signal. The encoder can bypass both the transform and quantization, i.e., the residual is coded directly without applying a transform or quantization process.
[0097] The encoder decodes the coded block to provide a reference for further prediction. The quantized transform coefficients are inverse quantized (140) and inverse transformed (150) to decode the prediction residual. The decoded prediction residual is combined with the prediction block (155) to reconstruct an image block. An in-loop filter (165) is applied to the reconstructed image to perform, for example, deblocking / sample adaptive offset (SAO) filtering to reduce coding artifacts. The filtered image is stored in a reference picture buffer (180).
[0098] Figure 13 shows a block diagram of a video decoder 200. In the decoder 200, the bitstream is decoded by a decoder element as described below. The video decoder 200 generally performs a decoding pass that is the inverse of the encoding pass, as described in Figure 12. The encoder 100 also generally performs video decoding as part of the encoding of the video data.
[0099] In particular, the decoder's input includes a video bitstream, which may be generated by the video encoder 100. The bitstream is first entropy decoded (230) to obtain transform coefficients, motion vectors, and other coded information. Picture partition information indicates how the picture is partitioned. Thus, the decoder can partition (235) the picture according to the decoded picture partition information. The transform coefficients are inverse quantized (240) and inverse transformed (250) to decode the prediction residual. The decoded prediction residual is combined with a predictive block (255) to reconstruct an image block. The predictive block may result from intra prediction (260) or motion compensated prediction (i.e., inter prediction) (275) (270). An in-loop filter (265) is applied to the reconstructed image. The filtered image is stored in a reference picture buffer (280).
[0100] The decoded image may further undergo post-decoding processing (285), such as an inverse color conversion (e.g., converting from YCbCr 4:2:0 to RGB 4:4:4), or an inverse remapping that performs the inverse of the remapping process performed in the pre-encoding process (101). The post-decoding process may use metadata derived in the pre-encoding process and signaled in the bitstream.
[0101] FIG. 14 illustrates a block diagram of an example of a system in which various aspects and embodiments are implemented. System 1000 may be embodied as a device including various components described below and configured to perform one or more aspects described in this document. Examples of such devices include various electronic devices, such as, but not limited to, personal computers, laptop computers, smartphones, tablet computers, digital multimedia set-top boxes, digital television receivers, personal video recording systems, connected home appliances, and servers. The elements of system 1000, alone or in combination, may be embodied in a single integrated circuit (IC), multiple ICs, and / or separate components. For example, in at least one embodiment, the processing elements and encoder / decoder elements of system 1000 are distributed across multiple ICs and / or separate components. In various embodiments, system 1000 is communicatively coupled to one or more other systems or other electronic devices, for example, via a communication bus or via dedicated input and / or output ports. In various embodiments, system 1000 is configured to implement one or more of the aspects described herein.
[0102] The system 1000 includes at least one processor 1010 configured to execute instructions loaded therein, for example to implement various aspects described herein. The processor 1010 may include embedded memory, input / output interfaces, and various other circuits known in the art. The system 1000 includes at least one memory 1020 (e.g., a volatile memory device and / or a non-volatile memory device). The system 1000 includes a storage device 1040 that may include non-volatile memory and / or volatile memory, including, but not limited to, Electrically Erasable Programmable Read-Only Memory (EEPROM), Read-Only Memory (ROM), Programmable Read-Only Memory (PROM), Random Access Memory (RAM), Dynamic Random Access Memory (DRAM), Static Random Access Memory (SRAM), Flash, magnetic disk drives, and / or optical disk drives. Storage 1040 may include, by way of non-limiting example, internal storage, attached storage (including removable and non-removable storage), and / or network-accessible storage.
[0103] The system 1000 includes, for example, an encoder / decoder module 1030 configured to process data to provide encoded or decoded video, which may include its own processor and memory. The encoder / decoder module 1030 represents a module that may be included in a device to perform encoding and / or decoding functions. As is known, a device may include one or both of an encoding module and a decoding module. Furthermore, the encoder / decoder module 1030 may be implemented as a separate element of the system 1000 or may be incorporated within the processor 1010 as a combination of hardware and software, as is known to those skilled in the art.
[0104] Program code to be loaded into the processor 1010 or the encoder / decoder 1030 to perform various aspects described herein may be stored in the storage device 1040 and then loaded into the memory 1020 for execution by the processor 1010. According to various embodiments, one or more of the processor 1010, the memory 1020, the storage device 1040, and the encoder / decoder module 1030 may store one or more of various items during execution of the processes described herein. Such stored items may include, but are not limited to, input video, decoded video or portions of decoded video, bitstreams, matrices, variables, and intermediate or final results from processing of equations, expressions, operations, and logical operations.
[0105] In some embodiments, memory internal to the processor 1010 and / or the encoder / decoder module 1030 is used to store instructions and provide working memory for processing required during encoding or decoding. However, in other embodiments, memory external to the processing device (e.g., the processing device can be either the processor 1010 or the encoder / decoder module 1030) is used for one or more of these functions. The external memory can be the memory 1020 and / or the storage device 1040, e.g., dynamic volatile memory and / or non-volatile flash memory. In some embodiments, the external non-volatile flash memory is used to store, for example, the television's operating system. In at least one embodiment, a high-speed external dynamic volatile memory, such as a RAM, is used as working memory for video encoding and decoding operations. Video encoding and decoding operations include MPEG-2 (MPEG stands for Moving Picture Experts Group, MPEG-2 is also known as ISO / IEC 13818, 13818-1 is also known as H.222, and 13818-2 is also known as H.262), HEVC (HEVC stands for High Efficiency Video Coding, also known as H.265 and MPEG-H), or VVC (Versatile Video Coding, a new standard being developed by JVET (Joint Video Experts Team)), etc.
[0106] Inputs to the elements of system 1000 may be provided via various input devices, as shown in block 1130. Such input devices include, but are not limited to, (i) a radio frequency (RF) section that receives RF signals transmitted over the air, for example by a broadcast station, (ii) a component (COMP) input terminal (or set of COMP input terminals), (iii) a universal serial bus (USB) input terminal, and / or (iv) a High Definition Multimedia Interface (HDMI) input terminal. Other examples not shown in FIG. 10 include composite video.
[0107] In various embodiments, the input devices of block 1130 have associated respective input processing elements as known in the art. For example, the RF section may be associated with elements suitable to (i) select a desired frequency (also referred to as selecting a signal or band-limiting a signal to a band of frequencies), (ii) downconvert the selected signal, (iii) band-limit again to a narrower band of frequencies (for example) to select a signal frequency band, which may be referred to as a channel in a particular embodiment, (iv) demodulate the downconverted band-limited signal, (v) perform error correction, and (vi) demultiplex to select a desired stream of data packets. The RF section of various embodiments includes one or more elements for performing these functions, e.g., a frequency selector, a signal selector, a band limiter, a channel selector, a filter, a downconverter, a demodulator, an error corrector, and a demultiplexer. The RF section may include, for example, a tuner to perform these various functions, including downconverting a received signal to a lower frequency (e.g., an intermediate frequency or near baseband frequency) or to baseband. In one set-top box embodiment, the RF section and its associated input processing elements receive RF signals transmitted over a wired (e.g., cable) medium and perform frequency selection by filtering, down-converting, and re-filtering to a desired frequency band. Various embodiments rearrange the order of the above (and other) elements, remove some of these elements, and / or add other elements that perform similar or different functions. Adding elements includes inserting elements between existing elements, such as, for example, inserting an amplifier or an analog-to-digital converter. In various embodiments, the RF section includes an antenna.
[0108] Additionally, the USB and / or HDMI terminals may include respective interface processors for connecting the system 1000 to other electronic devices via USB and / or HDMI connections. It should be understood that various aspects of the input processing, e.g., Reed-Solomon error correction, may be implemented, for example, in a separate input processing IC or within the processor 1010, as desired. Similarly, aspects of the USB or HDMI interface processing may be implemented, for example, in a separate interface IC or within the processor 1010, as desired. The demodulated, error corrected, and demultiplexed stream is provided to various processing elements, including, for example, the processor 1010 and an encoder / decoder 1030 operating in combination with memory and storage elements to process the data stream as required for presentation on an output device.
[0109] The various elements of system 1000 can be provided within an integrated housing in which the various elements can be interconnected and data transmitted therebetween using suitable connection configurations, e.g., internal buses known in the art, including Inter-IC (I2C), wiring, and printed circuit boards.
[0110] The system 1000 includes a communication interface 1050 that enables communication with other devices over a communication channel 1060. The communication interface 1050 may include, but is not limited to, a transceiver configured to transmit and receive data over the communication channel 1060. The communication interface 1050 may include, but is not limited to, a modem or a network card, and the communication channel 1060 may be implemented in a wired and / or wireless medium, for example.
[0111] Data is streamed or otherwise provided to the system 1000 in various embodiments using a wireless network such as a Wi-Fi network, e.g., IEEE 802.11 (IEEE refers to the Institute of Electrical and Electronics Engineers). The Wi-Fi signal in these embodiments is received via a communication channel 1060 and a communication interface 1050 adapted for Wi-Fi communication. The communication channel 1060 in these embodiments is typically connected to an access point or router that provides access to external networks, including the Internet, to enable streaming applications and other over-the-top communications. Other embodiments provide streamed data to the system 1000 using a set-top box that delivers data via an HDMI connection in the input block 1130. Still other embodiments provide streamed data to the system 1000 using an RF connection in the input block 1130. As discussed above, various embodiments provide data in a non-streaming manner. Additionally, various embodiments use wireless networks other than Wi-Fi, e.g., a cellular network or a Bluetooth network.
[0112] The system 1000 can provide output signals to various output devices including a display 1100, speakers 1110, and other peripherals 1120. The display 1100 of various embodiments includes, for example, one or more of a touch screen display, an organic light emitting diode (OLED) display, a curved display, and / or a foldable display. The display 1100 can be for a television, a tablet, a laptop, a mobile phone, or other device. The display 1100 can also be integrated with other components (e.g., as in a smartphone) or separately (e.g., an external monitor for a laptop). The other peripherals 1120 include, in various example embodiments, one or more standalone digital video discs (or digital versatile discs) (DVRs in both terms), disc players, stereo systems, and / or lighting devices. Various embodiments use one or more peripherals 1120 that provide functionality based on the output of the system 1000. For example, a disc player performs the function of playing the output of the system 1000.
[0113] In various embodiments, control signals are communicated between the system 1000 and the display 1100, speaker 1110, or other peripherals 1120 using signaling such as AV-Link, Consumer Electronics Control (CEC), or other communication protocols that allow inter-device control with or without user intervention. The output devices may be communicatively coupled to the system 1000 via dedicated connections via respective interfaces 1070, 1080, and 1090. Alternatively, the output devices may be connected to the system 1000 using a communication channel 1060 via the communication interface 1050. The display 1100 and speaker 1110 may be integrated into a single unit with other components of the system 1000 in an electronic device such as, for example, a television. In various embodiments, the display interface 1070 includes a display driver, such as, for example, a timing controller (T Con) chip.
[0114] Alternatively, the display 1100 and speakers 1110 may be separate from one or more of the other components, for example, if the RF portion of the input 1130 is part of a separate set-top box. In various embodiments in which the display 1100 and speakers 1110 are external components, the output signal may be provided via a dedicated output connection including, for example, an HDMI port, a USB port, or a COMP output.
[0115] The embodiments may be executed by the processor 1010 or by computer software implemented by hardware or by a combination of hardware and software. As a non-limiting example, the embodiments may be implemented by one or more integrated circuits. The memory 1020 may be of any type suitable for the technology environment and may be implemented using any suitable data storage technology, such as optical memory devices, magnetic memory devices, semiconductor-based memory devices, fixed memories, and removable memories, as non-limiting examples. The processor 1010 may be of any type suitable for the technology environment and may include one or more of a microprocessor, a general-purpose computer, a special-purpose computer, and a processor based on a multi-core architecture, as non-limiting examples.
[0116] Various implementations include decoding. As used in this application, "decoding" can include all or part of the processes performed, for example, on a received encoded sequence to generate a final output suitable for display. In various embodiments, such processes include one or more processes typically performed by a decoder, such as entropy decoding, inverse quantization, inverse transform, and differential decoding. In various embodiments, such processes also or instead include processes performed by the decoder of the various implementations described in this application, such as determining a scan pattern for a block based on one or more of the block shape, prediction tool, and transform type.
[0117] As a further example, in one embodiment, "decoding" refers to only entropy decoding, in another embodiment, "decoding" refers to only differential decoding, and in another embodiment, "decoding" refers to a combination of entropy and differential decoding. Whether the term "decoding process" is intended to refer specifically to a subset of operations or to the broader decoding process generally will be clear based on the context of a particular description and will be well understood by one of ordinary skill in the art.
[0118] Various implementations include encoding. Similar to the above description of "decoding," "encoding" as used in this application can include, for example, all or part of the processes performed on an input video sequence to generate an encoded bitstream. In various embodiments, such processes include one or more processes typically performed by an encoder, such as partitioning, differential encoding, transforming, quantizing, and entropy encoding. In various embodiments, such processes also or instead include processes performed by an encoder in various implementations described in this application, such as determining a scan pattern for a block based on one or more of a block shape, a predictor, and a transform type.
[0119] As a further example, in one embodiment, "encoding" refers to only entropy encoding, in another embodiment, "encoding" refers to only differential encoding, and in another embodiment, "encoding" refers to a combination of differential and entropy encoding. Whether the term "encoding process" is intended to refer specifically to a subset of operations or to the broader encoding process in general will be clear based on the context of a particular description and will be well understood by one of ordinary skill in the art.
[0120] Please note that the syntax elements used here, e.g., tu_coeffscan_type, are descriptive terms, so they do not preclude the use of other syntax element names.
[0121] It should be understood that where a diagram is presented as a flow diagram, it also provides a block diagram of the corresponding apparatus. Similarly, where a diagram is presented as a block diagram, it also provides a flow diagram of the corresponding method / process.
[0122] Various embodiments refer to rate-distortion (RD) optimization. In particular, the encoding process usually considers a balance or trade-off between rate and distortion, taking into account computational complexity constraints. Rate-distortion optimization is usually formulated as minimizing a rate-distortion function, which is a weighted sum of rate and distortion. There are various approaches to solving the rate-distortion optimization problem. For example, approaches are based on extensive testing of all encoding options, including all modes or coding parameter values considered, and a full evaluation of the coding cost and associated distortion of the reconstructed signal after coding and decoding. Faster approaches can also be used to save on encoding complexity, especially with the calculation of approximate distortion based on a predicted or predicted residual signal rather than on the reconstructed signal. A combination of these two approaches can also be used, such as using approximate distortion for only some of the possible encoding options and full distortion for others. Other approaches evaluate only a subset of the possible encoding options. More generally, many approaches employ any of a variety of techniques to perform the optimization, but the optimization is not necessarily a full evaluation of both the coding cost and the associated distortion.
[0123] The implementations and aspects described herein may be implemented, for example, in a method or process, an apparatus, a software program, a data stream, or a signal. Even if described only in the context of a single form of implementation (e.g., described only as a method), the implementation of the described functionality may also be implemented in other forms (e.g., an apparatus or a program). An apparatus may be implemented, for example, in appropriate hardware, software, and firmware. The methods may be implemented in a processor, which generally refers to a processing device, including, for example, a computer, a microprocessor, an integrated circuit, or a programmable logic device. Processors also include, for example, computers, communication devices such as mobile phones, personal digital assistants (PDAs), and other devices that facilitate communication of information between end users.
[0124] References to "one embodiment" or "embodiment" or "one implementation" or "implementation," as well as other variations thereof, mean that a particular feature, structure, characteristic, etc. described in connection with an embodiment is included in at least one embodiment. Thus, the appearances of "in one embodiment" or "in an embodiment" or "in one implementation" or "in an implementation" appearing in various places in this application, as well as any other variations thereof, are not necessarily all referring to the same embodiment.
[0125] Additionally, this application may refer to "determining" various information. Determining the information may include, for example, one or more of estimating the information, calculating the information, predicting the information, or retrieving the information from a memory.
[0126] Additionally, this application may refer to "accessing" various information. Accessing information may include, for example, one or more of receiving information, retrieving information (e.g., from a memory), storing information, moving information, copying information, calculating information, determining information, predicting information, or inferring information.
[0127] Additionally, this application may refer to "receiving" various information. Receiving, like "accessing," is intended as a broad term. Receiving information may include, for example, one or more of accessing information or retrieving information (e.g., from a memory). Additionally, "receiving" is typically involved in some way during an operation such as, for example, storing information, processing information, transmitting information, moving information, copying information, erasing information, calculating information, determining information, predicting information, or estimating information.
[0128] For example, in the cases of "A / B," "A and / or B," and "at least one of A and B," the use of any of " / ," "and / or," and "at least one of" should be understood to be intended to include the selection of only the first listed option (A), or the selection of only the second listed option (B), or the selection of both options (A and B). As a further example, in the cases of "A, B, and / or C" and "at least one of A, B, and C," such language is intended to include the selection of only the first listed option (A), or the selection of only the second listed option (B), or the selection of only the third listed option (C), or the selection of only the first and second listed options (A and B), or the selection of only the first and third listed options (A and C), or the selection of only the second and third listed options (B and C), or the selection of all three options (A, B, and C). This may be expanded for the number of items listed, as would be apparent to one of ordinary skill in the art and related fields.
[0129] Also, as used herein, the term "signal" refers to, among other things, indicating something to a corresponding decoder. For example, in a particular embodiment, the encoder signals a particular one of the scan patterns. Thus, in one embodiment, the same parameters are used on both the encoder and decoder sides. Thus, for example, the encoder may send a particular parameter to the decoder (explicit signaling) so that the decoder can use the same particular parameter. Conversely, if the decoder already has the particular parameter as well as other parameters, it may use signaling (implicit signaling) without sending the particular parameter so that the decoder can know and select the particular parameter. By avoiding the transmission of the actual function, bit savings are realized in various embodiments. It should be understood that signaling can be accomplished in various ways. For example, one or more syntax elements, flags, etc. are used in various embodiments to signal information to a corresponding decoder. Although the above is about the verb form of the word "signal", the word "signal" can also be used as a noun in this specification.
[0130] As will be apparent to one of ordinary skill in the art, implementations can generate a variety of signals formatted to carry information that can be, for example, stored or transmitted. The information can include, for example, instructions for performing a method or data generated by one of the described implementations. For example, a signal can be formatted to carry a bit stream of the described embodiments. Such a signal can be formatted, for example, as an electromagnetic wave (e.g., using a radio frequency portion of the spectrum) or as a baseband signal. Formatting can include, for example, encoding a data stream and modulating a carrier with the encoded data stream. The information the signal carries can be, for example, analog information or digital information. The signal can be transmitted over a variety of different wired or wireless links, as is known. The signal can be stored on a processor-readable medium.
[0131] Several embodiments are described. The features of these embodiments may be provided alone or in any combination. Furthermore, the embodiments may include one or more of the following features, devices, or aspects, alone or in any combination, across various claim categories and types: - Determining a scan pattern for the block based on one or more of the block shape, the prediction tool, and the transformation type. - A bitstream or signal containing one or more of the described syntax elements or their variations. - Inserting signaling syntax elements that allow the decoder to adapt the scanning order in a way that corresponds to the ordering used by the encoder. - generating, and / or transmitting, and / or receiving, and / or decoding a bitstream or signal that includes one or more of the described syntax elements or variations thereof; A television, set-top box, mobile phone, tablet, or other electronic device that performs scan order adaptation according to any of the described embodiments. a television, set-top box, mobile phone, tablet, or other electronic device that performs the scan order adaptation in accordance with any of the described embodiments and displays the resulting images (e.g., using a monitor, screen, or other type of display). - A television, set-top box, mobile phone, tablet, or other electronic device that tunes a channel (e.g., using a tuner) to receive a signal including the encoded image and performs the scanning order adaptation according to any of the described embodiments. - a television, set-top box, mobile phone, tablet, or other electronic device that receives a signal containing the encoded image wirelessly (e.g., using an antenna) and performs scan order adaptation according to any of the described embodiments.
[0132] According to a general aspect of at least one embodiment, a decoding method is presented that includes: The transform coefficients of the scanned block are entropy decoded according to a scan pattern, where the scan pattern is determined according to the shape of the block.
[0133] According to a general aspect of at least one embodiment, a method of encoding is presented that includes: - obtaining a transformation coefficient of the block; - scanning said coefficients for entropy coding according to a scan pattern, where the scan pattern is determined depending on the shape of the block.
[0134] According to a general aspect of at least one embodiment, a decoding device is presented that includes one or more processors configured to implement: The transform coefficients of the scanned block are entropy decoded according to a scan pattern, where the scan pattern is determined according to the shape of the block.
[0135] According to a general aspect of at least one embodiment, an encoding device is presented that includes one or more processors configured to implement: - obtaining a transformation coefficient of the block; - scanning the coefficients for entropy coding according to a scan pattern, where the scan pattern is determined depending on the shape of the block.
[0136] According to another general aspect of at least one embodiment, a bitstream is formatted to include a signal generated according to the encoding method described above.
[0137] One or more of the present embodiments also provide a computer-readable storage medium having stored thereon instructions for encoding or decoding video data according to at least a portion of any of the above methods. One or more embodiments also provide a computer-readable storage medium having stored thereon a bitstream generated according to the above encoding methods. One or more embodiments also provide methods and apparatus for transmitting or receiving a bitstream generated according to the above encoding methods. One or more embodiments also provide a computer program product including instructions for performing at least a portion of any of the above methods.
[0138] In one embodiment, the scan pattern is determined depending on whether the block is a square block, a horizontal rectangular block, or a vertical rectangular block.
[0139] In one embodiment, if the block is a rectangular vertical block, the coefficients are scanned according to a horizontal scan pattern, and if the block is a rectangular horizontal block, the coefficients are scanned according to a vertical scan pattern.
[0140] In one embodiment, if the ratio between the maximum dimension of the block and the shortest dimension of the block is greater than or equal to a certain value, the coefficients are scanned according to a vertical scan pattern if the block is a rectangular horizontal block, and according to a horizontal scan pattern if the block is a rectangular vertical block.
[0141] In one embodiment, if the block is a rectangular block, the coefficients are scanned according to a scan pattern selected from a set of patterns by a rate-distortion exhaustive search, where the selected scan pattern is coded into the bitstream.
[0142] In one embodiment, if a ratio between the longest dimension of the block and the smallest dimension of the block is greater than or equal to another value and the block is a rectangular block, the coefficients are scanned according to a scan pattern selected by a rate-distortion exhaustive search from among a set of patterns, where the selected scan pattern is encoded into the bitstream.
[0143] In one embodiment, if the block is a rectangular block, the coefficients are scanned according to a scan pattern defined such that the scan step size along the longest dimension is larger than the scan step size along the shortest dimension.
[0144] In one embodiment, if the block is a multi-hypothesis intra / inter predicted block, then vertical scanning is applied if the intra prediction mode is horizontal, and horizontal scanning is applied if the intra prediction mode is vertical.
[0145] In one embodiment, if the block is a multi-hypothesis predicted block, the coefficients are scanned according to a scan pattern selected from a set of patterns by a rate-distortion exhaustive search, where the selected scan pattern is coded into the bitstream.
[0146] In one embodiment, if the block is a triangular predicted block, the coefficients are scanned according to a scan pattern selected from a set of patterns by a rate-distortion exhaustive search, where the selected scan pattern is coded into the bitstream.
[0147] In one embodiment, obtaining the transform coefficients of the block includes transforming a residual of the block using at least one transform to obtain the transform coefficients, and the scan pattern is determined depending on the type of transform, the type of transform being defined by its low frequencies being constant and increasing or decreasing.
[0148] In one embodiment, if a constant low-frequency transform is applied both horizontally and vertically, the coefficients are scanned according to a diagonal scan pattern, and if one of the horizontal and vertical transforms is not a constant low-frequency transform, the coefficients are scanned according to a scan pattern selected by a rate-distortion exhaustive search from a set of patterns, and the selected scan pattern is encoded into the bitstream.
[0149] In one embodiment, if the vertical and horizontal transforms are DCT-2, the coefficients are scanned according to a diagonal scan pattern, and if the vertical and horizontal transforms are selected from DCT-7 and DCT-8 transforms, the coefficients are scanned according to a scan pattern selected by a rate-distortion exhaustive search from a set of patterns, and the selected scan pattern is encoded into the bitstream.
Claims
1. applying the transform function to the residual block of the current block of samples to obtain transform coefficients; scanning the transform coefficients according to a scan pattern for entropy coding; Equipped with Depending on the distribution statistics of the first class of the transform coefficients obtained for a first slope of the basis function of the transform function at the lowest frequency, the transform coefficients are scanned according to a first scanning order, otherwise the transform coefficients are scanned according to a second scanning order different from the first scanning order. Encoding method.
2. The method of claim 1 , wherein the scanning order is adapted to distribution statistics of the transform coefficients obtained from the transform function according to a slope of the basis function of the transform function at the lowest frequency.
3. 2. The method of claim 1 , wherein the first scan order is selected by a rate-distortion exhaustive search from among a set of scan orders that includes a scan order different from the second scan order, and the selected first scan order is encoded within video data representing at least the current block.
4. entropy decoding the transform coefficients of the current block ordered according to the scan pattern; obtaining a residual block of the current block from the transform coefficients and the scan pattern; Equipped with in response to a first slope of a basis function of the transform function at the lowest frequency providing a distribution statistic of a first class of the transform coefficients, the transform coefficients are scanned according to a first scanning order, otherwise the transform coefficients are scanned according to a second scanning order different from the first scanning order; Decryption method.
5. 5. The method of claim 4, wherein the scanning order is adapted to the distribution statistics of the transform coefficients obtained from the transform function according to the slope of the basis functions of the transform function at the lowest frequency.
6. applying the transform function to the residual block of the current block of samples to obtain transform coefficients; scanning the transform coefficients according to a scan pattern for entropy coding; 1. A coding device comprising one or more processors configured to execute a coding process for a first class of transform coefficients obtained for a first slope of a basis function of the transform function at a lowest frequency, the transform coefficients being scanned according to a first scanning order, and otherwise the transform coefficients being scanned according to a second scanning order different from the first scanning order.
7. 7. The encoding device of claim 6, wherein the scanning order is adapted to the distribution statistics of the transform coefficients obtained from the transform function according to the slope of the basis functions of the transform function at the lowest frequency.
8. 7. The encoding apparatus of claim 6, wherein the first scan order is selected by a rate-distortion exhaustive search from among a set of scan orders that includes a scan order different from the second scan order, and the selected first scan order is encoded within video data representing at least the current block.
9. entropy decoding the transform coefficients of the current block ordered according to the scan pattern; obtaining a residual block of the current block from the transform coefficients and the scan pattern; 1. A decoding device comprising one or more processors configured to execute: wherein, depending on a first slope of a basis function of a transform function at a lowest frequency that provides distribution statistics of a first class of the transform coefficients, the transform coefficients are scanned according to a first scanning order, and otherwise, the transform coefficients are scanned according to a second scanning order different from the first scanning order.
10. 10. The decoding device of claim 9, wherein the scanning order is adapted to a distribution statistic of the transform coefficients obtained from the transform function according to a slope of the basis function of the transform function at the lowest frequency.
11. A non-transitory information storage means storing program code instructions for carrying out the method of claim 1.
12. A non-transitory information storage means storing program code instructions for carrying out the method of claim 4.
Citation Information
Patent Citations
Moving picture encoding apparatus and moving picture decoding apparatus
JP2003204550A
Video coding using mapped transformations and scanning modes
JP2014504820A
Method and apparatus for scan order selection
WO2017138832A1
Image encoding / decoding method and device, and recording medium storing bitstream
WO2019027241A1
