Method and apparatus for picture encoding and decoding

Abstract

Methods and apparatus for picture encoding and decoding are provided. Wherein the encoding method comprises: applying a transform function to a residual block of a current block to obtain a transform coefficient; and scanning the coefficient according to a scanning mode to perform entropy coding, wherein the transform coefficients are scanned according to a first scan order in response to a first type of a basis function of the transform function at the lowest frequency, otherwise, the transform coefficients are scanned according to a second scan order. The transform coefficients are scanned according to a second scan order selected by an exhaustive rate distortion search among a plurality of scan orders including the first scan order and at least one additional scan order different from the first scan order.

Description

[0001] This application is a divisional application of the invention patent application with the application date of December 2, 2019, application number 201980080416.X, and invention name “Method and device for image encoding and decoding”. Technical Field

[0002] At least one embodiment of the present invention generally relates to a method and apparatus for picture encoding and decoding, and more specifically, to a method and apparatus for picture encoding and decoding, wherein scanned quantized transform coefficients are lengthened according to a scan pattern for entropy encoding and entropy decoding. Background Art

[0003] In order to achieve high compression efficiency, image and video coding schemes usually use prediction and transformation to exploit spatial and temporal redundancy in video content. Typically, intra-frame or inter-frame prediction is used to exploit frame correlation within or between frames, and then the difference between the original image block and the predicted image block, which is usually represented as a prediction error, prediction residual or predictor, is transformed, quantized and entropy encoded. During encoding, the original image block is usually divided / split into sub-blocks that may be divided using a quadtree. In order to reconstruct the video, the compressed data is decoded by the inverse process corresponding to prediction, transformation, quantization and entropy coding. Summary of the invention

[0004] According to a general aspect of at least one embodiment, a coding method is proposed, comprising: - applying a transform function to a residual block of a current block to obtain transform coefficients; and - scanning the coefficients according to a scanning pattern for entropy encoding; wherein in response to a first type of basis function of the transform function at the lowest frequency, the transform coefficients are scanned according to a first scanning order, otherwise, the transform coefficients are scanned according to a second scanning order selected by an exhaustive rate-distortion search from a plurality of scanning orders including the first scanning order and at least one additional scanning order different from the first scanning order.

[0005] According to a general aspect of at least one embodiment, a decoding method is proposed, comprising: - entropy decoding transform coefficients of a current block sorted according to a scanning pattern; and - obtaining a residual block of the current block from the transform coefficients and the scanning pattern of applying a transform function; wherein in response to a first type of basis function of the transform function at the lowest frequency, the transform coefficients are scanned according to a first scanning order, otherwise, the transform coefficients are scanned according to a second scanning order among a plurality of scanning orders including the first scanning order and at least one additional scanning order different from the first scanning order.

[0006] According to a general aspect of at least one embodiment, a coding device is proposed, comprising one or more processors, which are configured to perform: - applying a transform function to a residual block of a current block to obtain transform coefficients; and - scanning the coefficients according to a scanning pattern for entropy encoding; wherein in response to a first type of basis function of the transform function at the lowest frequency, the transform coefficients are scanned according to a first scanning order, otherwise, the transform coefficients are scanned according to a second scanning order selected by an exhaustive rate-distortion search from a plurality of scanning orders including the first scanning order and at least one additional scanning order different from the first scanning order.

[0007] According to a general aspect of at least one embodiment, a decoding device is proposed, comprising one or more processors, which are configured to perform: - entropy decoding of transform coefficients of a current block sorted according to a scanning pattern; and - obtaining a residual block of the current block from the transform coefficients and the scanning pattern of applying a transform function; wherein in response to a first type of basis function of the transform function at the lowest frequency, the transform coefficients are scanned according to a first scanning order, otherwise, the transform coefficients are scanned according to a second scanning order among a plurality of scanning orders including the first scanning order and at least one additional scanning order different from the first scanning order.

[0008] According to a general aspect of at least one embodiment, a decoding method is proposed, comprising:

[0009] Transform coefficients of a block scanned according to a scanning pattern are entropy decoded, wherein the scanning pattern is determined in response to a shape of the block.

[0010] According to a general aspect of at least one embodiment, a decoding method is proposed, comprising:

[0011] Entropy decoding is performed on transform coefficients of a block scanned according to a scanning mode, wherein, in the case where the block is a multi-hypothesis intra-inter prediction block, vertical scanning is applied when the intra prediction mode is horizontal, and horizontal scanning is applied when the intra prediction mode is vertical.

[0012] According to a general aspect of at least one embodiment, a decoding method is proposed, comprising:

[0013] Transform coefficients of a block scanned according to a scan pattern are entropy decoded, wherein the coefficients are scanned according to a scan pattern selected by an exhaustive rate-distortion search in a set of patterns if the block is a triangle prediction block, wherein the selected scan pattern is encoded.

[0014] According to a general aspect of at least one embodiment, a decoding method is proposed, comprising:

[0015] Transform coefficients of a block scanned according to a scanning pattern are entropy decoded, wherein the scanning pattern is determined in response to a type of inverse transform used to inverse transform the coefficients, wherein the type of transform is defined by whether its low frequency basis function is constant, increasing or decreasing.

[0016] According to a general aspect of at least one embodiment, a coding method is proposed, comprising:

[0017] - obtaining transform coefficients of the block; and

[0018] - scanning the coefficients according to a scanning pattern for entropy coding;

[0019] Wherein a scanning pattern is determined in response to the shape of the block.

[0020] According to a general aspect of at least one embodiment, a coding method is proposed, comprising:

[0021] - obtaining transform coefficients of the block; and

[0022] - scanning the coefficients according to a scanning pattern for entropy coding;

[0023] Wherein, in case the block is a multiple-hypothesis intra-inter predicted block, vertical scanning is applied if the intra prediction mode is horizontal, and horizontal scanning is applied if the intra prediction mode is vertical.

[0024] According to a general aspect of at least one embodiment, a coding method is proposed, comprising:

[0025] - obtaining transform coefficients of the block; and

[0026] - scanning the coefficients according to a scanning pattern for entropy coding;

[0027] Wherein in case the block is a triangle prediction block, the coefficients are scanned according to a scan pattern selected by an exhaustive rate-distortion search in a set of patterns, wherein the selected scan pattern is encoded.

[0028] According to a general aspect of at least one embodiment, a coding method is proposed, comprising:

[0029] - transforming the residual of the block using at least one transform to obtain transform coefficients; and

[0030] - scanning the coefficients according to a scanning pattern for entropy coding;

[0031] The scanning pattern is determined in response to a type of the transform, wherein the type of transform is defined by whether its low frequency basis function is constant, increasing or decreasing.

[0032] According to a general aspect of at least one embodiment, a decoding apparatus is provided, which includes one or more processors configured to implement:

[0033] Transform coefficients of a block scanned according to a scanning pattern are entropy decoded, wherein the scanning pattern is determined in response to a shape of the block.

[0034] According to a general aspect of at least one embodiment, a decoding apparatus is provided, which includes one or more processors configured to implement:

[0035] Entropy decoding is performed on transform coefficients of a block scanned according to a scanning mode, wherein, in the case where the block is a multi-hypothesis intra-inter prediction block, vertical scanning is applied when the intra prediction mode is horizontal, and horizontal scanning is applied when the intra prediction mode is vertical.

[0036] According to a general aspect of at least one embodiment, a decoding apparatus is provided, which includes one or more processors configured to implement:

[0037] Transform coefficients of a block scanned according to a scan pattern are entropy decoded, wherein the coefficients are scanned according to a scan pattern selected by an exhaustive rate-distortion search in a set of patterns if the block is a triangle prediction block, wherein the selected scan pattern is encoded.

[0038] According to a general aspect of at least one embodiment, a decoding apparatus is provided, which includes one or more processors configured to implement:

[0039] Transform coefficients of a block scanned according to a scanning pattern are entropy decoded, wherein the scanning pattern is determined in response to a type of inverse transform used to inverse transform the coefficients, wherein the type of transform is defined by whether its low frequency basis function is constant, increasing or decreasing.

[0040] According to a general aspect of at least one embodiment, a coding apparatus is provided, which includes one or more processors configured to implement:

[0041] - obtaining transform coefficients of the block; and

[0042] - scanning the coefficients according to a scanning pattern for entropy coding;

[0043] Wherein a scanning pattern is determined in response to the shape of the block.

[0044] According to a general aspect of at least one embodiment, a coding apparatus is provided, which includes one or more processors configured to implement:

[0045] - obtaining transform coefficients of the block; and

[0046] - scanning the coefficients according to a scanning pattern for entropy coding;

[0047] Wherein, in case the block is a multiple-hypothesis intra-inter predicted block, vertical scanning is applied if the intra prediction mode is horizontal, and horizontal scanning is applied if the intra prediction mode is vertical.

[0048] According to a general aspect of at least one embodiment, a coding apparatus is provided, which includes one or more processors configured to implement:

[0049] - obtaining transform coefficients of the block; and

[0050] - scanning the coefficients according to a scanning pattern for entropy coding;

[0051] Wherein in case the block is a triangle prediction block, the coefficients are scanned according to a scan pattern selected by an exhaustive rate-distortion search in a set of patterns, wherein the selected scan pattern is encoded.

[0052] According to a general aspect of at least one embodiment, a coding apparatus is provided, comprising one or more processors configured to implement:

[0053] - transforming the residual of the block using at least one transform to obtain transform coefficients; and

[0054] - scanning the coefficients according to a scanning pattern for entropy coding;

[0055] The scanning pattern is determined in response to the type of the transformation, wherein the type of transformation is defined by whether its low frequency odd function is constant, increasing or decreasing.

[0056] According to another general aspect of at least one embodiment, a bitstream is formatted to include a signal generated according to the encoding method described above.

[0057] One or more of the present embodiments further provide a computer program product comprising instructions, wherein when the instructions are executed by a processor, the processor is caused to implement the above method.

[0058] One or more of the present embodiments also provide a computer-readable storage medium storing instructions for encoding or decoding video data according to at least part of any of the above methods. One or more embodiments also provide a computer-readable medium storing a bitstream generated according to the above encoding method. One or more embodiments also provide a method and apparatus for sending or receiving a bitstream generated according to the above encoding method. One or more embodiments also provide a computer program product including instructions for executing at least part of any of the above methods. BRIEF DESCRIPTION OF THE DRAWINGS

[0059] Figure 1 Depicts several scanning modes in 8×8 blocks as used in HEVC;

[0060] Figure 2 and Figure 3 depicts a decision table associating a scan pattern with each block size;

[0061] Figure 4 is a flowchart of a decoding method according to an embodiment;

[0062] Figure 5 is a flow chart of an encoding method according to one embodiment;

[0063] Fig. 6A and Figure 6B depicts a decision table associating a scan mode with each block size according to one embodiment;

[0064] Figure 7 shows a conventional zigzag scan in a 4×4 block;

[0065] Figure 8 shows various scanning modes for rectangular blocks according to one embodiment;

[0066] Fig. 9 depicts a multi-hypothesis prediction block in case of a combination of inter and intra modes;

[0067] Fig.10 Various weightings for multi-hypothesis intra-inter prediction blocks for different intra prediction directions are shown;

[0068] Fig.11 depicts the blocks divided by the diagonal dividing lines and the associated weighting of the diagonal edges;

[0069] Fig.12 shows a block diagram of a video encoder according to an embodiment;

[0070] Fig.13 A block diagram of a video decoder according to an embodiment is shown; and

[0071] Fig.14 A block diagram is shown of an example of a system in which various aspects and embodiments are implemented. DETAILED DESCRIPTION

[0072] In HEVC encoding, pictures are divided into CTUs of a configurable size, typically 64×64 squares. In other video coding standards, the size can be 128×128 or 256×256. The CTU is the root of a quadtree, which is divided into 4 square coding units (CUs) of equal size (i.e., half the size of the parent block in width and height). A quadtree is a tree in which a parent node can be split into four child nodes, each of which can become a parent node for being split into four child nodes again. In HEVC, a coding block (CB) is divided into one or more prediction blocks (PBs) and forms the root of a quadtree divided into transform blocks (TBs). Corresponding to the coding blocks, prediction blocks, and transform blocks, a coding unit (CU) includes a prediction unit (PU) and a tree-structured transform unit (TU) set, the PU includes prediction information for all color components, and the TU includes a residual coding syntax structure for each color component. The sizes of the CB, PB, and TB of the luminance component apply to the corresponding CU, PU, ​​and TU.

[0073] In recent coding systems, CTU is the root of the coding tree divided into coding units (CU). The coding tree is a tree in which a parent node (usually corresponding to a block) can be split into child nodes (e.g., split into 2, 3 or 4 child nodes), and each child node can become a parent node for being split into child nodes again. In addition to the quadtree partitioning mode, new partitioning modes (binary tree symmetric partitioning mode, binary tree asymmetric partitioning mode, and ternary tree partitioning mode) are defined, which increases the total number of possible partitioning modes. The coding tree has a unique root node, such as CTU. The leaves of the coding tree are the terminal nodes of the tree. Each node of the coding tree represents a block that can be further divided into smaller blocks (also called sub-blocks). Once the partition that changes the CTU into a CU is determined, the CU corresponding to the leaf of the coding tree is encoded. The partition that changes the CTU into a CU and the coding parameters for encoding each CU (corresponding to the leaf of the coding tree) can be determined on the encoder side by a rate distortion optimization program.

[0074] In this application, the term "block" or "picture block" may be used to refer to any one of CTU, CU, PU, ​​TU, CB, PB, and TB. In addition, the term "block" or "picture block" may be used to refer to macroblocks, partitions, and subblocks specified in H.264 / AVC or other video coding standards, and more specifically may refer to sample arrays of many sizes.

[0075] In this application, the terms "reconstruction" and "decoding" can be used interchangeably, the terms "pixel" and "sample" can be used interchangeably, and the terms "image", "picture" and "frame" can be used interchangeably. Usually, but not necessarily, the term "reconstruction" is used on the encoder side and "decoding" is used on the decoder side.

[0076] In this application, the terms "scanning order" and "scanning mode" may be used interchangeably.

[0077] In HEVC, pictures are divided into so-called coding tree units (CTUs), and each CTU is represented by a coding unit (CU) in the compressed domain. Each CU is then given some intra or inter prediction parameters. To do this, it is spatially divided into one or more prediction units (PUs), and each PU is assigned some prediction information (e.g., intra prediction mode, reference index, motion vector). Intra or inter coding modes are assigned on the CU level.

[0078] After segmentation, intra-frame or inter-frame prediction is used to exploit frame correlation within or between frames, and then the difference between the original block and the predicted block, which is usually represented as a prediction error or prediction residual, is transformed, quantized, and entropy encoded. To reconstruct the video, the compressed data is decoded by the inverse process corresponding to entropy encoding, quantization, transformation, and prediction.

[0079] In HEVC, quantized transform coefficients are encoded using non-overlapping coefficient groups (CGs). The CG contains coefficients for a 4×4 block of a coding block. As an example, in Figure 1 The CG contained in an 8×8 block is shown in FIG. The CG in the coding block and the 16 transform coefficients within the CG are scanned and encoded according to the scanning mode selected in the three scanning orders (diagonal, horizontal, vertical). For inter-frame blocks, the CG is always used. Figure 1 Diagonal scan from the left. Figure 2The type of scanning mode to be used for each intra-frame block size is indicated. In the figure, the first vertical line indicates the block height, and the first horizontal line indicates the block width, D indicates the use of diagonal scanning mode, and M indicates the use of mode-dependent coefficient scanning (ModeDependent Coefficient Scanning, MDCS). In HEVC, this mode is only used for 4×4 and 8×8 intra-frame blocks. Therefore, for 4×4 and 8×8 intra-frame blocks, the scanning order depends on the intra-frame prediction mode that is valid for the block. MDCS is not currently implemented in VTM-3, which is a software model for VVC (Universal Video Coding). However, JVET-L0379, a contribution to the standard body, proposes to use MDCS in VVC and extend it to some rectangular block sizes introduced by quadtree (QT) / binary tree (BT) / ternary tree (TT) partitioning in VTM-3. QT is a partition that turns a block into 4 identical sub-blocks. BT is a partition that turns a block into two identical rectangles by vertical or horizontal lines. TT is the division of a block into 3 identical rectangles by two vertical lines or two horizontal lines.

[0080] In the case of MDCS for 4×4 and 8×8 blocks in HEVC, the scanning mode of the intra block is determined by the intra prediction mode using a fixed lookup table (LUT). The LUT for luma blocks in HEVC is given in Table 1: vertical modes 6-14 use horizontal scanning, horizontal modes 22-30 use vertical scanning, and other modes use diagonal scanning.

[0081]

[0082]

[0083] Table 1

[0084] In JVET-L0379, the Figure 2 Another decision table is used for switching between MDCS and diagonal scanning. Figure 3 This other decision table is described. The table is based on the ratio of the height to the width of the block. MDCS is applied when the ratio between the larger dimension of the block and its smaller dimension is equal to or greater than four. MDCS is also applicable to blocks of size 4×4, 4×8, 8×4 and 8×8. It should be noted that for luma, the maximum transform block size is 64×64 and for chroma, the maximum transform block size is 32×32. One disadvantage of this solution is that the scanning order of the coefficients depends only on the intra prediction mode of the intra blocks. Furthermore, there is no adaptive scanning order for inter blocks, i.e., inter blocks always use a diagonal scanning mode.

[0085] Entropy coding / decoding of transform coefficients consists of up to 5 passes, each dedicated to the coding of the syntax elements in the following list:

[0086] sig_flag or significant-coeff-flag: the validity of the coefficient, which indicates whether the value of the coefficient is equal to 0;

[0087] gt1_flag or coeff-abs-level-greater1-flag: indicates whether the absolute value of the coefficient level is greater than 1;

[0088] gt2_flag or coeff-abs-level-greater2-flag: indicates whether the absolute value of the coefficient level is greater than 2;

[0089] par_flag: indicates the parity of the coefficients when the dependent quantization method is used;

[0090] coeff-abs-level-remaining: the remaining value of the absolute value of the coefficient level (in case the coefficient value is larger than the coefficient value encoded in the previous pass);

[0091] coeff-sign-flag: The sign of the significant coefficient (0: positive, 1: negative).

[0092] Each scan pass follows a scan pattern.

[0093] In each pass, syntax is encoded only if necessary, as determined by the previous pass. For example, if the sig_flag of a coefficient is 0, then the remaining passes to encode the absolute value of that coefficient are unnecessary. Similarly for sign encoding, coeff-sign-flag is sent only for non-zero coefficients.

[0094] Before processing the next CG, all scans are performed for a given CG until all quantized coefficients in the CG can be reconstructed. The scan starts from the last significant coefficient in the coding block, sweeps from high-frequency coefficients to low-frequency coefficients, and then processes all coefficients until the DC coefficient.

[0095] The quantized transform coefficients are entropy encoded to output a bitstream. As a non-limiting example, context-based adaptive binary arithmetic coding (CABAC) can be used to encode syntax elements into a bitstream. In order to encode with CABAC, non-binary syntax element values ​​are mapped to binary sequences (called binary bit (bin) strings). For the binary bits, a context model is selected. The context model stores the possibility that each binary bit is "1" or "0", and the context model can be adaptive or static. The static model triggers the coding engine with equal possibilities of binary bits "0" or "1". In the adaptive coding engine, the context model is updated based on the actual encoded value of the binary bit. The operating modes corresponding to the adaptive model and the static model are respectively referred to as normal mode and bypass mode.

[0096] At least one embodiment aims to improve compression efficiency compared to existing video compression systems.

[0097] Therefore, at least one embodiment proposes to effectively adjust the scanning order of quantized transform coefficients in a way that provides good compression efficiency (rate-distortion performance) and minimizes the increase in complexity of the coding design.

[0098] Thus, at least one embodiment proposes some schemes to adapt the scanning order of quantized transform coefficients to one or more of block shape, prediction tool, and transform type.

[0099] In the following, for the sake of simplicity, the term "transform coefficient" is used instead of "quantized transform coefficient".

[0100] After transformation, the energy of the residual information is highly compressed to the upper left corner of the coding block, which indicates that the DC coefficient and the low-frequency transform coefficient have more information and higher absolute values ​​than the transform coefficients located in the high-frequency region.

[0101] A scanning pattern that follows a descending order of the probability that the coefficients are non-zero provides more accurate CABAC probability modeling. Diagonal scanning arranges the two-dimensional (2-D) transform coefficients into a one-dimensional (1-D) array according to their frequencies in this descending order. However, since this single scanning pattern is not applicable to the coefficient values, there is room for improvement by adopting an adaptive scanning order.

[0102] Figure 4 A flow chart of an encoding method according to a general embodiment is depicted. At S100, a scanning pattern is determined according to the shape of a block to be decoded. At S110, entropy decoding is performed on the transform coefficients of the block scanned according to the determined scanning pattern.

[0103] Figure 5A flowchart of a decoding method according to a general embodiment is depicted. At S200, a scanning pattern is determined according to the shape of a block to be encoded. At S210, entropy encoding is performed on transform coefficients of the block scanned according to the determined scanning pattern.

[0104] The first embodiment proposes a general adaptive scan order scheme that is directly related to the block shape.

[0105] The intra prediction mode is related to the block shape: horizontal blocks are more likely to choose horizontal intra prediction mode. In addition, the frequency region is also related to the block shape. In fact, when the block is rectangular, a larger number of zero-valued coefficients are located in the high-frequency region along the longer dimension than the shorter dimension. Therefore, if diagonal scanning is used, the low-frequency and high-frequency coefficients may be interlaced too much. Take the vertical block as an example. Along the vertical direction, which is the longer dimension, more coefficients tend to be zero in the high-frequency region. By using coefficient scanning directed in the horizontal direction instead of diagonal scanning, the non-zero coefficients in the 2D matrix can be further compressed towards the beginning of the 1D vector. This in turn improves the entropy coding efficiency.

[0106] The basic idea according to the first embodiment is that the scanning order selection depends on the block shape, which can also be called "block shape-based coefficient scanning". For square blocks, diagonal scanning or MDCS can be applied. For rectangular blocks, an adaptive scanning order is used. The shape of a rectangular block is defined by the values ​​of its height and width and its orientation. Therefore, blocks of size 8×16 (vertical blocks) and blocks of size 16×8 (horizontal blocks) have different shapes.

[0107] In a first variant, vertical scanning is used with horizontal blocks and horizontal scanning is used with vertical blocks. Otherwise only diagonal scanning or MDCS is applied. Fig. 6A A second variant is shown.

[0108] In a second variant, in case the ratio of the larger block dimension to the smaller block dimension is equal to or greater than a value T, vertical scanning is used for horizontal blocks and horizontal scanning is used for vertical blocks. Otherwise, only diagonal scanning or MDCS is applied. In case the value T is equal to 4, the second variant is as follows Figure 6B shown.

[0109] In a third variant, for rectangular blocks, the best scanning order is selected by exhaustive search in a set of scanning modes (e.g., traditional diagonal, horizontal, and vertical scanning modes). The exhaustive search may also be performed in another set of scanning modes, and is not limited to the above list. The best scanning order for rectangular blocks is selected based on rate-distortion (RD) estimation, and the best scanning order for rectangular blocks is signaled to the decoder by encoding an additional syntax element tu_coeffscan_type. Therefore, the scanning mode that provides the best rate-distortion compromise is selected in the set of scanning modes. In other cases, only diagonal scanning or MDCS is applied.

[0110] In a fourth variant, when the ratio of the dimension of the larger block to the dimension of the smaller block is equal to or greater than a value T, the best scanning order is selected by an exhaustive search in a set of scanning modes (e.g., traditional diagonal, horizontal and vertical scanning modes). The exhaustive search can also be done in another set of scanning modes and is not limited to the above list. The best scanning order for the blocks is selected based on the RD estimate and is signaled to the decoder by encoding an additional syntax element tu_coeffscan_type as indicated in the syntax table. In other cases, only diagonal scanning or MDCS is applied.

[0111] As indicated for variants three and four, additional syntax elements are encoded / decoded. Details of how to signal the best scan order type selected by exhaustive search in diagonal, horizontal and vertical scan modes based on RD estimation are given in the syntax table below. Specific signaling is indicated in italics.

[0112]

[0113]

[0114]

[0115]

[0116]

[0117]

[0118] tu_coeffscan_type specifies the coefficient scanning type selected from, for example, diagonal, horizontal, and vertical scanning for the transform block. The value of tu_coeffscan_type can be 0, 1, or 2, which can represent diagonal (0), horizontal (1), and vertical (2), respectively. In a variant, tu_coeffscan_type specifies the coefficient scanning type / mode selected from another set of scanning modes, the other set of scanning modes including scanning modes other than diagonal, horizontal, and vertical scanning modes.

[0119] ScanOrder specifies the scan order array, which generates the positions of the coefficients in the scan order. The ScanOrder array is initialized differently for each scan type and is obtained by calling the function getScanOrder().

[0120] In another variant, additional adaptive scanning orders are defined that are different from the traditional diagonal, horizontal, and vertical scanning patterns. For this purpose, specific vertical and horizontal scanning orders are designed for rectangular blocks. Figure 7 As shown, the design of the adaptive scan order is based on a zigzag scan order. A specific vertical and horizontal scan order is designed so that the coefficients along the longest dimension in the scan order are compressed more than along the shortest dimension. According to this variant, the scan step size Step along the longest dimension is L will be the scan step along the shortest dimension S (Usually the default Step s =1). Figure 8 As shown, the scanning step size is the distance between the first coefficient and the last coefficient along one dimension before scanning the coefficient along another dimension. This multiple is the same as the longest dimension Size of the coding block. L and the shortest dimension Size S is related to the ratio of , as defined below:

[0121]

[0122] Figure 8 Some examples of proposed adaptive zig-zag scanning orders for rectangular blocks of size 8×4, 4×8 and 16×4 are given in .

[0123] As mentioned above, when MDCS is activated, an intra block can select a scan mode based on the intra prediction mode of the block. A limitation of MDCS or current algorithms is that no adaptive scan order scheme is defined for inter blocks. The diagonal scan mode is the only scan mode used with inter blocks. A second embodiment proposes a general adaptive scan order scheme for inter blocks that is dependent on the prediction tool used.

[0124] The new video codec VTM-3 implements more inter-coding tools (e.g., Multi-hypothesis prediction (MH)) to improve coding efficiency. The general concept of MH is to combine the inter-prediction performed in merge mode with the intra-prediction mode or with another inter-prediction mode (e.g., unidirectional prediction AMVP, skip, and merge). In merge mode, a list of merge candidates (reference index, motion value) is established, and a merge index identifying a candidate is signaled. The motion information of the block in merge mode is then derived from the candidate identified by the merge index and used for motion compensated prediction. As Fig. 9 As shown, in the inter-frame intra MH, the final prediction is a weighted average of the prediction of the merged index and the prediction generated by the intra-frame prediction mode, where different weights are applied depending on the combination. The current block is divided into 4 regions of equal area. As the region moves away from the intra-frame reference sample, the weight gradually decreases. Each weight set (denoted as (w_intrai, w_interi), where i is 1 to 4 and (w_intra1, w_inter1) = (6, 2), (w_intra2, w_inter2) = (5, 3), (w_intra3, w_inter3) = (3, 5) and (w_intra4, w_inter4) = (2, 6)) will be applied to the corresponding region. Fig.10 An example of intra vertical / horizontal prediction and intra DC / plane prediction used in multiple hypotheses is depicted in FIG.

[0125] According to a second embodiment, adaptive scanning order is extended to inter blocks depending on the prediction tool (eg, whether it uses some specific inter prediction tool such as MH, OBMC, triangle prediction, etc.) For multi-hypothesis predicted blocks, adaptive scanning order is used.

[0126] In a first variant, MDCS is reused for multi-hypothesis intra-inter predicted blocks regardless of block size. Vertical scanning is applied in case the intra prediction mode is horizontal. In contrast, horizontal scanning is used for vertical intra prediction mode.

[0127] In a second variant, for a multi-hypothesis predicted block (i.e., either inter-intra or inter-inter), the best scanning order is selected by an exhaustive search in a set of scanning modes (e.g., the traditional diagonal, horizontal and vertical scanning modes). The exhaustive search may also be done in another set of scanning modes and is not limited to the above list. The best scanning order for the block is selected based on the RD estimate and is signaled to the decoder by encoding an additional syntax element tu_coeffscan_type as indicated in the syntax table.

[0128] Defines geometry (e.g., triangle) modes that allow more flexibility in dividing a picture into blocks before encoding. Fig.11 , various triangle splits are depicted above. The blocks are split using diagonal split lines. Fig.11 Additional coding performance can be obtained by performing weighting of the two predictors P1 and P2 on the CU (gray area at the bottom). The triangle prediction unit mode is only applied to CUs with block size greater than or equal to 8×8, and its coding prediction mode is one of the skip or merge modes. The triangle prediction mode has many benefits for scenes where objects move differently in the same block, or where a diagonal line divides the block into two partitions. Therefore, the triangle prediction mode is often applied to uneven blocks. After transformation and quantization, the low-frequency and high-frequency transform coefficients may still be too interlaced if only diagonal scanning is used.

[0129] In a third variant, for triangular prediction blocks, the best scanning order is selected by an exhaustive search in a set of scanning modes (e.g., the traditional diagonal, horizontal and vertical scanning modes). The exhaustive search can also be done in another set of scanning modes and is not limited to the above list. The best scanning order for the block is selected based on the RD estimate and is signaled to the decoder by encoding an additional syntax element tu_coeffscan_type as indicated in the syntax table.

[0130] Considering some relationships between activating adaptive scanning order and transform types applied to blocks, the third embodiment proposes a general adaptive scanning order scheme.

[0131] Multiple transform selection (MTS) is implemented in the current VTM-3. According to MTS, DST-7 and DCT-8 are used in addition to DCT-2. There are separate MTS enable flags for intra-frame and inter-frame at the SPS level. When MTS is enabled, a single flag is encoded / decoded to signal that DCT-2 is applied in both directions. When MTS is enabled and DCT-2 is not used in both directions, a 2-bit index (MTS_Idx) is encoded / decoded to signal 4 pairs of horizontal and vertical transforms. For this reason, DCT8 and DST7 are two possible transforms for the horizontal and vertical directions.

[0132] Some of the DCT / DST transform family can be classified into increasing, decreasing, and constant first basis functions as in Table 2. As described in Table 2, DCT-2 has a constant first basis function, DST-7 has an increasing first basis function, and DCT-8 has a decreasing first basis function. The first basis function represents the basis function of the transform under consideration at the lowest frequency. As an example, the DCT2 basis function T i (j), i,j=0,1,…,N-1 are defined as follows:

[0133] in

[0134] The first basis function is for i=0, so for DCT2, the first basis function is constant.

[0135] As another example, the DCT8 basis function T i (j),i,j=0,1,…,N-1 are defined as follows:

[0136]

[0137] The first basis function is for i=0, so for DCT8 the first basis function is reducing.

[0138] As another example, the DST7 basis function T i (j),i,j=0,1,…,N-1 are defined as follows:

[0139]

[0140] The first basis function is for i=0, so for DST7 the first basis function is increasing.

[0141] When 2-D DCT-2 is implemented by performing 1-D DCT-2 twice horizontally and vertically, respectively, the signal correlation along the horizontal or vertical direction can be captured by this separable 2-D transform. However, when there are directional edges, this 2-D DCT-2 may be suboptimal. Therefore, different combinations of DST-7 and DCT-8 applied in the horizontal and vertical directions, respectively, are complementary for these cases.

[0142] Although the distribution statistics of transform coefficients after adaptive transforms such as MTS show less mode-dependent characteristics, there is still some room for adaptive scanning order to avoid excessive interlaced low-frequency and high-frequency transform coefficients.

[0143] Transform First base type DCT1 Constant DCT2 Constant DCT3 reduce DCT4 reduce DCT5 Constant DCT6 Constant DCT7 reduce DCT8 reduce DST1 Constant DST2 Constant DST3 Increase DST4 Increase DST7 Increase DST8 Increase

[0144] Table 2

[0145] In a third embodiment, the scanning order selection depends on the transform type, which may also be referred to as "transform type dependent coefficient scanning".

[0146] In a first variant, in the case of using a separable transformation scheme, the following applies:

[0147] - applying a diagonal scan while applying a transform with a constant first basis function both in the horizontal and vertical directions for the block;

[0148] Otherwise, in case a transform with a constant first basis function is not used in one of the horizontal and vertical directions, an adaptive scanning order is activated for the block. For this purpose, the best scanning order is selected by an exhaustive search in a set of scanning patterns (e.g. the conventional diagonal, horizontal and vertical scanning patterns). The exhaustive search can also be done in another set of scanning patterns and is not limited to the above list. The best scanning order for the block is selected based on the RD estimation and by an additional syntax element as indicated in the syntax table

[0149] tu_coeffscan_type is encoded to signal the optimal scan order to the decoder.

[0150] In the second variant, in case of using MTS, the following applies:

[0151] - in case a separable 2-D DCT-2 transform is applied for the block, a diagonal scan is applied;

[0152] Otherwise, the adaptive scan order is activated with DST-7 and DCT-8 applied in the horizontal and vertical directions, respectively, for the block.

[0153] For example, the best scanning order is selected by exhaustive search in a set of conventional diagonal, horizontal and vertical scanning patterns. Exhaustive search can also be performed in another set of scanning patterns.

[0154] The optimal scanning order of blocks is selected based on the RD estimation and by adding additional syntax elements as indicated in the syntax table.

[0155] tu_coeffscan_type is encoded to signal the optimal scanning order to the decoder.

[0156] The first embodiment, the second embodiment and the third embodiment can be used completely independently or in combination. In an example, the second embodiment and the third embodiment can use the new scanning order defined in the first embodiment, for example, Figure 8In order to select a new scanning mode to be used for a block, the shape of the block is needed. Finally, for a rectangular block, a final scanning mode is selected among D, H, V and an additional new scanning mode based on the prediction tool and transform type as in the second and third embodiments.

[0157] If an additional syntax tu_coeffscan_type is applied, a context for entropy encoding / decoding thereof may be selected based on block shape, prediction tool and transform type, where tu_coeffscan_type represents a scanning order selected from a set of scanning modes (eg, diagonal, horizontal and vertical scanning modes).

[0158] The present application describes a number of aspects, including tools, features, embodiments, models, methods, etc. Many of these aspects are specifically described, and at least in order to illustrate the various characteristics, are usually described in a manner that sounds limited. However, this is for the purpose of describing clearly, and does not limit the application or scope of those aspects. In fact, all different aspects can be combined and interchanged to provide further aspects. In addition, these aspects can also be combined and interchanged with the aspects described in the previous documents.

[0159] The aspects described and contemplated in this application may be implemented in many different forms. Fig.12 , Fig.13 and Fig.14 Some embodiments are provided, but other embodiments are contemplated and are not intended to be construed as Fig.12 , Fig.13 and Fig.14 The discussion does not limit the breadth of implementation. At least one of the aspects generally relates to video encoding and decoding, and at least one other aspect generally relates to sending a generated or encoded bitstream. These and other aspects can be implemented as methods, apparatus, computer-readable storage media having stored thereon instructions for encoding or decoding video data according to any of the methods described, and / or computer-readable storage media having stored thereon bitstreams generated according to any of the methods described.

[0160] Various methods are described herein, and each method includes one or more steps or actions for implementing the method. Unless a specific order of steps or actions is required for proper operation of the method, the order and / or use of specific steps and / or actions may be adjusted or combined.

[0161] Various methods and other aspects described in this application can be used to adjust modules, such as Fig.12 and Fig.13The entropy encoding and / or decoding modules (145, 230) of the video encoder 100 and decoder 200 shown in . Furthermore, the present aspects are not limited to VVC or HEVC, and may be applied, for example, to other standards and recommendations (whether previously existing or developed in the future), as well as extensions of any such standards and recommendations (including VVC and HEVC). Unless otherwise indicated or technically excluded, the aspects described in this application may be used alone or in combination.

[0162] Various numerical values ​​are used in this application, for example, the value T. The specific values ​​are for example purposes, and the described aspects are not limited to these specific values.

[0163] Fig.12 An encoder 100 is shown. Although variations of the encoder 100 are contemplated, for purposes of clarity, the encoder 100 is described below without describing all contemplated variations.

[0164] Before being encoded, the video sequence may be processed before encoding (101), for example, applying a color transform to the input color picture (e.g., conversion from RGB 4:4:4 to YCbCr 4:2:0), or performing a remapping of the input picture components to obtain a signal distribution that is more resilient to compression (e.g., using histogram equalization of one of the color components). Metadata may be associated with the pre-processing and appended to the bitstream.

[0165] In encoder 100, a picture is encoded by encoder elements as described below. The picture to be encoded is divided (102) and processed in units such as CUs. Each unit is encoded using one of, for example, intra or inter modes. When a unit is encoded in intra mode, it performs intra prediction (160). In inter mode, motion estimation (175) and compensation (170) are performed. The encoder decides which of intra mode or inter mode is used to encode the unit, and indicates the intra / inter decision by, for example, a prediction mode flag. For example, a prediction residual is calculated by subtracting (110) the prediction block from the original image block.

[0166] The prediction residual is then transformed (125) and quantized (130). The quantized transform coefficients, along with motion vectors and other syntax elements, are entropy encoded (145) to output a bitstream. The encoder may skip the transform and apply quantization directly to the untransformed residual signal. The encoder may bypass both the transform and quantization, i.e., encode the residual directly without applying the transform or quantization process.

[0167] The encoder decodes the encoded block to provide a reference for further prediction. The quantized transform coefficients are dequantized (140) and inverse transformed (150) to decode the prediction residual. The decoded prediction residual and the prediction block are combined (155) to reconstruct the image block. An in-loop filter (165) is applied to the reconstructed picture to perform, for example, deblocking / SAO (Sample Adaptive Offset) filtering to reduce coding artifacts. The filtered image is stored in a reference picture buffer (180).

[0168] Fig.13 2 shows a block diagram of a video decoder 200. In the decoder 200, the bitstream is decoded by the decoder elements described below. The video decoder 200 generally performs the same Fig.12 The decoding pass is the inverse of the encoding pass described in . Encoder 100 also typically performs video decoding as part of encoding the video data.

[0169] Specifically, the input to the decoder includes a bitstream that may be generated by the video encoder 100. The bitstream is first entropy decoded (230) to obtain transform coefficients, motion vectors, and other encoded information. Picture partition information indicates how the picture is partitioned. Therefore, the decoder can partition (235) the picture according to the decoded picture partition information. The transform coefficients are dequantized (240) and inverse transformed (250) to decode the prediction residual. The image block is reconstructed by combining (255) the decoded prediction residual and the prediction block. The prediction block can be obtained (270) from intra-frame prediction (260) or motion compensated prediction (i.e., inter-frame prediction) (275). An in-loop filter (265) is applied to the reconstructed image. The filtered image is stored in a reference picture buffer (280).

[0170] The decoded picture may also be processed by post-decoding processing (285), such as an inverse color transform (e.g., a transform from YCbCr 4:2:0 to RGB 4:4:4) or an inverse remapping, which performs the inverse of the remapping process performed in the pre-encoding processing (101). The post-decoding processing may use metadata derived in the pre-encoding processing and signaled in the bitstream.

[0171] Fig.14A block diagram of an example of a system in which various aspects and embodiments are implemented is shown. System 1000 may be embodied as a device, which includes various components described below and is configured to perform one or more of the aspects described in this document. Examples of such devices include, but are not limited to, various electronic devices such as personal computers, laptop computers, smart phones, tablet computers, digital multimedia set-top boxes, digital television receivers, personal video recording systems, connected home appliances, and servers. The elements of system 1000 may be embodied in a single integrated circuit (IC), multiple ICs, and / or discrete components, either alone or in combination. For example, in at least one embodiment, the processing of system 1000 and encoder / decoder elements are distributed across multiple ICs and / or discrete components. In multiple embodiments, system 1000, for example, is coupled to one or more other systems or other electronic devices via a communication bus or through dedicated input and / or output ports, and is communicatively coupled. In multiple embodiments, system 1000 is configured to implement one or more of the aspects described in this document.

[0172] The system 1000 includes at least one processor 1010 configured to execute instructions loaded therein for implementing, for example, various aspects described in this document. The processor 1010 may include embedded memory, input and output interfaces, and various other circuits known in the art. The system 1000 includes at least one memory 1020 (e.g., a volatile memory device, and / or a non-volatile memory device). The system 1000 includes a storage device 1040, which may include a non-volatile memory and / or a volatile memory device, including, but not limited to, an electrically erasable programmable read-only memory (EEPROM), a read-only memory (ROM), a programmable read-only memory (PROM), a random access memory (RAM), a dynamic random access memory (DRAM), a static random access memory (SRAM), a flash memory, a magnetic disk drive, and / or an optical disk drive. The storage device 1040 may include an internal storage device, an attached storage device (including a removable and non-removable storage device), and / or a network accessible storage device.

[0173] System 1000 includes a coder / decoder module 1030 configured to, for example, process data to provide a coded video or a decoded video, and the coder / decoder module 1030 may include its own processor and memory. Coder / decoder module 1030 represents that (multiple) modules may be included in a device to perform coding and / or decoding functions. As is well known, a device may include one or both of the coding and decoding modules. In addition, the coder / decoder module 1030 may be implemented as a separate element of system 1000, or may be incorporated into the processor 1010 as a combination of hardware and software known to those skilled in the art.

[0174] Program code to be loaded onto the processor 1010 or encoder / decoder 1030 to perform various aspects described in this document may be stored in the storage device 1040 and subsequently loaded onto the memory 1020 for execution by the processor 1010. According to various embodiments, during execution of the processes described in this document, one or more of the processor 1010, memory 1020, storage device 1040, and encoder / decoder module 1030 may store one or more of various items. Such stored items may include, but are not limited to, input video, decoded video or portions of decoded video, bitstreams, matrices, variables, and intermediate or final results from the processing of equations, formulas, operations, and operational logic.

[0175] In some embodiments, memory internal to the processor 1010 and / or encoder / decoder 1030 is used to store instructions and provide working memory for processing required during encoding or decoding. However, in other embodiments, memory external to the processing device (e.g., the processing device may be the processor 1010 or the encoder / decoder module 1030) is used for one or more of these functions. The external memory may be memory 1020 and / or storage device 1040, such as dynamic volatile memory and / or non-volatile flash memory. In several embodiments, external non-volatile flash memory is used to store an operating system (e.g., of a television). In at least one embodiment, a fast external dynamic volatile memory such as RAM is used as working memory for video encoding and decoding operations, such as for MPEG-2 (MPEG refers to Moving Picture Experts Group, MPEG-2 is also known as ISO / IEC13818, and 13818-1 is also known as H.222, and 13818-2 is also known as H.262), HEVC (HEVC refers to High Efficiency Video Coding, also known as H.265 and MPEG-H Part 2), or VVC (Versatile Video Coding, a new standard being developed by JVET (Joint Video Experts Group)).

[0176] Inputs to the elements of system 1000 may be provided through various input devices as indicated in block 1130. Such input devices include, but are not limited to, (i) a radio frequency (RF) section that receives RF signals, such as those transmitted by a broadcast station over the air, (ii) a component (COMP) input terminal (or a collection of COMP input terminals), (iii) a universal serial bus (USB) input terminal, and / or (iv) a high-definition multimedia interface (HDMI) input terminal. Fig.10 Other examples shown in include composite video.

[0177] In various embodiments, the input device of block 1130 has an associated corresponding input processing unit as is known in the art. For example, the RF section may be associated with elements adapted to: (i) select a desired frequency (also referred to as selecting a signal or this limits the signal to a frequency band), (ii) downconvert the selected signal, (iii) again limit the signal to a narrower frequency band to select (for example) a channel which may be referred to in an embodiment as a frequency band, (iv) demodulate the downconverted and bandwidth limited signal, (v) perform error correction, and (vi) demultiplex to select a desired stream of data packets. The RF section of various embodiments includes one or more elements performing these functions, for example, a frequency selector, a signal selector, a band limiter, a channel selector, a filter, a downconverter, a demodulator, an error corrector, and a demultiplexer. The RF section may include a tuner that performs various of these functions, including, for example, downconverting a received signal to a lower frequency (for example, an intermediate frequency or near baseband frequency) or downconverting to baseband. In a set-top box embodiment, the RF part and its associated input processing element receive the RF signal sent by wired (for example cable) medium, and filter to the frequency band of expectation again by filtering, down-conversion and perform frequency selection.Various embodiments rearrange the order of above-mentioned (and other) elements, remove some in these elements, and / or add other elements that perform similar or different functions.Adding element can include inserting element between existing element, for example, inserting amplifier and analog-to-digital converter.In various embodiments, the RF part comprises antenna.

[0178] In addition, the USB and / or HDMI terminals may include corresponding interface processors for connecting the system 1000 to other electronic devices via USB and / or HDMI connections. It should be understood that various aspects of input processing (e.g., Reed-Solomon error correction) may be implemented, for example, in a separate input processing IC or in the processor 1010 as needed. Similarly, various aspects of USB or HDMI interface processing may be implemented, for example, in a separate interface IC or in the processor 1010 as needed. The demodulated, error-corrected, and demultiplexed streams are provided to various processing elements (including, for example, the processor 1010 and the encoder / decoder 1030) operating in combination with memory and storage elements to process the data streams as needed for presentation on output devices.

[0179] The various components of system 1000 may be arranged in an integrated housing. Within the integrated housing, suitable connection arrangements (e.g., internal buses known in the art, including inter-chip (I2C) buses, wiring, and printed circuit boards) may be used to interconnect the various components and send data between them.

[0180] The system 1000 includes a communication interface 1050 that enables communication with other devices via a communication channel 1060. The communication interface 1050 may include, but is not limited to, a transceiver configured to send and receive data over the communication channel 1060. The communication interface 1050 may include, but is not limited to, a modem or a network card, and the communication channel 1060 may be implemented, for example, within a wired and / or wireless medium.

[0181] In a number of embodiments, a wireless network such as a Wi-Fi (e.g., IEEE 802.11 (IEEE refers to the Institute of Electrical and Electronics Engineers)) network is used to stream data to the system 1000, or otherwise provide the system 1000. The Wi-Fi signals of these embodiments are received through a communication channel 1060 and a communication interface 1050 suitable for Wi-Fi communication. The communication channel 1060 of these embodiments is typically connected to an access point or router that provides an interface to an external network including the Internet to allow streaming applications and other over-the-top communications. Other embodiments use a set-top box to provide streaming data to the system 1000, which passes the data through the HDMI connection of the input block 1130. Still other embodiments use the RF connection of the input block 1130 to provide streaming data to the system 1000. As indicated above, various embodiments provide data in a non-streaming manner. In addition, various embodiments use wireless networks other than Wi-Fi, such as cellular networks or Bluetooth networks.

[0182] The system 1000 can provide output signals to various output devices including a display 1100, a speaker 1110, or other peripherals 1120. The display 1100 of various embodiments includes, for example, one or more of a touch screen display, an organic light emitting diode (OLED) display, a curved display, and / or a foldable display. The display 1100 can be used for a television, a tablet, a laptop computer, a mobile phone, or other devices. The display 1100 can also be integrated with other components (for example, as in a smart phone), or independent (for example, an external display screen of a laptop computer). In various examples of embodiments, other peripherals 1120 include one or more of an independent digital video disc (or digital versatile disc) (DVR, for both terms), a disk player, a stereo system, and / or a lighting system. Various embodiments use one or more peripherals 1120 that provide functions based on the output of the system 1000. For example, a disk player performs the function of playing the output of the system 1000.

[0183] In various embodiments, control signals are communicated between the system 1000 and the display 1100, speaker 1110, or other peripheral device 1120 using signaling such as AV.Link, Consumer Electronics Control (CEC), or other communication protocols that enable device-to-device control with or without user intervention. Output devices can be communicatively coupled to the system 1000 via dedicated connections through corresponding interfaces 1070, 1080, and 1090. Alternatively, the output device can be connected to the system 1000 using a communication channel 1060 via a communication interface 1050. The display 1100 and speaker 1110 can be integrated in a single unit with other components of the system 1000 in an electronic device (e.g., a television). In various embodiments, the display interface 1070 includes a display driver, such as a timing controller (TCon) chip.

[0184] For example, if the RF portion of input 1130 is part of a separate set-top box, then alternatively display 1100 and speaker 1110 may be separate from one or more of the other components. In embodiments where display 1100 and speaker 1110 are external components, output signals may be provided via dedicated output connections including, for example, an HDMI port, a USB port, or a COMP output.

[0185] The embodiments may be implemented by computer software implemented by the processor 1010 or by hardware, or by a combination of hardware and software. As a non-limiting example, the embodiments may be implemented by one or more integrated circuits. The memory 1020 may be of any type suitable for the technical environment and may be implemented using any suitable data storage technology, such as optical memory devices, magnetic memory devices, semiconductor-based memory devices, fixed memory, and removable memory, as non-limiting examples. The processor 1010 may be of any type suitable for the technical environment, and may include, as a non-limiting example, one or more of a microprocessor, a general-purpose computer, a special-purpose computer, and a processor based on a multi-core architecture.

[0186] Various implementations involve decoding. "Decoding," as used in this application, may encompass, for example, all or part of a process performed on a received encoded sequence to produce a final output suitable for display. In various embodiments, such a process includes one or more of the processes typically performed by a decoder (e.g., entropy decoding, inverse quantization, inverse transform, and differential decoding). In various embodiments, such a process also includes, or alternatively includes, a process performed by a decoder of the various implementations described in this application (e.g., determining a scan pattern for a block based on one or more of a block shape, a prediction tool, and a transform type).

[0187] As another example, in one embodiment "decoding" refers only to entropy decoding, in another embodiment "decoding" refers only to differential decoding, and in another embodiment "decoding" refers to a combination of entropy decoding and differential decoding. Whether the phrase "decoding process" is intended to refer specifically to a subset of operations or broadly to a broader decoding process will be clear based on the context of the particular description and is believed to be well understood by those skilled in the art.

[0188] Various implementations involve encoding. In a manner similar to the discussion above regarding "decoding", "encoding" as used in this application can encompass, for example, all or part of a process performed on an input video sequence to produce an encoded bitstream. In various embodiments, such a process includes one or more of the processes typically performed by an encoder (e.g., partitioning, differential encoding, transforms, quantization, and entropy encoding). In various embodiments, such a process also includes, or alternatively includes, a process performed by an encoder of the various implementations described in this application (e.g., determining a scanning pattern for a block based on one or more of a block shape, a prediction tool, and a transform type).

[0189] As another example, in one embodiment "encoding" refers only to entropy encoding, in another embodiment "encoding" refers only to differential encoding, and in another embodiment "encoding" refers to a combination of differential encoding and entropy encoding. Whether the phrase "encoding process" is intended to refer specifically to a subset of operations or broadly to a more extensive encoding process will be clear based on the context of the particular description and is believed to be well understood by those skilled in the art.

[0190] Note that the syntax elements (e.g., tu_coeffscan_type) used herein are illustrative terms. As such, they do not preclude the use of other syntax element names.

[0191] When a figure is presented as a flow chart, it should be understood that it also provides a block diagram of the corresponding apparatus. Similarly, when a figure is presented as a block diagram, it should be understood that it also provides a flow chart of the corresponding method / process.

[0192] Various embodiments relate to rate-distortion (RD) optimization. Specifically, during the encoding process, a balance or trade-off between rate and distortion is often considered, often given constraints on computational complexity. Rate-distortion optimization is typically formulated as minimizing a rate-distortion function that is a weighted sum of rate and distortion. There are a variety of methods for solving the rate-distortion optimization problem. For example, these methods can be based on extensive testing of all coding options, including all considered modes or coding parameter values, and a complete evaluation of their coding costs and associated distortions of the reconstructed signal after encoding and decoding. Faster methods can also be used to save coding complexity, particularly by calculating approximate distortions based on predictions or prediction residual signals rather than reconstructed signals. A mixture of these two methods can also be used, such as using approximate distortions only for some possible coding options and using full distortions for other coding options. Other methods evaluate only a subset of possible coding options. More generally, many methods employ any of a variety of techniques to perform optimization, but optimization is not necessarily a complete evaluation of both coding costs and associated distortions.

[0193] The implementations and aspects described herein may be implemented, for example, in methods or processes, devices, software programs, data streams, or signals. Even if discussed only in the context of a single form of implementation (e.g., discussed only as a method), the implementation of the features discussed may also be implemented in other forms (e.g., devices or programs). The device may be implemented, for example, with appropriate hardware, software, and firmware. The method may be implemented in a processor, for example, generally referring to a processing device, which includes, for example, a computer, a microprocessor, an integrated circuit, or a programmable logic device. The processor also includes communication devices, such as computers, mobile phones, portable / personal digital assistants ("PDAs"), and other devices that facilitate the communication of information between end users.

[0194] Reference to "one embodiment" or "an embodiment" or "an implementation" or "an implementation" and other variations thereof means that the specific features, structures, characteristics, etc. described in conjunction with the embodiment are included in at least one embodiment. Therefore, the presentation of the phrases "in one embodiment" or "in an embodiment" or "in an implementation" or "in an implementation" and other variations thereof appearing in various places throughout this application do not necessarily all refer to the same embodiment.

[0195] Additionally, the present application may refer to “determining” various information. Determining information may include, for example, one or more of estimating information, calculating information, predicting information, or retrieving information from a memory.

[0196] Furthermore, the present application may refer to "accessing" various information. Accessing information may include, for example, receiving information, retrieving information (e.g., from a memory), storing information, moving information, copying information, calculating information, determining information, predicting information, or estimating information.

[0197] Additionally, the present application may involve "receiving" various information. Receiving, like "accessing," is a broad term. Receiving information may include, for example, one or more of accessing information or retrieving information (e.g., from a memory). Additionally, during operations such as storing information, processing information, sending information, moving information, copying information, erasing information, calculating information, determining information, predicting information, or estimating information, "receiving" is often involved in some way.

[0198] It should be understood that the use of any of " / ", "and / or", and "at least one of" below, for example, in the case of "A / B", "A and / or B", and "at least one of A and B", is intended to cover selecting only the first listed option (A), or only the second listed option (B), or both options (A and B). As a further example, in the case of "A, B, and / or C" and "at least one of A, B, and C", such phrases are intended to cover selecting only the first listed option (A), or only the second listed option (B), or only the third listed option (C), or only the first and second listed options (A and B), or only the first and third listed options (A and C), or only the second and third listed options (B and C), or all three options (A and B and C). This may be extended to as many items as listed, as would be apparent to one of ordinary skill in this and related arts.

[0199] In addition, as used herein, the word "signaling" especially refers to indicating something to a corresponding decoder. For example, in some embodiments, the encoder signals a specific one of the scanning modes. In this way, in an embodiment, the same parameters are used on both the encoder side and the decoder side. Therefore, for example, the encoder can send (explicit signaling) specific parameters to the decoder so that the decoder can use the same specific parameters. On the contrary, if the decoder already has specific parameters and other parameters, signaling can be used without sending (implicit signaling) to simply allow the decoder to know and select specific parameters. By avoiding sending any actual function, bit saving is achieved in various embodiments. It should be understood that signaling can be done in a variety of ways. For example, in various embodiments, one or more syntax elements, flags, etc. are used to signal information to the corresponding decoder. Although the verb form of the word "signaling" is mentioned above, the word "signal" can also be used as a noun in this article.

[0200] It will be apparent to one of ordinary skill in the art that implementations may generate various signals formatted to carry information that may be stored or transmitted, for example. Information may include, for example, instructions for executing a method or data generated by one of the described implementations. For example, a signal may be formatted to carry a bitstream of the described embodiments. Such a signal may be formatted as, for example, an electromagnetic wave (e.g., using a radio frequency portion of a spectrum) or a baseband signal. Formatting may include, for example, encoding a data stream and modulating a carrier with the encoded data stream. The information carried by the signal may be, for example, analog or digital information. As is well known, a signal may be sent via a variety of different wired or wireless links. The signal may be stored on a processor readable medium.

[0201] We have described a number of embodiments. Features of these embodiments may be provided individually or in any combination. In addition, embodiments may include, individually or in any combination, one or more of the following features, devices, or aspects across various claim categories and types:

[0202] Determining a scan mode for a block based on one or more of block shape, prediction tool, and transform type;

[0203] • A bitstream or signal comprising one or more of the described syntax elements or variants thereof.

[0204] • Insertion of signalling syntax elements that enable the decoder to reprogram the scan order in a manner corresponding to that used by the encoder.

[0205] • Creating and / or sending and / or receiving and / or decoding a bitstream or signal comprising one or more of said syntax elements or variants thereof.

[0206] A television, set-top box, mobile phone, tablet or other electronic device performing adaptation of the scan order according to any of the described embodiments.

[0207] A television, set-top box, mobile phone, tablet or other electronic device that performs the adaptation of the scan order according to any of the described embodiments, and that displays (e.g., using a display, screen, or other type of display) the resulting image.

[0208] • A television, set-top box, mobile phone, tablet or other electronic device that tunes (eg, using a tuner) a channel to receive a signal comprising an encoded image and performs adaptation of the scan order according to any of the described embodiments.

[0209] • A television, set-top box, mobile phone, tablet or other electronic device that receives over the air (eg using an antenna) a signal comprising an encoded image and performs adaptation of the scan order according to any of the described embodiments.

[0210] According to a general aspect of at least one embodiment, a decoding method is proposed, comprising:

[0211] Transform coefficients of a block scanned according to a scanning pattern are entropy decoded, wherein the scanning pattern is determined in response to a shape of the block.

[0212] According to a general aspect of at least one embodiment, a coding method is proposed, comprising:

[0213] - obtaining transform coefficients of the block; and

[0214] - scanning the coefficients according to a scanning pattern for entropy coding;

[0215] Wherein a scanning pattern is determined in response to the shape of the block.

[0216] According to a general aspect of at least one embodiment, a decoding apparatus is provided, which includes one or more processors configured to implement:

[0217] Transform coefficients of a block scanned according to a scanning pattern are entropy decoded, wherein the scanning pattern is determined in response to a shape of the block.

[0218] According to a general aspect of at least one embodiment, a coding apparatus is provided, which includes one or more processors configured to implement:

[0219] - obtaining transform coefficients of the block; and

[0220] - scanning the coefficients according to a scanning pattern for entropy coding;

[0221] Wherein a scanning pattern is determined in response to the shape of the block.

[0222] According to another general aspect of at least one embodiment, a bitstream is formatted to include a signal generated according to the encoding method described above.

[0223] One or more of the present embodiments also provide a computer-readable storage medium on which instructions are stored, and the instructions are used to encode or decode video data according to at least part of any of the methods described above. One or more embodiments also provide a computer-readable storage medium on which a bit stream generated according to the encoding method described above is stored. One or more embodiments also provide methods and devices for sending and receiving a bit stream generated according to the encoding method described above. One or more embodiments also provide a computer program product including instructions for performing at least part of any of the methods described above.

[0224] In one embodiment, the scanning mode is determined in response to whether the block is a square block, a horizontal rectangular block, or a vertical rectangular block.

[0225] In one embodiment, the coefficients are scanned according to a horizontal scanning pattern in case the blocks are rectangular vertical blocks, and the coefficients are scanned according to a vertical scanning pattern in case the blocks are rectangular horizontal blocks.

[0226] In one embodiment, in a case where a ratio between a maximum dimension of the block and a minimum dimension of the block is equal to or greater than a value, coefficients are scanned according to a vertical scanning mode when the block is a rectangular horizontal block, and coefficients are scanned according to a horizontal scanning mode when the block is a rectangular vertical block.

[0227] In one embodiment, in case the block is a rectangular block, the coefficients are scanned according to a scan pattern selected by an exhaustive rate-distortion search among a set of patterns, and wherein the selected scan pattern is encoded in the bitstream.

[0228] In one embodiment, in the case where the ratio between the longest dimension of the block and the smallest dimension of the block is equal to or greater than another value and the block is a rectangular block, the coefficients are scanned according to a scanning pattern selected by an exhaustive rate-distortion search in a set of patterns, and wherein the selected scanning pattern is encoded in the bitstream.

[0229] In one embodiment, where the block is a rectangular block, the coefficients are scanned according to a scan pattern defined such that the scan step size along the longest dimension is larger than the scan step size along the shortest dimension.

[0230] In one embodiment, in case the block is a multiple-hypothesis intra-inter predicted block, vertical scanning is applied if the intra prediction mode is horizontal and horizontal scanning is applied if the intra prediction mode is vertical.

[0231] In one embodiment, in case the block is a multi-hypothesis prediction block, the coefficients are scanned according to a scanning pattern selected by an exhaustive rate-distortion search among a set of patterns, and wherein the selected scanning pattern is encoded in the bitstream.

[0232] In one embodiment, in case the block is a triangle prediction block, the coefficients are scanned according to a scan pattern selected by an exhaustive rate-distortion search among a set of patterns, and wherein the selected scan pattern is encoded in the bitstream.

[0233] In one embodiment, obtaining transform coefficients of the block comprises transforming a residual of the block using at least one transform to obtain transform coefficients and determining a scanning pattern in response to a type of the transform, wherein the type of transform is defined by whether its low frequency is constant, increasing or decreasing.

[0234] In one embodiment, in case a transform with a constant low frequency is applied both horizontally and vertically, the coefficients are scanned according to a diagonal scanning pattern, and wherein in case one of the horizontal and vertical transforms is not a transform with a constant low frequency, the coefficients are scanned according to a scanning pattern selected by an exhaustive rate-distortion search in a set of patterns, and wherein the selected scanning pattern is encoded in the bitstream.

[0235] In one embodiment, where the vertical and horizontal transforms are DCT-2, the coefficients are scanned according to a diagonal scanning pattern, and where the vertical and horizontal transforms are selected in DST-7 and DCT-8 transforms, the coefficients are scanned according to a scanning pattern selected by an exhaustive rate-distortion search in a set of patterns, and where the selected scanning pattern is encoded in the bitstream.

Claims

1. A coding method, comprising: - Applying the transform function to the residual block of the current block to obtain transform coefficients; as well as - scanning the coefficients according to a scanning pattern for entropy coding; Wherein, in response to a first type of basis function of the transform function at the lowest frequency, the transform coefficients are scanned according to a first scanning order, otherwise, the transform coefficients are scanned according to a second scanning order selected by an exhaustive rate-distortion search from a plurality of scanning orders including the first scanning order and at least one additional scanning order different from the first scanning order.

2. The method of claim 1, wherein the selected second scanning order is encoded in video data representing at least the current block.

3. A decoding method, comprising: - entropy decoding of the transform coefficients of the current block ordered according to the scanning pattern; as well as - obtaining a residual block of the current block from the transform coefficients and the scanning pattern applying the transform function; Wherein, in response to a first type of basis function of the transform function at the lowest frequency, the transform coefficients are scanned according to a first scanning order, otherwise, the transform coefficients are scanned according to a second scanning order among a plurality of scanning orders including the first scanning order and at least one additional scanning order different from the first scanning order.

4. The method of claim 3, wherein the second scanning order is decoded from video data representing at least the current block.

5. A coding apparatus comprising one or more processors, wherein the one or more processors are configured to perform: - applying a transform function to the residual block of the current block to obtain transform coefficients; and - scanning the coefficients according to a scanning pattern for entropy coding; Wherein, in response to a first type of basis function of the transform function at the lowest frequency, the transform coefficients are scanned according to a first scanning order, otherwise, the transform coefficients are scanned according to a second scanning order selected by an exhaustive rate-distortion search from a plurality of scanning orders including the first scanning order and at least one additional scanning order different from the first scanning order.

6. The encoding apparatus of claim 5, wherein the selected second scanning order is encoded in video data representing at least the current block.

7. A decoding device, comprising one or more processors, wherein the one or more processors are configured to perform: - entropy decoding the transform coefficients of the current block ordered according to the scanning pattern; and - obtaining a residual block of the current block from the transform coefficients and the scanning pattern applying the transform function; Wherein, in response to a first type of basis function of the transform function at the lowest frequency, the transform coefficients are scanned according to a first scanning order, otherwise, the transform coefficients are scanned according to a second scanning order among a plurality of scanning orders including the first scanning order and at least one additional scanning order different from the first scanning order.

8. The decoding apparatus of claim 7, wherein the second scanning order is decoded from video data representing at least the current block.

9. A computer program product comprising instructions, wherein when the instructions are executed by a processor, the processor is caused to implement the method according to claim 1.

10. A computer program product comprising instructions, wherein when the instructions are executed by a processor, the processor is caused to implement the method according to claim 3.

11. A computer-readable storage medium having instructions stored thereon, wherein when a processor executes the instructions, the processor is caused to implement the method according to claim 1.

12. A computer-readable storage medium having instructions stored thereon, wherein when a processor executes the instructions, the processor is caused to implement the method according to claim 3.

Images