Coding and decoding method and apparatus, method for transmitting bit stream, program product and medium

Abstract

Disclosed are an encoding and decoding method, an apparatus, a method of transmitting a bitstream, a program product, and a medium. The decoding method includes decoding, from video data, a first motion vector signaled for a first block of a picture, the first block being predicted in time domain from a first region of a first reference picture motion compensated using the second motion vector, refining the first motion vector to generate a second motion vector; and decoding a second block from the video data, the second block being predicted in time domain from a second region of a second reference picture motion compensated using a third motion vector determined from the predicted motion vector, whether the predicted motion vector is a first motion vector or a second motion vector depends on whether the first block and the second block belong to the same set of consecutive samples.

Description

[0001] This application is a divisional application of the invention patent application with application date of May 17, 2019, application number 201980035741.4, and invention name “Data Dependence in Decoding / Decoding”. Technical Field

[0002] The technical field of one or more embodiments generally relates to video compression. The present embodiment generally relates to methods and devices for picture encoding and decoding. Background Art

[0003] In order to achieve high compression efficiency, image and video coding schemes usually use prediction and transformation to make full use of spatial and temporal redundancy in video content. In general, intra-frame or inter-frame prediction is used to exploit intra-frame or inter-frame correlation, and then the difference between the original image block and the predicted image block (often expressed as prediction error or prediction residual) is transformed, quantized, and entropy decoded. During encoding, it is usually possible to partition / split the original image block into sub-blocks using quadtree partitioning. In order to reconstruct the video, the compressed data is decoded through the inverse process corresponding to prediction, transformation, quantization, and entropy decoding. Summary of the invention

[0004] A decoding method is disclosed, comprising:

[0005] - decoding a flag indicating whether to decode a current block of a picture with tools in a toolset including tools that use reconstructed samples of neighboring blocks decoded before the current block or tools that use blocks of the picture that are larger than the hardware unit only if the current block has a parent hardware unit; and

[0006] - Decode the current block in response to the flag.

[0007] A decoding device is disclosed, comprising at least one processor, wherein the processor is configured to execute:

[0008] - decoding a flag indicating whether to decode a current block of a picture with tools in a toolset including tools that use reconstructed samples of neighboring blocks decoded before the current block or tools that use blocks of the picture that are larger than the hardware unit only if the current block has a parent hardware unit; and

[0009] - Decode the current block in response to the flag.

[0010] A coding method is disclosed, comprising:

[0011] - encoding a flag indicating whether to encode a current block of a picture with tools from a toolset including tools that use reconstructed samples of neighboring blocks that were coded before the current block or tools that use blocks of the picture that are larger than the hardware unit only if the current block has a parent hardware unit; and

[0012] - Encode the current block in response to the flag.

[0013] A coding device comprises at least one processor, the processor being configured to execute:

[0014] - encoding a flag indicating whether to encode a current block of a picture with tools from a toolset including tools that use reconstructed samples of neighboring blocks that were coded before the current block or tools that use blocks of the picture that are larger than the hardware unit only if the current block has a parent hardware unit; and

[0015] - Encode the current block in response to the flag.

[0016] A signal is disclosed, comprising:

[0017] a flag indicating whether to encode a current block of a picture with tools in a toolset including tools that use reconstructed samples of neighboring blocks that were coded before the current block or tools that use blocks of the picture that are larger than a hardware unit only if the current block has a parent hardware unit; and

[0018] - Data representing the current block, which is decoded in response to the flag.

[0019] A device is disclosed, comprising:

[0020] A decoding device according to any disclosed embodiment; and

[0021] At least one of: (i) an antenna configured to receive a signal over the air, the signal comprising video having an image, (ii) a bandwidth limiter configured to limit the received signal to a frequency band comprising the video having an image, or (iii) a display configured to display an output.

[0022] A computer program product is disclosed, which includes instructions. When the program is executed by one or more processors, the one or more processors perform the decoding or encoding method of any disclosed embodiment. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] FIG1 shows a coding tree unit (CTU) divided into coding units (CUs) in HEVC;

[0024] FIG2 shows a CTU that is spatially partitioned into one or more prediction units (PUs) and transform units (TUs) in HEVC;

[0025] FIG3 illustrates the process of finding a match between two blocks along the motion trajectory of the current CU in two different reference pictures;

[0026] FIG4 illustrates the process of finding a match between a template in the current picture and a corresponding template;

[0027] FIG5 illustrates a process of deriving illumination compensation parameters by comparing an L-shape located in a current picture with a corresponding L-shape in a reference picture;

[0028] FIG6 depicts two decoding units and corresponding hardware units in the spatial domain according to a general aspect of at least one embodiment;

[0029] FIG. 7 illustrates the granularity of processing at each stage of the overall decoding process according to a general aspect of at least one embodiment;

[0030] FIG8 depicts an example of a processing pipeline for decoding inter-frames according to a general aspect of at least one embodiment;

[0031] FIG9 illustrates a block diagram of an embodiment of a video encoder;

[0032] FIG10 illustrates a block diagram of an embodiment of a video decoder;

[0033] FIG11 depicts an example of a split tree and associated split CTUs;

[0034] FIG. 12 illustrates a process for determining a parent hardware unit (HU) according to a general aspect of at least one embodiment;

[0035] FIG. 13 illustrates a process for deciding whether to activate a tool in Tdep according to a general aspect of at least one embodiment;

[0036] FIG. 14 depicts in gray a CU for which the Tdep tool may be used in accordance with general aspects of at least one embodiment;

[0037] FIG15 illustrates a process for parsing a flag according to a general aspect of at least one embodiment;

[0038] FIG16 illustrates motion vector predictor derivation for merging candidates;

[0039] FIG. 17 illustrates a process for motion vector update (on the left) and motion predictor derivation (on the right) for a CU in accordance with a general aspect of at least one embodiment;

[0040] 18 illustrates a process for motion vector update for a CU (left) and a process for motion predictor derivation depending on a HU (right) according to a general aspect of at least one embodiment;

[0041] FIG. 19 illustrates a process of splitting a CU into 2 HUs according to a general aspect of at least one embodiment;

[0042] FIG. 20 depicts an example of splitting a CU according to a general aspect of at least one embodiment;

[0043] FIG21 illustrates a block diagram of a system in which aspects of the present embodiment may be implemented;

[0044] FIG. 22 depicts a flowchart of a decoding method according to an embodiment; and

[0045] FIG23 depicts a flowchart of an encoding method according to an embodiment. DETAILED DESCRIPTION

[0046] In the HEVC video compression standard, motion compensated temporal prediction is used to exploit the redundancy between consecutive pictures of a video.

[0047] To do this, a motion vector is associated with each prediction unit (PU), which we will now introduce. Each CTU (coding tree unit) is represented by a coding tree in the compressed domain. This is a quadtree partitioning of the CTU, where each leaf is called a coding unit (CU), as shown in Figure 1.

[0048] Each CU is then given some intra or inter prediction parameters (prediction information). To do this, it is spatially split into one or more prediction units (PUs), each of which is assigned some prediction information. Intra or inter coding modes are assigned at the CU level, as shown in Figure 2.

[0049] A motion vector is assigned to each PU in HEVC. This motion vector is used for temporal prediction of motion compensation of the considered PU. Therefore, in HEVC, the motion model of a linked predicted block and its reference blocks consists only of translation.

[0050] In the Joint Exploration Model (JEM) developed by the JVET (Joint Video Exploration Team) group, some motion models are supported to improve temporal prediction. To do this, the PU can be partitioned into sub-PUs in the spatial domain, and a dedicated motion vector can be assigned to each sub-PU using a model.

[0051] In other versions of JEM, CU is no longer divided into PU or TU, and some motion data is directly allocated to each CU. In this new codec design, CU can be divided into sub-CUs, and motion vectors can be calculated for each sub-CU.

[0052] For inter-frame motion compensation, a set of new tools leveraging decoder-side parameter estimation were developed: FRUC (frame rate up-conversion) merge, FRUC bilateral, and IC (illumination compensation).

[0053] The FRUC tool is described below. FRUC allows the motion information of a CU to be derived at the decoder side without signaling. This mode is signaled at the CU level using the FRUC flag and an additional FRUC mode flag to indicate which matching cost function (bilateral or template) to use to derive the motion information of the CU.

[0054] On the encoder side, the decision of whether to use FRUC merge mode for a CU is based on RD cost selection. Both matching modes (bilateral and template) will be checked for the CU. The decoding mode that results in the minimum RD cost can be further compared with other decoding modes. If the FRUC mode is the most efficient, the FRUC flag is set to "true" for the CU and the relevant matching mode is used.

[0055] The motion derivation process in FRUC merge mode has two steps. First, a CU-level motion search is performed, and then a sub-CU-level motion refinement is performed. At the CU level, an initial motion vector is derived from a list of MV candidates for the entire CU based on bilateral or template matching. The candidate that results in the minimum matching cost is selected as the starting point for further CU-level refinement. Then, a local search based on bilateral or template matching around the starting point is performed, and the MV that results in the minimum matching cost is used as the MV for the entire CU. Subsequently, the motion information is further refined at the sub-CU level with the derived CU motion vector as the starting point.

[0056] As shown in Figure 3, the motion information of the current CU is derived using a bilateral matching cost function by finding the best match between two blocks along the motion trajectory of the current CU in two different reference pictures. Under the assumption of continuous motion trajectory, the motion vectors MV0 and MV1 pointing to the two reference blocks should be proportional to the temporal distance between the current picture and the two reference pictures (TD0 and TD1).

[0057] As shown in FIG. 4 , the template matching cost function is used to derive the motion information of the current CU by finding the best match between the template in the current picture (the top and / or left neighboring blocks of the current CU) and the corresponding template in the reference picture (same size as the template).

[0058] Note that this FRUC mode using template matching cost function also applies to AMVP (Advanced Motion Vector Prediction) mode. In this case, there are two candidates for AMVP. Use the FRUC tool with template matching to derive a new candidate. If this FRUC candidate is different from the first existing AMVP candidate, then insert it to the very beginning of the AMVP candidate list, and then set the list size to two (meaning remove the second existing AMVP candidate). When applied to AMVP mode, only CU level search is applied.

[0059] In inter-mode, IC (Illumination Compensation) allows to correct the block prediction samples obtained via motion compensation (MC) by taking into account spatial or temporal local illumination variations. The IC parameters are estimated by comparing a set S of reconstructed neighboring samples (L-shape-current, L-shape-cur) with neighboring samples (L-shape-reference i, L-shape-ref-i) of a reference i block (i = 0 or 1), as depicted in Figure 5.

[0060] The IC parameters minimize the difference (least squares) between samples in L-shape-current (L-shape-cur) and samples in L-shape-ref-i corrected with the IC parameters. Typically, the IC model is linear: IC(x) = a.x + b, where x is the value of the sample to be compensated.

[0061] The parameters a and b are derived by least squares minimization of the L-shape at the encoder (and decoder):

[0062]

[0063] Finally, a i Transformed into integer weights (a i ) and shift (sh i ), and then use IC to correct the MC block:

[0064]

[0065] A problem addressed by at least one embodiment is how to constrain data dependencies beyond a given block size.

[0066] Figure 6 depicts an example of possible processing at a hardware decoder. In this example, we assume that the maximum size of a hardware unit (HU) that the decoder can process (usually set by the maximum transform of the codec) is 64 pixels. To increase parallelism, the decoder splits CUs larger than 64 pixels into blocks of maximum size 64, which are then sent to the computation modules (usually motion compensation and residual computation).

[0067] Figure 7 depicts an example of the granularity of each stage of the entire decoding process (implying possible parallelism). First, the bitstream is parsed and a set of CUs is generated for decoding. Each CU is decoded, giving a final value for calculating the reconstructed samples. Note that there can be some causal relationships between CU decodings (usually the value of a motion vector can be predicted from the value of an already decoded (but not reconstructed) CU). Then, the CU is split into several HUs as needed (depending on the size). At this point, all calculations inside a HU should be performed independently of other HUs. Note that for intra CUs, there are still causal relationships between reconstructed CUs because the intra prediction process needs to access neighboring samples during the prediction stage.

[0068] A problem solved by at least one embodiment is how to mitigate data dependencies created by tools such as FRUC. Figure 8 depicts an example of a processing pipeline for decoding inter-frames. One problem caused by tools such as FRUC or LIC (local illumination compensation) when considering such a pipeline is that it introduces data dependencies on reconstructed samples of previous CUs, breaking the possible parallelism of HU processing.

[0069] At least one of the embodiments described herein is a method that avoids this dependency and allows for highly parallel pipelining at the decoder.

[0070] FRUC, DMVR (Decoder Motion Vector Refinement), and LIC are new modes in JEM, so pipeline stalls are a relatively new problem.

[0071] The basic idea of ​​at least one embodiment is to break dependencies at the HU level. At least one of the proposed embodiments involves a specification modification of the codec: the encoding and decoding processes are completely symmetrical. The affected codec modules of one or more embodiments are modules 170 and possibly 160 of FIG. 9 , and modules 275 and possibly 260 of FIG. 10 .

[0072] We refer to Tdep as a set of tools that use reconstructed samples from a previous CU (i.e., a CU coded before the current CU) or a CU with a HU greater than HU at the decoder side, typically:

[0073] - FRUC in template mode can use reconstructed samples outside the current CU;

[0074] -LIC can use reconstructed samples outside the current CU;

[0075] -DMVR can use CUs larger than HU size;

[0076] -BIO (Bidirectional Optical Flow) can use CUs larger than HU size.

[0077] FIG. 22 depicts a flow chart of a decoding method according to an embodiment. The method starts at S10. A flag is decoded (S12). The flag indicates whether a tool in a tool set (Tdep) is used to decode a current block of a picture. The flag is decoded only if the current block has a parent hardware unit. The tool set includes tools that use reconstructed samples of neighboring blocks decoded before the current block or tools that use blocks of the picture that are larger than the hardware unit. In response to the flag, the current block is decoded (S14). The method ends at S16.

[0078] FIG23 depicts a flow chart of an encoding method according to an embodiment. The method starts at S20. A flag is encoded (S22). The flag indicates whether a current block of a picture is encoded with a tool in a tool set (Tdep). The flag is encoded only if the current block has a parent hardware unit. The tool set includes a tool that uses a reconstructed sample of a neighboring block that was encoded before the current block or a tool that uses a block of the picture that is larger than a hardware unit. In response to the flag, the current block is encoded (S24). The method ends at S26.

[0079] HU limit

[0080] To enable the use of a standalone HU-based process, add new constraints to the use of the tool in the Tdep tool:

[0081] 1. The current CU has a parent HU.

[0082] 2. The left and top neighboring CUs are inside the parent HU of the current CU.

[0083] Figure 11 depicts an example of a split tree and associated split CTUs. The parent HU of a CU is defined as the first CU in the split tree whose dimensions (width and height) are less than or equal to S. As an example, S=64. For example, in Figure 11, C and D are parent HUs.

[0084] FIG12 depicts a process for determining the parent HU of a block (e.g., CU). The process starts at step S120. At step S121, it is checked whether h or w is greater than S, where h and w are the height and width of the current block, respectively. In the case of h>S or w>S, the process continues at step S122, otherwise the process continues at step S125.

[0085] At S122 , the current block has no parent HU.

[0086] At step S123, it is checked whether the current block is further split into child blocks. If not, the process ends (S129). If yes, the current block is further split into child blocks (S124), and the process continues for the child blocks at S121.

[0087] If yes, then the process continues at S124 , otherwise the process ends ( S129 ).

[0088] At S125 , HU is set equal to the current block (such as, for example, block C of its sub-block).

[0089] At S126 , the parent HU of the current block is HU.

[0090] At S127, it is checked whether the current block is further split. If not, the process ends (S129). If yes, the current block is further split (S128), and then the process continues for the sub-blocks at S126.

[0091] The second criterion checks whether the neighboring CUs of the current CU share the same parent HU as the current CU. Figure 13 depicts a flow chart of the process of deciding whether to use a tool in the Tdep tool.

[0092] The process starts from S130 .

[0093] At step S132, it is checked whether the current CU has a parent HU, such as depicted on FIG. 12. If not, the process ends at S137 (without using the Tdep tool). Otherwise, the process continues at S133 to check whether the neighboring CUs (e.g., top and left) share the same parent HU as the current CU. If not, the process ends. Otherwise, the process continues at S134.

[0094] At S134, a flag indicating whether to use the Tdep tool is decoded. If the flag is true (S135), the current CU is decoded using the corresponding tool (S136). If not, the process ends (S137), that is, the corresponding Tdep tool is not used.

[0095] FIG14 shows some examples of CUs (in gray) for which the Tdep tool can be used. The HU size is set to 64. In one embodiment, the parsing of the flags of the tools in the Tdep tool is constrained by the process described in FIG13. In one embodiment, the decoding of the flags of the tools in the Tdep tool is constrained only by the presence of a parent HU, as shown in FIG15.

[0096] The process starts (S130). Check if the current CU has a parent HU (S132). In case the flag is true (S135), the current CU is decoded using the corresponding tool (S136). If not, the process ends (S137), i.e., the corresponding Tdep tool is not used.

[0097] Motion Vector Propagation HU

[0098] Since the reconstruction of CUs within a HU is independent, the prediction of values ​​between HUs can also be restricted.

[0099] Here is an example of the values ​​generated by the tools in the Tdep set:

[0100] -FRUC: The final value of the motion vector of the CU is given after the final motion refinement.

[0101] -DMVR: gives the final value of the motion vector of the CU after the final motion refinement.

[0102] The generated value can then be used as a prediction for another CU (see FIG16 for an example of the predicted value). At S160, it is checked whether the given motion predictor is in the list of motion vector candidates. If not, it is added to the list (S162).

[0103] For tools that generate new motion vectors for a CU, motion vector prediction can also be constrained within the HU. In this embodiment, the predictor used by the CU is always the value before motion refinement, i.e., for blocks with decoder-side motion vector derivation tools (FRUC, DMVR, etc.), the final motion vector is not updated. From S170-1 to S178-1, the CU refines the motion vector candidates for its own motion compensation according to the FRUC flag. However, it uses the unrefined motion vector of the neighboring CU (S170-2 to S178-2) as the predictor. Figure 17 shows the process of motion vector update for the current CU (left) and the process of motion predictor derivation (right).

[0104] The motion vector update process starts from S170-1. At S172, the MV candidate is decoded.

[0105] At S174, MV1 is set equal to the decoded MV candidate.

[0106] At S175 , it is checked whether FRUC is used (flag is true). If so, the motion vector MV1 is refined with FRUC to obtain MV2 ( S176 ). The current CU is motion compensated with MV2 ( S177 ).

[0107] The process ends (S178-1).

[0108] The motion predictor derivation process starts at S170-2. A neighboring CU denoted CUn is considered for prediction (S171). The motion vector predictor used in S172 is set equal to MV1, rather than equal to the refined vector of the neighboring CU.

[0109] Only for the propagation of CUs within the same HU

[0110] 18 illustrates a process of motion vector update for a CU (left) and a process of motion predictor derivation according to a HU (right) according to a general aspect of at least one embodiment. From S180-1 to S190-1, the CU refines motion vector candidates according to the FRUC flag.

[0111] However, if the neighboring CU is outside the HU of the current CU ( S183 ), the unrefined motion vector of the neighboring CU is used as the predictor ( S185 ); whereas if they are in the same HU, the refined motion vector is used ( S187 ).

[0112] The process of motion vector updating starts from S180-1. At S182, MV candidates are decoded. At S184, MV1 and MV2 are set as decoded motion vector candidates.

[0113] At S186, it is checked whether FRUC is used (flag is true). If so, the motion vector MV1 is refined with FRUC to obtain MV2 (S188). The current CU is motion compensated with MV2 (S189).

[0114] The process ends (S190-1).

[0115] The motion predictor derivation process starts at S180-2. A neighboring CU denoted as CUn is considered for prediction (S181). It is checked whether CUn is in the same HU as the current CU. If so, the refined motion vector is used for prediction (S187). If not, the unrefined motion vector is used (S185). The method ends at S190-2.

[0116] In this embodiment, the motion vector predictor used depends on the parent HU of the CU.

[0117] The associated syntax for constraints on the size and / or value propagation of the HU, such as one or more flags, selections from lists, other indicators (e.g., restrictions on FRUC or IC) may be signaled at one or more of the slice, PPS (picture parameter set) or SPS (sequence parameter set) levels. Other levels, high-level syntax or other approaches are used in other embodiments. The associated syntax for this signaling includes, for example, one or more flags, selections from lists, other indicators.

[0118] Multiple HUs of each CU are independently calculated

[0119] As depicted in Figure 6, a CU may be split into several HUs for computation. In this case, a single syntax element is signaled at the CU level, but processed per HU. Note that the computation may result in different results within a CU.

[0120] Examples of tools affected by this treatment are:

[0121] -DMVR;

[0122] - LIC in bilateral mode.

[0123] In this case, processing (e.g., bilateral motion vector refinement or bidirectional illumination compensation parameters for DMVR) is performed independently for each HU.

[0124] FIG19 shows an example where a CU is split into 2 HUs: the unique flag of the CU is parsed and decoded, and then if the CU size is larger than the HU size, each sub-HU runs an independent DMVR process.

[0125] From S191 to S201, if the CU size is larger than the HU size, the CU using the DMVR mode is split into several HUs. Then, the generated HUs use independent DMVR processes (S199 and S200).

[0126] The process starts at S191. A flag is decoded (S192) indicating whether DMVR is applicable. If DMVR is applicable, then check (S194) whether the CU size is larger than the HU size. If not, then DMVR is applied to the CU (S198). If yes, then the CU is split (S196) into smaller hardware units (also called sub-HUs) HU0 and HU1. The CU can be split into more than two sub-HUs, as depicted in Figure 20. Each sub-HU runs an independent DMVR process (S199 and S200). The process ends at S201.

[0127] HU Split

[0128] Splitting a CU into sub-HUs can be done in several ways:

[0129] - Split by maximum HU size (both horizontally and vertically), and if the last HU (horizontally and / or vertically) is outside the CU, resize it (Figure 20)

[0130] - Split the CU into equal number of HUs in horizontal and vertical directions given the constraint that the size of each HU should not be larger than the maximum size and should also be a multiple of 4 (minimum CU size)

[0131] Various embodiments include one or more of the following:

[0132] - Restriction of the use of tools in Tdep at HU level + additional constraints on causal boundaries of HUs.

[0133] - Adapt the grammar to these constraints

[0134] -Several variations of HU topology on the frame.

[0135] - Adjust signaling and HU handling of some tools (DMVR, LIC bidir)

[0136] Various aspects including tools, features, embodiments, models, methods, etc. are described herein. Many of these aspects are specifically described, and at least for the purpose of illustrating the various features, are often described in a manner that may sound restrictive. However, this is for clarity of description and does not limit the application or scope of those aspects. In fact, all of the different aspects can be combined and interchanged to provide additional aspects. Moreover, these aspects can also be combined and interchanged with aspects described in previous filings.

[0137] The aspects described and considered in this document can be implemented in many different forms. Figures 10, 11 and 21 below provide some embodiments, but other embodiments are expected, and the discussion of Figures 10, 11 and 21 does not limit the breadth of the implementation. At least one of these aspects generally relates to video encoding and decoding, and at least one other aspect generally relates to transmitting the generated or encoded bitstream. These and other aspects can be implemented as methods, devices, computer-readable storage media having instructions stored thereon for encoding or decoding video data according to any of the methods described, and / or computer-readable storage media having bitstreams generated according to any of the methods described stored thereon.

[0138] In this application, the terms "reconstructed" and "decoded" are used interchangeably, the terms "pixel" and "sample" are used interchangeably, and the terms "image", "picture" and "frame" are used interchangeably. Usually, but not necessarily, the term "reconstructed" is used on the encoder side, while "decoded" is used on the decoder side.

[0139] Various methods are described above, and each method includes one or more steps or actions for implementing the described method. Unless a specific order of steps or actions is required for the correct operation of the method, the order and / or use of specific steps and / or actions may be modified or combined.

[0140] The various methods and other aspects described herein may be used to modify modules of the JVET or HEVC encoder 100 and decoder 200, such as, for example, motion compensation and possibly intra prediction modules (170, 275, 160, 260), as shown in FIGS. 10 and 11. Moreover, the current aspects are not limited to JVET or HEVC, and may be applied, for example, to other standards and recommendations, whether pre-existing or developed in the future, as well as extensions of any such standards and recommendations (including JVET and HEVC). Unless otherwise indicated or technically prohibited, the various aspects described in this document may be used alone or in combination.

[0141] Various numerical values ​​may be shown in this document. The specific values ​​are for example purposes only, and the described aspects are not limited to these specific values.

[0142] 10 illustrates an exemplary encoder 100. Variations of this encoder 100 are contemplated, but for the sake of clarity, the encoder 100 is described below without describing all contemplated variations.

[0143] Before being encoded, the video sequence may be processed by pre-encoding (101), for example, applying a color transform to an input color picture (e.g., conversion from RGB 4:4:4 to YCbCr 4:2:0), or performing a remapping of input picture components in order to obtain a signal distribution that is more resilient to compression (e.g., using histogram equalization of one of the color components). Metadata may be associated with the pre-processing and appended to the bitstream.

[0144] In an exemplary encoder 100, a picture is encoded by an encoder element as described below. The picture to be encoded is segmented (102) and processed in units such as CUs. Each unit is encoded using, for example, either intra or inter mode. When the unit is encoded in intra mode, it performs intra prediction (160). In inter mode, motion estimation (175) and compensation (170) are performed. The encoder decides (105) which of intra mode or inter mode to use for encoding the unit, and indicates the intra / inter decision by, for example, a prediction mode flag. For example, a prediction residual is calculated by subtracting (110) the predicted block from the original image block.

[0145] The prediction residual is then transformed (125) and quantized (130). The quantized transform coefficients, along with motion vectors and other syntax elements, are entropy coded (145) to output a bitstream. The encoder may skip the transform and apply quantization directly to the untransformed residual signal. The encoder may bypass both the transform and quantization, i.e., the residual is directly coded without applying the transform or quantization process.

[0146] The encoder decodes the encoded block to provide a reference for further prediction. The quantized transform coefficients are dequantized (140) and inversely transformed (150) to decode the prediction residual. The image block is reconstructed by combining (155) the decoded prediction residual and the predicted block. A loop filter (165) is applied to the reconstructed picture to perform, for example, deblocking / SAO (sample adaptive offset) filtering to reduce coding artifacts. The filtered image is stored in a reference picture buffer (180).

[0147] FIG11 illustrates a block diagram of an exemplary video decoder 200. In the exemplary decoder 200, a bitstream is decoded by decoder elements as described below. The video decoder 200 generally performs a decoding process corresponding to the encoding process (pass) as described in FIG10. The encoder 100 also generally performs video decoding as part of encoding the video data.

[0148] In particular, the input to the decoder includes a video bitstream, which may be generated by the video encoder 100. The bitstream is first entropy decoded (230) to obtain transform coefficients, motion vectors and other decoded information. Picture segmentation information indicates how the picture is segmented. Therefore, the decoder can divide (235) the picture according to the decoded picture segmentation information. The transform coefficients are dequantized (240) and inversely transformed (250) to decode the prediction residual. The image block is reconstructed by combining (255) the decoded prediction residual and the predicted block. The predicted block can be obtained (270) from an intra-frame prediction (260) or a motion compensated prediction (i.e., inter-frame prediction) (275). A loop filter (265) is applied to the reconstructed image. The filtered image is stored in a reference picture buffer (280).

[0149] The decoded picture may also undergo post-decoding processing (285), such as performing an inverse color transform (e.g., conversion from YCbCr 4:2:0 to RGB 4:4:4) or inverse remapping that is the inverse of the remapping process performed in the pre-encoding process (101). The post-decoding processing may use metadata derived in the pre-encoding process and signaled in the bitstream.

[0150] Figure 21 illustrates a block diagram of an exemplary system in which various aspects and exemplary embodiments are implemented. System 1000 can be implemented as the equipment including the various components described below, and is configured to perform one or more aspects described in this document. Examples of such equipment include, but are not limited to, personal computers, laptop computers, smart phones, tablet computers, digital multimedia set-top boxes, digital television receivers, personal video recording systems, connected household appliances, and servers. System 1000 can be coupled to other similar systems communicatively, and coupled to displays via communication channels, as shown in Figure 21 and as known to those skilled in the art, to implement the various aspects described in this document.

[0151] The system 1000 may include at least one processor 1010 configured to execute instructions loaded therein to implement various aspects described in this document. The processor 1010 may include embedded memory, input and output interfaces, and various other circuits known in the art. The system 1000 may include at least one memory 1020 (e.g., a volatile storage device, a non-volatile storage device). The system 1000 may include a storage device 1040, which may include a non-volatile memory, including but not limited to EEPROM, ROM, PROM, RAM, DRAM, SRAM, flash memory, a magnetic disk drive, and / or an optical disk drive. As non-limiting examples, the storage device 1040 may include an internal storage device, an attached storage device, and / or a network accessible storage device. The system 1000 may include an encoder / decoder module 1030 configured to process data to provide encoded video or decoded video, and the encoder / decoder module 1030 may include its own processor and memory.

[0152] The encoder / decoder module 1030 represents a module(s) that may be included in a device to perform encoding and / or decoding functions. As is known, a device may include one or both of the encoding and decoding modules. In addition, the encoder / decoder module 1030 may be implemented as a separate element of the system 1000, or may be incorporated into the processor 1010 as a combination of hardware and software, as known to those skilled in the art.

[0153] Program code to be loaded onto the processor 1010 to perform various aspects described in this document may be stored in the storage device 1040 and then loaded onto the memory 1020 for execution by the processor 1010. According to an exemplary embodiment, during execution of the processes described in this document, one or more of the processor(s) 1010, the memory 1020, the storage device 1040, and the encoder / decoder module 1030 may store one or more of various items including, but not limited to, input video, decoded video, bitstreams, equations, formulas, matrices, variables, operations, and operation logic.

[0154] The system 1000 may include a communication interface 1050 that enables communication with other devices via a communication channel 1060. The communication interface 1050 may include, but is not limited to, a transceiver configured to transmit and receive data from the communication channel 1060. The communication interface 1050 may include, but is not limited to, a modem or a network card, and the communication channel may be implemented in a wired and / or wireless medium. The various components of the system 1000 may all be connected or communicatively coupled together using various suitable connections, including, but not limited to, internal buses, wires, and printed circuit boards.

[0155] The exemplary embodiments may be implemented by computer software implemented by the processor 1010 or by hardware or by a combination of hardware and software. As a non-limiting example, the exemplary embodiments may be implemented by one or more integrated circuits. The memory 1020 may be of any type suitable for the technical environment and may be implemented using any appropriate data storage technology (such as optical memory devices, magnetic memory devices, semiconductor-based memory devices, fixed memory, and removable memory, as non-limiting examples). The processor 1010 may be of any type suitable for the technical environment and may include, as a non-limiting example, one or more of a microprocessor, a general-purpose computer, a special-purpose computer, and a processor based on a multi-core architecture.

[0156] The embodiments and aspects described herein may be implemented, for example, in a method or process, a device, a software program, a data stream, or a signal. Even if only discussed in the context of a single form of embodiment (e.g., discussed only as a method), the embodiments of the features discussed may also be implemented in other forms (e.g., a device or program). The device may be implemented, for example, in appropriate hardware, software, and firmware. The method may be implemented in, for example, a device (such as, for example, a processor), which generally refers to a processing device, including, for example, a computer, a microprocessor, an integrated circuit, or a programmable logic device. The processor also includes a communication device, such as, for example, a computer, a mobile phone, a portable / personal digital assistant ("PDA"), and other devices that facilitate information communication between end users.

[0157] References to "one embodiment" or "an embodiment" or "one implementation" or "an implementation" and other variations thereof refer to particular features, structures, characteristics, etc. described in conjunction with the embodiment being included in at least one embodiment. Thus, the appearance of the phrase "in one embodiment" or "in an embodiment" or "in an implementation" or "in an implementation" and any other variations in various places herein do not necessarily all refer to the same embodiment.

[0158] Additionally, this document may refer to "determining" various information. Determining information may include one or more of: for example, estimating information, calculating information, predicting information, or retrieving information from a memory.

[0159] Additionally, this document may refer to "accessing" various information. Accessing information may include one or more of: for example, receiving information, retrieving information (e.g., from a memory), storing information, processing information, transmitting information, moving information, copying information, erasing information, calculating information, determining information, predicting information, or estimating information.

[0160] Additionally, this document may refer to "receiving" various information. Like "accessing," receiving is a broad term. Receiving information may include one or more of: for example, accessing information or retrieving information (e.g., from storage). Additionally, in one way or another, during operations such as, for example: storing information, processing information, sending information, moving information, copying information, erasing information, calculating information, determining information, predicting information, or estimating information, "receiving" is generally involved.

[0161] As will be apparent to one of ordinary skill in the art, embodiments may generate various signals formatted to carry information that may be stored or transmitted, for example. The information may include, for example, instructions for executing a method or data generated by one of the described embodiments. For example, a signal may be formatted to carry a bit stream of the described embodiments. Such a signal may be formatted, for example, as an electromagnetic wave (e.g., using a radio frequency portion of a spectrum) or a baseband signal. Formatting may include, for example, encoding a data stream and modulating a carrier with the encoded data stream. The information carried by the signal may be, for example, analog or digital information. As is known, the signal may be transmitted over a variety of different wired or wireless links. The signal may be stored on a processor readable medium.

[0162] We have described a number of embodiments. These embodiments provide at least the following broad invention and claims across various claim categories and types, including all combinations:

[0163] Relax, reduce or otherwise modify data dependencies created by transcoding and / or decoding tools:

[0164] The tools include FRUC, DMVR, LIC, BIO

[0165] The data dependency is the dependency between the decoded block (current block) and the neighboring blocks (eg, CU).

[0166] Restrict the use of some tools at the HU level, including FRUC, DMVR, LIC, BIO

[0167] Some tools are used at the HU level based on neighboring block restrictions.

[0168] In the case where at least one neighboring block (eg, top and / or left neighboring blocks) shares the same parent HU as the current block, some of the tools are used.

[0169] The neighboring blocks are the top and left blocks of the current block.

[0170] Adjusted HU handling of some tools (e.g., DMVR and LIC bidir).

[0171] Limit the predicted value between HU.

[0172] A bitstream or signal comprising one or more of the described syntax elements or variations thereof.

[0173] Syntax elements are inserted in the signaling, which enable the decoder to process the bitstream in a way that corresponds to the way used by the encoder.

[0174] Creating and / or transmitting and / or receiving and / or decoding a bitstream or signal including one or more of the described syntax elements or variations thereof.

[0175] A TV, set-top box, mobile phone, tablet computer or other electronic device implementing any of the embodiments described.

[0176] A TV, set-top box, mobile phone, tablet computer or other electronic device that executes any of the described embodiments and displays (eg, using a monitor, screen or other type of display) the resulting images.

[0177] A TV, set-top box, mobile phone, tablet or other electronic device tunes (eg, using a tuner) to a channel to receive a signal including an encoded image and executes any of the embodiments described.

[0178] A TV, set-top box, mobile phone, tablet or other electronic device receives over the air (eg, using an antenna) a signal including the encoded image and performs any of the described embodiments.

[0179] Various other general and specific inventions and claims are also supported and contemplated throughout this disclosure.

[0180] A decoding method is disclosed, comprising:

[0181] - decoding a flag indicating whether to decode a current block of a picture with tools in a toolset including tools that use reconstructed samples of neighboring blocks decoded before the current block or tools that use blocks of the picture that are larger than the hardware unit only if the current block has a parent hardware unit; and

[0182] - Decode the current block in response to the flag.

[0183] A decoding device is disclosed, comprising at least one processor, wherein the processor is configured to execute:

[0184] - decoding a flag indicating whether to decode a current block of a picture with tools in a toolset including tools that use reconstructed samples of neighboring blocks decoded before the current block or tools that use blocks of the picture that are larger than the hardware unit only if the current block has a parent hardware unit; and

[0185] - Decode the current block in response to the flag.

[0186] A coding method is disclosed, comprising:

[0187] - encoding a flag indicating whether to encode a current block of a picture with tools from a toolset including tools that use reconstructed samples of neighboring blocks that were coded before the current block or tools that use blocks of the picture that are larger than the hardware unit only if the current block has a parent hardware unit; and

[0188] - Encode the current block in response to the flag.

[0189] A coding device comprises at least one processor, the processor being configured to execute:

[0190] - encoding a flag indicating whether to encode a current block of a picture with tools from a toolset including tools that use reconstructed samples of neighboring blocks that were coded before the current block or tools that use blocks of the picture that are larger than the hardware unit only if the current block has a parent hardware unit; and

[0191] - Encode the current block in response to the flag.

[0192] A signal is disclosed, comprising:

[0193] a flag indicating whether to encode a current block of a picture with tools in a toolset including tools that use reconstructed samples of neighboring blocks that were coded before the current block or tools that use blocks of the picture that are larger than a hardware unit only if the current block has a parent hardware unit; and

[0194] - Data representing the current block, which is decoded in response to the flag.

[0195] A device is disclosed, comprising:

[0196] A decoding device according to any disclosed embodiment; and

[0197] At least one of: (i) an antenna configured to receive a signal over the air, the signal comprising video having an image, (ii) a bandwidth limiter configured to limit the received signal to a frequency band comprising the video having an image, or (iii) a display configured to display an output.

[0198] A computer program product is disclosed, which includes instructions. When the program is executed by one or more processors, the one or more processors perform the decoding or encoding method of any disclosed embodiment.

[0199] In an embodiment, the flag is decoded (respectively encoded) only if the current block has a parent hardware unit and at least one neighboring block is located inside the same parent hardware unit as the current block.

[0200] In an embodiment, the toolset includes the following tools: frame rate up-conversion, local illumination compensation, decoder motion vector refinement, and bidirectional optical flow.

[0201] In an embodiment, the tool uses a template and only uses the portion of the template that is inside the same parent hardware unit as the current block.

[0202] In an embodiment, wherein the tool uses a motion vector predictor with refinement, motion vector refinement is added to the motion vector predictor if the current block and the neighboring block are located in the same hardware unit, and wherein the motion vector refinement is not added to the motion vector predictor if the current block and the neighboring block are located in different hardware units.

Claims

1. A decoding method, comprising: - decoding from the video data a first motion vector signaled for a first block of a picture, - refining the first motion vector to generate a second motion vector, the first block being temporally predicted from a first region of a first reference picture for motion compensation using the second motion vector; as well as - decoding a second block from the video data, the second block being predicted in the temporal domain from a second area of ​​a second reference picture for motion compensation using a third motion vector, the third motion vector being determined from a predicted motion vector, and whether the predicted motion vector is the first motion vector or the second motion vector depends on whether the first block and the second block belong to the same set of consecutive samples.

2. The decoding method of claim 1, wherein the refinement of the first motion vector for obtaining the second motion vector uses a template, using only a portion of the template that is located inside the same set of consecutive samples as the first block.

3. A decoding device, comprising at least one processor, wherein the processor is configured to execute: - decoding from the video data a first motion vector signaled for a first block of a picture, - refining the first motion vector to generate a second motion vector, the first block being temporally predicted from a first region of a first reference picture for motion compensation using the second motion vector; and - decoding a second block from the video data, the second block being predicted in the temporal domain from a second area of ​​a second reference picture for motion compensation using a third motion vector, the third motion vector being determined from a predicted motion vector, and whether the predicted motion vector is the first motion vector or the second motion vector depends on whether the first block and the second block belong to the same set of consecutive samples.

4. The decoding apparatus according to claim 3, wherein the refinement of the first motion vector for obtaining the second motion vector uses a template, using only a portion of the template that is located inside the same set of consecutive samples as the first block.

5. A coding method comprising: - signalling in the video data a first motion vector for a first block of a picture; - refining the first motion vector to generate a second motion vector, the first block being temporally predicted from a first region of a first reference picture for motion compensation using the second motion vector; as well as A second block is encoded in the video data, wherein the second block is predicted in the temporal domain from a second area of ​​a second reference picture using a third motion vector for motion compensation, wherein the third motion vector is determined from a predicted motion vector, and whether the predicted motion vector is the first motion vector or the second motion vector depends on whether the first block and the second block belong to the same set of consecutive samples.

6. The encoding method of claim 5, wherein the refinement of the first motion vector for obtaining the second motion vector uses a template, using only a portion of the template that is located inside the same set of consecutive samples as the first block.

7. A coding apparatus comprising at least one processor configured to execute: - signalling in the video data a first motion vector for a first block of a picture; - refining the first motion vector to generate a second motion vector, the first block being temporally predicted from a first region of a first reference picture for motion compensation using the second motion vector; and A second block is encoded in the video data, wherein the second block is predicted in the temporal domain from a second area of ​​a second reference picture using a third motion vector for motion compensation, wherein the third motion vector is determined from a predicted motion vector, and whether the predicted motion vector is the first motion vector or the second motion vector depends on whether the first block and the second block belong to the same set of consecutive samples.

8. The encoding apparatus of claim 7, wherein refinement of the first motion vector for obtaining the second motion vector uses a template, using only a portion of the template that is located inside the same set of consecutive samples as the first block.

9. A method for sending a bit stream, wherein the bit stream is generated by the encoding method according to any one of claims 5-6.

10. A method for transmitting a bit stream, wherein the bit stream is generated by the encoding device according to any one of claims 7 to 8.

11. A computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the decoding method according to any one of claims 1 to 2 is implemented.

12. A computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the encoding method according to any one of claims 5 to 6 is implemented.

13. A computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the decoding method according to any one of claims 1 to 2.

14. A computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the encoding method according to any one of claims 5 to 6.

Images