Data Dependencies in Encoding / Decoding

Abstract

A decoding method is disclosed. A flag is first decoded. The flag indicates whether a current block of a picture is to be decoded with a tool from a set of tools. The set of tools includes tools that use reconstructed samples of neighboring blocks decoded before the current block or that use blocks of the picture that are larger than the hardware unit. The flag is decoded only if the current block has a parent hardware unit. The current block is then decoded in response to the flag. [Selection diagram] Figure 22

Description

[Technical field]

[0001] The technical field of one or more implementations generally relates to video compression. The present embodiments generally relate to methods and devices for picture coding and decoding. [Background technology]

[0002] To achieve high compression efficiency, image and video coding schemes usually exploit spatial and temporal redundancy in the video content using prediction and transformation. In general, intra- or inter-prediction is used to exploit correlation within or between frames, and then the difference between the original and predicted image blocks, often denoted as prediction error or prediction residual, is transformed, quantized, and entropy coded. During coding, the original image block is usually partitioned / divided into sub-blocks, possibly using quadtree partitioning. To reconstruct the video, the compressed data is decoded by an inverse process corresponding to prediction, transformation, quantization, and entropy coding. Summary of the Invention

[0003] 1. A method of decoding, comprising: - decoding a flag indicating whether the current block of the picture is decoded with a tool of a set of tools only if the current block has a parent hardware unit, the set of tools including a tool that uses reconstructed samples of a neighboring block decoded before the current block or that uses a block of the picture that is larger than the hardware unit; - decoding the current block in response to the flag.

[0004] A decoding device comprising at least one processor, the at least one processor comprising: - decoding a flag indicating whether the current block of the picture is decoded with a tool of a set of tools only if the current block has a parent hardware unit, the set of tools including a tool that uses reconstructed samples of a neighboring block decoded before the current block or that uses a block of the picture that is larger than the hardware unit; - decoding the current block in response to the flag.

[0005] 1. A method of encoding comprising the steps of: - coding a flag indicating whether the current block of the picture is coded with a tool of a set of tools only if the current block has a parent hardware unit, the set of tools including a tool that uses reconstructed samples of a neighboring block coded before the current block or that uses a block of the picture larger than the hardware unit; - encoding the current block in response to the flag.

[0006] A coding device comprising at least one processor, the at least one processor comprising: - coding a flag indicating whether the current block of the picture is coded with a tool of a set of tools only if the current block has a parent hardware unit, the set of tools including a tool that uses reconstructed samples of a neighboring block coded before the current block or that uses a block of the picture larger than the hardware unit; - encoding the current block in response to the flag.

[0007] A signal, a flag indicating whether the current block of the picture is coded with a tool of a set of tools only if the current block has a parent hardware unit, the set of tools including a tool that uses reconstructed samples of a neighboring block coded before the current block or that uses a block of the picture that is larger than the hardware unit; - data representing the current block encoded in response to the flag.

[0008] A device, comprising: A decoding device according to any of the disclosed embodiments; A device is disclosed that includes at least one of: (i) an antenna configured to wirelessly receive a signal, the signal including a video having an image; (ii) a band limiter configured to limit the received signal to a band of frequencies that includes the video having an image; or (iii) a display configured to display an output.

[0009] A computer program product is disclosed that includes instructions that, when executed by one or more processors, cause the one or more processors to perform the decoding or encoding methods of any of the disclosed embodiments. [Brief description of the drawings]

[0010] [Figure 1] Represents a coding tree unit (CTU) divided into coding units (CUs) in HEVC. [Diagram 2] Represents a CTU that is spatially divided into one or more Prediction Units (PUs) and Transform Units (TUs) in HEVC. [Diagram 3] Illustrates the process of finding a match between two blocks along the motion trajectory of the current CU in two different reference pictures. [Figure 4] 1 illustrates the process of finding a match between a template of a current picture and a corresponding template. [Diagram 5]The process of deriving illumination compensation parameters by comparing an L-shape located in a current image with a corresponding L-shape in a reference image is illustrated. [Figure 6] 1 illustrates two coding units and a spatially corresponding hardware unit in accordance with a general aspect of at least one embodiment. [Figure 7] 1 illustrates the accuracy of processing at each stage throughout the decoding process, in accordance with a general aspect of at least one embodiment. [Figure 8] 1 illustrates an example processing pipeline for inter-frame decoding in accordance with a general aspect of at least one embodiment. [Figure 9] 1 shows a block diagram of one embodiment of a video encoder. [Figure 10] 1 shows a block diagram of one embodiment of a video encoder. [Figure 11] 1 shows an example of a partition tree and associated partition CTUs. [Figure 12] 1 illustrates a process for determining a parent hardware unit (HU) in accordance with a general aspect of at least one embodiment. [Figure 13] 1 illustrates a process for determining whether to launch a tool in Tdep, according to a general aspect of at least one embodiment. [Figure 14] 1 illustrates a gray CU in which the Tdep tool can be used, in accordance with a general aspect of at least one embodiment. [Figure 15] 1 illustrates a process for parsing flags in accordance with a general aspect of at least one embodiment. [Figure 16] 4 illustrates the derivation of a motion vector predictor for a merge candidate. [Figure 17] 1 illustrates a process of motion vector updating (left) and motion predictor derivation (right) for a CU in accordance with a general aspect of at least one embodiment. [Figure 18] 1 illustrates a process of updating a motion vector for a CU (left) and deriving a motion predictor as a function of a HU (right) in accordance with a general aspect of at least one embodiment. [Figure 19]1 illustrates a process for splitting a CU into two HUs in accordance with a general aspect of at least one embodiment. [Figure 20] 1 illustrates an example of splitting a CU in accordance with a general aspect of at least one embodiment. [Figure 21] 1 illustrates a block diagram of a system in which aspects of the present embodiments may be implemented. [Figure 22] 2 shows a flowchart of a decoding method according to one embodiment. [Diagram 23] 1 shows a flowchart of an encoding method according to one embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0011] The HEVC video compression standard uses motion-compensated temporal prediction to exploit the redundancy that exists between successive pictures of a video.

[0012] To do so, a motion vector, which we introduce here, is associated with each prediction unit (PU). Each CTU (coding tree unit) is represented by a coding tree in the compressed domain, which is a quadtree decomposition of the CTU, as shown in Figure 1, where each leaf is called a coding unit (CU).

[0013] Then, each CU is given some intra or inter prediction parameters (prediction information). To do so, it is spatially divided into one or more prediction units (PUs), and each PU is assigned some prediction information. As shown in Figure 2, the intra or inter coding mode is assigned at the CU level.

[0014] A motion vector is assigned to each PU in HEVC. This motion vector is used for the motion compensated temporal prediction of the considered PU. Thus, in HEVC, the motion model linking a predicted block with its reference block consists solely of a translation.

[0015] In the Joint Search Model (JEM), developed by the JVET (Joint Video Search Team) group, several motion models are supported to improve temporal prediction. To do so, a PU can be spatially partitioned into sub-PUs, and the model can be used to assign a dedicated motion vector to each sub-PU.

[0016] In other versions of JEM, CUs are no longer split into PUs or TUs, and some motion data is assigned directly to each CU. In this new codec design, CUs can be split into sub-CUs, and motion vectors can be calculated for each sub-CU.

[0017] For inter-frame motion compensation, a set of new tools has been developed that utilize decoder-side parameter estimation: FRUC (Frame Rate Up Conversion) Merging, FRUC Bilateral, and IC (Intensity Compensation).

[0018] The description of the FRUC tool is as follows: FRUC allows deriving the motion information of a decoder-side CU without signaling. This mode is signaled at the CU level with the FRUC flag and an additional FRUC mode flag, indicating which matching cost function (bilateral or template) should be used to derive the motion information of the CU.

[0019] On the encoder side, the decision to use the FRUC merge mode for a CU is based on the RD cost selection. Both two matching modes (bilateral and template) are checked for a CU. The one that leads to the smallest RD cost is further compared with other encoding modes. If the FRUC mode is the most efficient, the FRUC flag is set to true for the CU and the associated matching mode is used.

[0020] The motion derivation process in FRUC merge mode has two steps. First, a CU-level motion search is performed, followed by sub-CU-level motion refinement. At the CU level, an initial motion vector is derived from a list of CU-wide MV candidates based on bilateral or template matching. The candidate that leads to the minimum matching cost is selected as the starting point for further CU-level refinement. Then, a local search based on bilateral or template matching around the starting point is performed, and the MV that leads to the minimum matching cost is adopted as the CU-wide MV. Then, the motion information is further refined at the sub-CU level with the derived CU motion vector as the starting point.

[0021] As shown in Figure 3, a bilateral matching cost function is used to derive the motion information of the current CU by finding the best match between two blocks along the motion trajectory of the current CU in two different reference pictures. Under the assumption of continuous motion trajectories, the motion vectors MV0 and MV1 pointing to the two reference blocks are proportional to the temporal distances (TD0 and TD1) between the current picture and the two reference pictures.

[0022] As shown in FIG. 4, a template matching cost function is used to derive the motion information of the current CU by finding the best match between a template in the current picture (the neighboring blocks on the top and / or left of the current CU) and a corresponding template in the reference picture (of the same size as the template).

[0023] Note that this FRUC mode using template matching cost function also applies to AMVP (Advanced Motion Vector Prediction) mode. In this case, AMVP has two candidates. The new candidate is derived using the FRUC tool along with template matching. If this FRUC candidate is different from the first existing AMVP candidate, it is inserted at the beginning of the AMVP candidate list, and then the list size is set to 2 (i.e., the second existing AMVP candidate is deleted). When applied to AMVP mode, only CU-level search is applied.

[0024] In inter modes, IC (luminance compensation) allows the correction of block prediction samples obtained through motion compensation (MC) by taking into account spatial or temporal local luminance variations. IC parameters are estimated by comparing a set S of reconstructed neighboring samples (L-shape-cur) with the neighboring samples (L-shape-ref-i) of the reference-i block (i=0 or 1) as shown in Figure 5.

[0025] The IC parameters minimize the difference (least squares method) between the samples of L-shape-cur and the samples of L-shape-ref-i corrected by the IC parameters. Usually, the IC model is linear, i.e. IC(x)=a.x+b, where x is the value of the sample to be compensated.

[0026] The parameters a and b are derived by solving an L-shaped least squares minimization at the encoder (and decoder).

number

[0027] The problem solved by at least one embodiment is how to restrict data dependencies outside of a given block size.

[0028] Figure 6 shows an example of possible processing in a hardware decoder. In this example, we assume that the maximum size of a hardware unit (HU) that the decoder can process (usually set by the maximum transform of the codec) is 64 pixels. To increase parallelism, the decoder splits CUs larger than 64 pixels into blocks of maximum size 64 before sending them to the computation module (usually motion compensation and residual calculation).

[0029] Figure 7 shows an example of the accuracy (meaning possible parallelism) at each stage of the entire decoding process. First, the bitstream is parsed to generate a set of CUs to decode. Each CU is decoded to obtain a final value that is used to calculate the reconstructed samples. Note that there may be some causal relationships between CU decoding (usually the values ​​of motion vectors can be predicted from the values ​​of already decoded (but not reconstructed) CUs). Then, if necessary, split the CU into several HUs (depending on their size). At this point, all calculations within a HU must be performed independently of other HUs. Note that for intra CUs, there are still causal relationships between reconstructed CUs, since the intra prediction process needs to access neighboring samples at the prediction stage.

[0030] A problem solved by at least one embodiment is how to mitigate data dependencies introduced by tools such as FRUC. Figure 8 shows an example of a processing pipeline for inter-frame decoding. One of the problems that arises with tools such as FRUC or LIC (Local Illumination Compensation) when considering this kind of pipeline is that they introduce data dependencies on the reconstructed samples of previous CUs, thus breaking possible parallelism of HU processing.

[0031] At least one embodiment described herein is a method to avoid this dependency and enable highly parallel pipelining in the decoder.

[0032] FRUC, DMVR (decoder motion vector refinement), and LIC are new modes in JEM, so pipeline stalls are a relatively new issue.

[0033] The basic idea of ​​at least one embodiment is to break the dependency at the HU level. At least one of the proposed embodiments involves a normative modification of the codec, i.e. the encoding and decoding processes are completely symmetrical. The codec modules affected by one or more embodiments are modules 170 and possibly 160 in FIG. 9, and 275 and possibly 260 in FIG. 10.

[0034] Tdep is referred to as the set of tools that uses samples reconstructed from previous CUs, i.e., CUs coded before the current CU, or uses CUs larger than HU on the decoder side. - FRUC in template mode can use reconstructed samples outside the current CU. -LIC can use samples reconstructed outside the current CU. -DMVR can use CU larger than HU size. -BIO (Bidirectional Optical Flow) can use CUs larger than HU size.

[0035] Figure 22 shows a flow chart of a decoding method according to one embodiment. The method starts at S10. A flag is decoded (S12). The flag indicates whether a current block of a picture is decoded with a tool of a set of tools (Tdep). The flag is decoded only if the current block has a parent hardware unit. The set of tools includes tools that use reconstructed samples of neighboring blocks decoded before the current block or that use a block of said picture that is larger than the hardware unit. The current block is decoded in response to the flag (S14). The method ends at S16.

[0036] Figure 23 shows a flow chart of a coding method according to at least one embodiment. The method starts at S20. A flag is coded (S22). The flag indicates whether a current block of a picture is coded with a tool of a set of tools (Tdep). The flag is coded only if the current block has a parent hardware unit. The set of tools includes tools that use reconstructed samples of neighboring blocks coded before the current block or that use a block of said picture that is larger than the hardware unit. The current block is coded in response to the flag (S24). The method ends at S26.

[0037] HU Limit To allow for independent HU-based processing, new constraints for using the tools in the Tdep toolchain have been added. 1. The current CU has a parent HU. 2. The left and top neighboring CUs are within the parent HU of the current CU.

[0038] Figure 11 shows an example of a partition tree and associated partition CTU. The parent HU of a CU is defined as the first CU in the partition tree whose size (width and height) is less than or equal to S. For example, S = 64. For example, in Figure 11, C and D are parent HUs.

[0039] 12 shows a process of determining a parent HU, e.g., CU, of a block. The process starts at step S120. At step S121, it is checked whether h or w is greater than S, where h and w are the height and width of the current block, respectively. If h>S or w>S, the process continues at step S122, otherwise the process continues at step S125.

[0040] In S122, the current block does not have a parent HU.

[0041] In step S123 it is checked whether the current block is further divided into child blocks. If not, the process ends (S129). If yes, the current block is further divided into child blocks (S124) and the process continues for the child blocks at S121.

[0042] If yes, the process continues to S124, otherwise the process ends (S129).

[0043] In S125, HU is set equal to the current block (eg, its child block, block C).

[0044] In S126, the parent HU of the current block is the HU.

[0045] In S127 it is checked whether the current block is further divided. If not, the process ends (S129). If yes, the current block is further divided (S128) and the process continues for the child block in S126.

[0046] The second criterion checks whether the neighboring CUs of the current CU share the same parent HU with the current CU. Figure 13 shows a flowchart of the process of deciding whether to use the Tdep tool.

[0047] The process begins at S130.

[0048] In step S132, it is checked whether the current CU has a parent HU as shown in Fig. 12. If no, the process ends at S137 (Tdep tool is not used). If not, the process continues to S133, where it is checked whether the neighboring CUs (e.g., top and left) share the same parent HU as the current CU. If not, the process ends. If not, the process continues to S134.

[0049] In S134, a flag indicating whether a Tdep tool is used is decoded. If the flag is true (S135), the corresponding tool is used to decode the current CU (S136). If not, the process ends (S137), i.e., the corresponding Tdep tool is not used.

[0050] Figure 14 shows (in grey) some example CUs that can use the Tdep tool. The HU size is set to 64. In one embodiment, the parsing of the tool's flags in the Tdep tool is constrained by the process described in Figure 13. In one embodiment, the decoding of the tool's flags in the Tdep tool is constrained only by the presence of a parent HU, as shown in Figure 15.

[0051] The process starts (S130). It is checked whether the current CU has a parent HU (S132). If the flag is true (S135), the corresponding tool is used to decrypt the current CU (S136). If not, the process ends (S137), i.e. the corresponding Tdep tool is not used.

[0052] HU motion vector propagation Since the reconstructions of CUs within a HU are independent, the prediction of values ​​between HUs can also be constrained.

[0053] Below are some example values ​​generated by the tool for the Tdep set: - FRUC: The final value of the motion vector for a CU is given after the final motion refinement. -DMVR: The final value of the motion vector of a CU is given after the final motion refinement.

[0054] The generated value can be used as a prediction for further CUs (see FIG. 16 for an example of a predicted value). In S160, it is checked whether the given motion predictor is in the list of motion vector candidates. If not, it is added to the list (S162).

[0055] For the tool that generates the new motion vector of the CU, the prediction of the motion vector can also be constrained within the HU. In this embodiment, the predictor used by the CU is always the value before motion refinement, i.e., for blocks with decoder-side motion vector derivation tools (such as FRUC, DMVR, etc.), the final motion vector is not updated. From S170-1 to S178-1, the CU refines its own motion compensation motion vector candidate according to the FRUC flag. However, it uses the unrefined motion vector of the neighboring CU (S170-2 to S178-2) as a predictor. Figure 17 shows the process of motion vector update for the current CU (left) and the derivation of the motion predictor (right).

[0056] The motion vector update process starts at S170-1. At S172, the MV candidates are decoded.

[0057] In S174, MV1 is set equal to the decoded MV candidate.

[0058] In S175, it is checked whether FRUC is used (flag is true). If yes, the motion vector MV1 is refined with FRUC to obtain MV2 (S176). The current CU is motion compensated with MV2 (S177).

[0059] The process ends (S178-1).

[0060] The process of deriving a motion predictor starts at S170-2. Neighboring CUs, denoted by CUn, are considered for prediction (S171). The motion vector predictor used at S172 is set equal to MV1, rather than the refined vector of the neighboring CU.

[0061] Propagation only for CUs within the same HU 18 illustrates a process of updating a motion vector for a CU (left) and deriving a motion predictor according to a HU (right) according to a general aspect of at least one embodiment. From S180-1 to S190-1, the CU refines motion vector candidates according to the FRUC flag.

[0062] However, if the neighboring CU is outside the HU of the current CU (S183), then the unrefined motion vector of the neighboring CU (S185) is used as the predictor, and if they are within the same HU, then the refined motion vector (S187) is used.

[0063] The motion vector update process starts at S180-1. At S182, the MV candidates are decoded. At S184, MV1 and MV2 are set to the decoded motion vector candidates.

[0064] In S186, it is checked whether FRUC is used (flag is true). If yes, the motion vector MV1 is refined with FRUC to obtain MV2 (S188). The current CU is motion compensated with MV2 (S189).

[0065] The process ends (S190-1).

[0066] The motion predictor derivation process starts at S180-2. The neighboring CUs, denoted by CUn, are considered for prediction (S181). It is checked whether CUn is within the same HU as the current CU. If yes, the refined motion vector is used for prediction (S187). If not, the unrefined motion vector is used (S185). The method ends at S190-2.

[0067] In this embodiment, the motion vector predictor used depends on the parent HU of the CU.

[0068] Syntax associated with HU size and / or value propagation constraints, such as one or more flags, selection from a list, other indicators, e.g., regarding FRUC or IC limitations, can be signaled, e.g., at one or more of the slice, PPS (Picture Parameter Set), or SPS (Sequence Parameter Set) levels. In other embodiments, other levels, higher level syntax, or other methods are used. The associated syntax used for this signaling can include, for example, one or more flags, selection from a list, other indicators.

[0069] Multi-HU independent calculation for each CU A CU can be divided into several HUs for computation, as shown in Figure 6. In this case, a single syntax element is signaled at the CU level, but the processing is performed per HU. Note that the computation may produce different results within a CU. Examples of tools affected by this process are: -DMVR; - LIC in bidirectional mode.

[0070] In this case, processing (eg, refinement of bilateral motion vectors of DMVR, or bidirectional illumination compensation parameters) is performed for each HU separately.

[0071] FIG. 19 shows an example in which a CU is split into two HUs, and the unique flag is parsed and decoded for the CU, and if the CU size is larger than the HU size, each sub-HU performs independent DMVR processing.

[0072] From S191 to S201, if the CU size is larger than the HU size, the CU using the DMVR mode is divided into several HUs. The generated HUs then use independent DMVR processes (S199 and S200).

[0073] The process starts at S191. A flag indicating whether DMVR is applied is decoded (S192). If DMVR is applied, it is checked whether the CU size is larger than the HU size (S194). If not, DMVR is applied to the CU (S198). If yes, the CU is split into smaller hardware units (also called sub-HUs) HU0 and HU1 (S196). As shown in Figure 20, a CU can be split into three or more sub-HUs. Each sub-HU performs independent DMVR processing (S199 and S200). The process ends at S201.

[0074] HU division The division of a CU into sub-HUs can be performed in several ways. - Divide by the maximum HU size (both horizontal and vertical) and adapt the size of the last HU (horizontal and / or vertical) if they are outside the CU (Figure 20) -Divide the CU to have the same number of HUs horizontally and vertically, since there is a constraint that the size of each HU cannot be larger than the maximum size and must be a multiple of 4 (the minimum CU size). Various embodiments include one or more of the following: -Restriction on the use of Tdep's tools at the HU level + additional constraints on the causal boundaries of HU. -Adapt your syntax to these constraints -Several variants of the HU topology on the frame. -Adaptation of signaling and HU handling for some tools (DMVR, LIC bidir)

[0075] Various aspects are described in this document, including tools, features, embodiments, models, approaches, and the like. Many of these aspects are described specifically and at least to indicate their individual characteristics, and are described in what may often sound limiting manner. However, this is for clarity of description and is not intended to limit the application or scope of these aspects. Indeed, all of the various aspects can be combined and interchanged to provide additional aspects. Additionally, these aspects can be combined and interchanged with aspects described in previous applications.

[0076] The aspects described and contemplated in this document can be implemented in many different forms. Figures 10, 11, and 21 below provide some embodiments, but other embodiments are contemplated, and the discussion of Figures 10, 11, and 21 is not intended to limit the breadth of implementations. At least one of these aspects generally relates to video coding and decoding, and at least one other aspect generally relates to transmitting a generated or coded bitstream. These and other aspects can be implemented as a method, an apparatus, a computer-readable storage medium having stored thereon instructions for coding or decoding video data according to any of the described methods, and / or a computer-readable storage medium having stored thereon a bitstream generated according to any of the described methods.

[0077] In this application, the terms "reconstructed" and "decoded" may be used interchangeably, the terms "pixel" and "sample" may be used interchangeably, and the terms "image", "picture", and "frame" may be used interchangeably. Usually, but not necessarily, the term "reconstructed" is used on the encoder side, while "decoded" is used on the decoder side.

[0078] Various methods have been described above, each method including one or more steps or actions for achieving the described method. Unless a specific order of steps or actions is required for the correct operation of the method, the order and / or use of specific steps and / or actions may be modified or combined.

[0079] Various methods and other aspects described herein can be used to modify modules such as, for example, the motion compensation and, in some cases, intra prediction modules (170, 275, 160, 260) of the JVET or HEVC encoder 100 and decoder 200, as shown in Figures 10 and 11. Moreover, the aspects are not limited to JVET or HEVC, but can be applied to, for example, other standards and recommendations, whether existing or developed in the future, and extensions of such standards and recommendations (including JVET and HEVC). Unless otherwise specified or technically excluded, the aspects described herein can be used individually or in combination.

[0080] Various values ​​may be given in this document. The values ​​are for illustrative purposes only and the aspects described depend on these specific values. Not limited.

[0081] 10 illustrates an exemplary encoder 100. Although variations of this encoder 100 are contemplated, the encoder 100 is described below for clarity without describing all possible variations.

[0082] Before being coded, a video sequence may undergo a pre-coding process (101), such as applying a color transformation to the input color picture (e.g., from RGB 4:4:4 to YCbCr 4:2:0) or performing a remapping of the input picture components to obtain a signal distribution that is more resilient to compression (e.g., using histogram equalization of one of the color components). Metadata may be associated with the pre-processing and may be attached to the bitstream.

[0083] In the exemplary encoder 100, a picture is coded by the encoder elements as described below. The picture to be coded is divided (102) and processed, for example, in units of CUs. Each unit is coded, for example, using either intra mode or inter mode. When a unit is coded in intra mode, intra prediction is performed (160). In inter mode, motion estimation (175) and motion compensation (170) are performed. The encoder decides (105) whether intra mode or inter mode is used to code the unit, and indicates the intra / inter decision, for example, by a prediction mode flag. A prediction residual is calculated, for example, by subtracting (110) the predicted block from the original image block.

[0084] The prediction residual is then transformed (125) and quantized (130). The quantized transform coefficients, plus motion vectors and other syntax elements, are entropy coded (145) to output a bitstream. The encoder may skip the transform and apply quantization directly to the untransformed residual signal. The encoder may also bypass both the transform and quantization, i.e., the residual is directly coded without applying a transform or quantization process.

[0085] The encoder decodes the coded block to provide a reference for further prediction. The quantized transform coefficients are dequantized (140) and inverse transformed (150) to decode the prediction residual. The decoded prediction residual is combined (155) with the predicted block to reconstruct an image block. An in-loop filter (165) is applied to the reconstructed picture, for example to perform deblocking / SAO (sample adaptive offset) filtering to reduce coding artifacts. The filtered image is stored in a reference picture buffer (180).

[0086] Figure 11 shows a block diagram of an exemplary video decoder 200. In the exemplary decoder 200, the bitstream is decoded by a decoder element as described below. The video decoder 200 generally performs a decoding pass that is the inverse of the coding pass as described in Figure 10. The encoder 100 also generally performs video decoding as part of the coding of the video data.

[0087] In particular, the input of the decoder includes a video bitstream, which may be generated by the video encoder 100. First, the bitstream is entropy decoded (230) to obtain transform coefficients, motion vectors, and other coded information. Picture partition information indicates how the picture is partitioned. Thus, the decoder can partition the picture according to the decoded picture partition information (235). The transform coefficients are dequantized (240) and inverse transformed (250), and the prediction residual is decoded. The decoded prediction residual is combined with a predicted block (255) to reconstruct an image block. The predicted block may be obtained from intra prediction (260) or motion compensated prediction (i.e., inter prediction) (275) (270). An in-loop filter (265) is applied to the reconstructed image. The filtered image is stored in a reference picture buffer (280).

[0088] The decoded picture may further undergo post-decoding processing (285), such as an inverse color conversion (e.g., converting from YCbCr 4:2:0 to RGB 4:4:4) or inverse remapping that performs the inverse of the remapping process performed in the pre-coding process (101). The post-decoding process may use metadata derived in the pre-coding process and signaled in the bitstream.

[0089] FIG. 21 illustrates a block diagram of an exemplary system in which various aspects and exemplary embodiments are implemented. System 1000 may be embodied as a device including various components described below and configured to perform one or more of the aspects described herein. Examples of such devices include, but are not limited to, personal computers, laptop computers, smartphones, tablet computers, digital multimedia set-top boxes, digital television receivers, personal video recording systems, connected home appliances, and servers. System 1000 may be communicatively coupled to other similar systems and to displays via communication channels as shown in FIG. 21 and known to those skilled in the art, to implement various aspects described herein.

[0090] The system 1000 may include at least one processor 1010 configured to execute instructions loaded therein to implement various aspects described herein. The processor 1010 may include embedded memory, input / output interfaces, and various other circuits as is well known in the art. The system 1000 may include at least one memory 1020 (e.g., volatile memory device, non-volatile memory device). The system 1000 may include a storage device 1040, which may include non-volatile memory including, but not limited to, EEPROM, ROM, PROM, RAM, DRAM, SRAM, flash, magnetic disk drive, and / or optical disk drive. The storage device 1040 may include, as non-limiting examples, an internal storage device, an attached storage device, and / or a network-accessible storage device. The system 1000 may include an encoder / decoder module 1030 configured to process data to provide coded or decoded video, which may include its own processor and memory.

[0091] The encoder / decoder module 1030 represents a module or modules that may be included in a device to perform encoding and / or decoding functions. As is known, a device may include one or both of encoding and decoding modules. Furthermore, the encoder / decoder module 1030 may be implemented as a separate element of the system 1000 or may be incorporated within the processor 1010 as a combination of hardware and software as known to those skilled in the art.

[0092] Program code that is loaded onto the processor 1010 to perform various aspects described herein may be stored in the storage device 1040 and then loaded onto the memory 1020 for execution by the processor 1010. According to an example embodiment, one or more of the processor(s) 1010, the memory 1020, the storage device 1040, and the encoder / decoder module 1030 may store one or more of a variety of items, including, but not limited to, input video, decoded video, bitstreams, equations, formulas, matrices, variables, operations, and computational logic, during execution of processes described herein.

[0093] The system 1000 may include a communication interface 1050 that allows for communication with other devices over a communication channel 1060. The communication interface 1050 may include, but is not limited to, a transceiver configured to transmit and receive data from the communication channel 1060. The communication interface 1050 may include, but is not limited to, a modem or a network card, and the communication channel may be implemented in a wired and / or wireless medium. The various components of the system 1000 may all be connected or communicatively coupled to one another using a variety of suitable connections, including, but not limited to, an internal bus, wires, and printed circuit boards.

[0094] Exemplary embodiments may be implemented by computer software implemented by the processor 1010, or by hardware, or by a combination of hardware and software. As a non-limiting example, exemplary embodiments may be implemented by one or more integrated circuits. The memory 1020 may be of any type suitable for the technical environment, and may be implemented using any suitable data storage technology, such as, as non-limiting examples, optical memory devices, magnetic memory devices, semiconductor-based memory devices, fixed memories, and removable memories. The processor 1010 may be of any type suitable for the technical environment, and may include, as non-limiting examples, one or more of a microprocessor, a general-purpose computer, a special-purpose computer, and a processor based on a multi-core architecture.

[0095] The implementations and aspects described herein may be implemented, for example, in a method or process, an apparatus, a software program, a data stream, or a signal. Even if discussed in the context of only a single implementation (e.g., discussed only as a method), the implementation of the discussed features may also be implemented in other forms (e.g., an apparatus or a program). An apparatus may be implemented, for example, in appropriate hardware, software, and firmware. A method may be implemented in an apparatus such as, for example, a processor, which generally refers to a processing device including, for example, a computer, a microprocessor, an integrated circuit, or a programmable logic device. Processors also include communication devices, for example, computers, mobile phones, portable / personal digital assistants ("PDAs"), and other devices that facilitate the transfer of information between end users.

[0096] References to "one embodiment" or "an embodiment" or "one implementation" or "an implementation," as well as other variations thereof, mean that a particular feature, structure, characteristic, etc. described in connection with that embodiment is included in at least one embodiment. Thus, appearances of the phrases "in one embodiment" or "in an embodiment" or "in one implementation" or "in an implementation" in various places throughout this document, as well as any other variations, are not necessarily all referring to the same embodiment.

[0097] Additionally, this document may refer to "determining" various portions of information. Determining the information may include, for example, one or more of estimating the information, calculating the information, predicting the information, or retrieving the information from a memory.

[0098] Additionally, this document may refer to "accessing" various portions of the information.

[0099] Accessing information may include, for example, one or more of receiving information, retrieving information (e.g., from a memory), storing information, processing information, transmitting information, moving information, copying information, erasing information, calculating information, determining information, predicting information, or inferring information.

[0100] Additionally, this document may refer to "receiving" various portions of information. Receiving, like "accessing," is intended to be a broad term. Receiving information may include, for example, one or more of accessing information or retrieving information (e.g., from a memory). Additionally, "receiving" is typically included in some manner, such as, for example, storing information, processing information, transmitting information, moving information, copying information, erasing information, calculating information, determining information, predicting information, or estimating information.

[0101] As will be apparent to one of ordinary skill in the art, implementations can generate a wide variety of signals formatted to carry information that can be, for example, stored or transmitted. Information can include, for example, instructions for performing a method or data generated by one of the described implementations. For example, a signal can be formatted to carry a bit stream of the described embodiments. Such a signal can be formatted, for example, as an electromagnetic wave (e.g., using a radio frequency portion of the spectrum) or as a baseband signal. Formatting can include, for example, encoding a data stream and modulating a carrier wave with the encoded data stream. The information that the signal carries can be, for example, analog or digital information. The signal can be transmitted over a variety of different wired or wireless links, as is well known. The signal can be stored on a processor-readable medium.

[0102] Several embodiments have been described that provide at least the following generalized invention and claims, including all combinations thereof, across a variety of different claim categories and types. Relax, reduce or otherwise modify data dependencies introduced by encoding and / or decoding tools: ○Tools include FRUC, DMVR, LIC, and BIO Data dependency is the dependency between the block being decoded (current block) and neighboring blocks, e.g., CUs · Restrict the use of some tools at HU level if the tools include FRUC, DMVR, LIC, BIO · Restrict the use of some tools at HU level based on neighboring blocks. · Using some tools when at least one neighboring block (e.g. the top and / or left neighboring block) shares the same parent HU with the current block. Adjacent blocks are the blocks above and to the left of the current block. Adapting HU handling for some tools such as DMVR and LICbidir. · Limiting the predictive value between HUs. A bitstream or signal containing one or more of the listed syntax elements or variants thereof. · Inserting in signaling syntax elements that allow the decoder to process the bitstream in a way that corresponds to the way used by the encoder. Creating and / or transmitting and / or receiving and / or decoding a bitstream or signal comprising one or more of the listed syntax elements or variants thereof. A television, set-top box, mobile phone, tablet, or other electronic device running any of the described embodiments. A television, set-top box, mobile phone, tablet, or other electronic device that performs any of the described embodiments and displays the resulting images (e.g., using a monitor, screen, or other type of display). A television, set-top box, mobile phone, tablet, or other electronic device that tunes a channel (e.g., using a tuner) to receive a signal including the coded image and performs any of the described embodiments. A television, set-top box, mobile phone, tablet, or other electronic device that receives a signal containing the coded image over the air (e.g., using an antenna) and executes any of the described embodiments.

[0103] Various other generalized and specific inventions and claims are also supported and contemplated throughout this disclosure.

[0104] 1. A method of decoding, comprising: - decoding a flag indicating whether the current block of the picture is decoded with a tool of a set of tools only if the current block has a parent hardware unit, the set of tools including a tool that uses reconstructed samples of a neighboring block decoded before the current block or that uses a block of the picture that is larger than the hardware unit; - decoding the current block in response to the flag.

[0105] A decoding device comprising at least one processor, the at least one processor comprising: - decoding a flag indicating whether the current block of the picture is decoded with a tool of a set of tools only if the current block has a parent hardware unit, the set of tools including a tool that uses reconstructed samples of a neighboring block decoded before the current block or that uses a block of the picture that is larger than the hardware unit; - decoding the current block in response to the flag.

[0106] 1. A method of encoding, comprising: - coding a flag indicating whether the current block of the picture is coded with a tool of a set of tools only if the current block has a parent hardware unit, the set of tools including a tool that uses reconstructed samples of a neighboring block coded before the current block or that uses a block of the picture larger than the hardware unit; - encoding the current block in response to the flag.

[0107] A coding device comprising at least one processor, the at least one processor comprising: - coding a flag indicating whether the current block of the picture is coded with a tool of a set of tools only if the current block has a parent hardware unit, the set of tools including a tool that uses reconstructed samples of a neighboring block coded before the current block or that uses a block of the picture larger than the hardware unit; - coding the current block in response to the flag.

[0108] A signal, a flag indicating whether the current block of the picture is coded with a tool of a set of tools only if the current block has a parent hardware unit, the set of tools including a tool that uses reconstructed samples of a neighboring block coded before the current block or that uses a block of the picture that is larger than the hardware unit; - data representing the current block encoded in response to the flag.

[0109] A device, comprising: A decoding device according to any of the disclosed embodiments; A device is disclosed that includes at least one of: (i) an antenna configured to wirelessly receive a signal, the signal including a video having an image; (ii) a band limiter configured to limit the received signal to a band of frequencies that includes the video having an image; or (iii) a display configured to display an output.

[0110] A computer program product is disclosed that includes instructions that, when executed by one or more processors, cause the one or more processors to perform the decoding or encoding methods of any of the disclosed embodiments.

[0111] In one embodiment, the flags are decoded (respectively coded) only if the current block has a parent hardware unit and at least one neighboring block is located within the same parent hardware unit as the current block.

[0112] In one embodiment, the set of tools includes the following tools: frame rate up-conversion, local illumination compensation, decoder motion vector refinement, and bidirectional optical flow.

[0113] In one embodiment, the tool uses templates, and only the parts of the templates that are located in the same parent hardware unit as the current block are used.

[0114] In one embodiment, the tool uses a motion vector predictor with refinement, and when the current block and the adjacent block are located on the same hardware unit, the motion vector refinement is added to the motion vector predictor, and when the current block and the adjacent block are located on different hardware units, the motion vector refinement is not added to the motion vector predictor.

Claims

1. A decoding method comprising the steps of: decoding a first motion vector signaled for a first block of a picture from the video data; refining the first motion vector to generate a second motion vector, the first block being temporally predicted from a first area of ​​a first reference picture specified by the second motion vector; decoding a second block from the video data, the second block being temporally predicted from a second area of ​​a second reference picture specified by a third motion vector, the third motion vector being calculated as a sum of the second motion vector and a motion vector residual decoded from the video data in response to a first condition that the first block and the second block are in the same hardware unit, and the third motion vector being calculated as a sum of the first motion vector and a motion vector residual decoded from the video data in response to a second condition that the first block and the second block are not in the same hardware unit; A decoding method comprising:

2. The decoding method of claim 1, wherein the third motion vector is calculated after decoding of the first motion vector in response to the first condition being satisfied.

3. A decoding device comprising at least one processor, The at least one processor decoding a first motion vector signaled for a first block of a picture from the video data; refining the first motion vector to generate a second motion vector, the first block being temporally predicted from a first area of ​​a first reference picture specified by the second motion vector; decoding a second block from the video data, the second block being temporally predicted from a second area of ​​a second reference picture specified by a third motion vector, the third motion vector being calculated as a sum of the second motion vector and a motion vector residual decoded from the video data in response to a first condition that the first block and the second block are in the same hardware unit, and the third motion vector being calculated as a sum of the first motion vector and a motion vector residual decoded from the video data in response to a second condition that the first block and the second block are not in the same hardware unit; A decoding device configured to execute the

4. The decoding device of claim 3, wherein the third motion vector is calculated after decoding of the first motion vector in response to the first condition being satisfied.

5. An encoding method comprising the steps of: signaling in the video data a first motion vector for a first block of a picture; refining the first motion vector to generate a second motion vector, the first block being temporally predicted from a first area of ​​a first reference picture specified by the second motion vector; signaling a second block into the video data, the second block being temporally predicted from a second area of ​​a second reference picture specified by a third motion vector, the third motion vector being signaled in the video data with motion information comprising a residual motion vector corresponding to a difference between the second motion vector and the third motion vector in response to a first condition that the first block and the second block are on a same hardware unit, and the third motion vector being signaled in the video data with motion information comprising a residual motion vector corresponding to a difference between the first motion vector and the third motion vector in response to a second condition that the first block and the second block are not on the same hardware unit; 13. An encoding method comprising:

6. The encoding method of claim 5, wherein the third motion vector is determined after determining the first motion vector in response to the first condition being satisfied.

7. An encoding device comprising at least one processor, The at least one processor signaling in the video data a first motion vector for a first block of a picture; refining the first motion vector to generate a second motion vector, the first block being temporally predicted from a first area of ​​a first reference picture specified by the second motion vector; signaling a second block into the video data, the second block being temporally predicted from a second area of ​​a second reference picture specified by a third motion vector, the third motion vector being signaled in the video data with motion information comprising a residual motion vector corresponding to a difference between the second motion vector and the third motion vector in response to a first condition that the first block and the second block are on a same hardware unit, and the third motion vector being signaled in the video data with motion information comprising a residual motion vector corresponding to a difference between the first motion vector and the third motion vector in response to a second condition that the first block and the second block are not on the same hardware unit; 23. An encoding device configured to:

8. The encoding device described in Claim 7, wherein the third motion vector is determined after determining the first motion vector in response to the first condition being satisfied.

9. A computer program comprising program code instructions for carrying out the method of claim 1.

10. A computer program comprising program code instructions for carrying out the method of claim 5.

11. An information storage medium storing program code instructions for carrying out the method of claim 1.

12. An information storage medium storing program code instructions for carrying out the method of claim 5.

Images