VIDEO ENCODING METHOD, PROGRAM, AND BITSTREAM TRANSMISSION METHOD - Patent application

Abstract

To provide a system and a method for performing video encoding by restricting and adjusting application of a CIIP (combined inter and intra prediction) mode.SOLUTION: A video decoding method according to the invention includes obtaining a first reference picture and a second reference picture associated with a current block. In display order, the first reference picture is before the current picture and the second reference picture is after the current picture. The method also includes: determining whether to apply a DMVR operation based on a CIIP flag of the current block; determining whether to apply a BDOF (bidirectional optical flow) operation based on the CIIP flag; and calculating a prediction for the current block based on the first reference picture, the second reference picture, and determination on whether to apply the DMVR operation and the BDOF operation.SELECTED DRAWING: Figure 7

Description

[Technical field]

[0001] FIELD OF THE DISCLOSURE This disclosure relates generally to video coding, and more particularly to a system and method for performing video coding with constraints and adjustments for the application of Combined Inter- and Intra-Prediction (CIIP) modes. [Background technology]

[0002] This section provides background information related to the present disclosure. The information contained in this section should not necessarily be construed as prior art.

[0003] Any of a variety of video encoding techniques may be used to compress the video data. The video encoding may be performed according to one or more video encoding standards. Some exemplary video encoding standards include versatile video coding (VVC), joint exploration test model (JEM) coding, H.265 / HEVC (high-efficiency video coding), H.264 / AVC (advanced video coding), and moving picture experts group (MPEG) coding.

[0004] Video coding generally employs prediction methods (e.g., inter-prediction, intra-prediction, etc.) that exploit the redundancy inherent in a video image or sequence. One goal of video coding techniques is to compress video data into a format that uses a lower bit rate while avoiding or minimizing degradation of video quality.

[0005] Prediction methods utilized in video coding typically involve performing spatial (intra-frame) prediction and / or temporal (inter-frame) prediction to reduce or remove redundancy inherent in video data, and are typically associated with block-based video coding. Summary of the Invention [Means for solving the problem]

[0006] This section provides a general overview of the disclosure and is not an exhaustive disclosure of its entire scope or any of its features.

[0007] A first aspect is a video decoding method, comprising: determine whether to apply a decoder-side motion vector refinement (DMVR) operation based on the combined inter and intra prediction (CIIP) flag, determine whether to apply a bidirectional optical flow (BDOF) operation based on the combined inter and intra prediction (CIIP) flag, and calculate a prediction of the current block based on the decision regarding whether to apply the DMVR operation and the BDOF operation.

[0008] A second aspect is a video decoding method of the first aspect, wherein determining whether to apply the DMVR operation includes determining not to apply the DMVR operation in response to the CIIP flag indicating that a CIIP is applied to the current block.

[0009] A third aspect is the video decoding method of the first aspect, further comprising: applying the BDOF calculation. Determining whether or not includes determining not to apply the BDOF calculation in response to the CIIP flag indicating that CIIP is applied to the current block.

[0010] A fourth aspect is a video decoding method of the first aspect, wherein calculating a prediction of the current block includes, in response to determining not to apply the DMVR operation and the BDOF operation, calculating a bi-prediction of the current block based on a first prediction L0 and a second prediction L1, wherein the first prediction L0 is obtained based on a first motion vector MV0 from the current block to a reference block of a first reference picture that is before the current picture in display order, and the second prediction L1 is obtained based on a second motion vector MV1 from the current block to a reference block of a second reference picture that is after the current picture in display order.

[0011] A fifth aspect is a video decoding method of the first aspect, wherein determining whether to apply the DMVR operation includes determining to apply the DMVR operation in response to the CIIP flag indicating that CIIP is not applied to the current block.

[0012] A sixth aspect is a video decoding method of the first aspect, comprising adjusting a first motion vector MV0 and a second motion vector MV1 to generate a first updated prediction L0' and a second updated prediction L1', where the first motion vector MV0 is a vector from the current block to a reference block of a first reference picture that is before the current picture in display order, and the second motion vector MV1 is a vector from the current block to a reference block of a second reference picture that is after the current picture in display order.

[0013] A seventh aspect is a video decoding method of the sixth aspect, wherein determining whether to apply the BDOF operation includes deciding to apply the BDOF operation in response to the CIIP flag indicating that the CIIP is not applied to the current block.

[0014] An eighth aspect is a video decoding method of the seventh aspect, further comprising calculating a first horizontal and vertical gradient value associated with the first updated prediction L0' and calculating a second horizontal and vertical gradient value associated with the second updated prediction L1'.

[0015] A 9th aspect is a video decoding method of the 8th aspect, wherein calculating a prediction of the current block includes calculating a bi-prediction of the current block based on the first updated prediction L0', the second updated prediction L1', the first horizontal and vertical gradient values, and the second horizontal and vertical gradient values. [Brief description of the drawings]

[0016] [Figure 1] 1 is a block diagram illustrating an example block-based hybrid video encoder that can be used in conjunction with many video encoding standards. [Diagram 2] 1 is a block diagram showing an example video decoder that can be used in conjunction with many video encoding standards. [Figure 3A] 4 illustrates an example of a division type, a 4-division, according to an exemplary embodiment. [Figure 3B] 1 illustrates a horizontal bisection, which is an example of a division type, according to an exemplary embodiment. [Figure 3C] 1 illustrates a vertical bisection, which is an example of a division type, according to an exemplary embodiment. [Figure 3D] 13 illustrates an example of a split type, horizontal 3 split, according to an exemplary embodiment. [Figure 3E] 13 illustrates an example of a division type, a vertical third division, according to an exemplary embodiment. [Figure 4A] FIG. 13 is a diagram illustrating combining inter-prediction and intra-prediction in CIIP mode. [Figure 4B] FIG. 13 is a diagram illustrating combining inter-prediction and intra-prediction in CIIP mode. [Figure 4C] FIG. 13 is a diagram illustrating combining inter-prediction and intra-prediction in CIIP mode. [Figure 5A]1 constitutes a pair of flowcharts illustrating an exemplary generation process for an MPM candidate list in a current VVC. [Figure 5B] 1 constitutes a pair of flowcharts illustrating an exemplary generation process for an MPM candidate list in a current VVC. [Figure 6] 1 is a flowchart illustrating an example workflow of a CIIP design using Bi-Directional Optical Flow (BDOF) in current VVC. [Figure 7] 11 is a flowchart illustrating an example workflow for selectively bypassing DMVR and BDOF operations when computing a bi-prediction of a current predicted block. [Figure 8] 1 is a flowchart illustrating an exemplary workflow of a CIIP design proposed in the present disclosure. [Figure 9] 1 is a flowchart illustrating an exemplary workflow of the proposed second CIIP design in the present disclosure. [Figure 10A] 1 constitutes a pair of flowcharts illustrating exemplary workflows of two proposed methods for processing CIIP-encoded blocks in the MPM candidate list generation process in the present disclosure. [Figure 10B] 1 constitutes a pair of flowcharts illustrating exemplary workflows of two proposed methods for processing CIIP-encoded blocks in the MPM candidate list generation process in the present disclosure. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0017] A set of exemplary and non-limiting embodiments of the present disclosure will be described below in conjunction with the accompanying drawings. Structural, method, or functional modifications may be implemented by those skilled in the relevant art based on the examples presented herein, and all such modifications are included within the scope of the present disclosure. In the absence of any inconsistency, the teachings of different embodiments can be combined with each other, but do not necessarily have to be combined.

[0018] The terms used in this disclosure are intended to describe specific examples, not to limit the disclosure. As used in this disclosure as well as in the appended claims, singular forms such as "a" and "an" also refer to the plural, unless otherwise clearly indicated in the context. It should be understood that the term "and / or" as used herein refers to any or all possible combinations of one or more of the associated listed items.

[0019] Terms such as "first," "second," and "third" may be used herein to describe various pieces of information, but should not be construed as limiting the information. These terms are used only to distinguish one category of information from another. For example, the first information may be referred to as the second information, and similarly, the second information may be referred to as the first information, without departing from the scope of this disclosure. As used herein, the term "when" means "when" or "in the event of" or "in response to," depending on the context.

[0020] Throughout this specification, references to "one embodiment," "another embodiment," or the like means that one or more particular features, structures, or characteristics described in connection with one embodiment are included in at least one embodiment of the disclosure, and thus, the appearances of phrases such as "in one embodiment" or "in another embodiment" in one or more places throughout this specification are not necessarily all referring to the same embodiment. Furthermore, particular features, structures, or characteristics in one or more embodiments may be combined in any suitable manner.

[0021] Conceptually, many video coding standards are similar, including those mentioned in the background section, for example, virtually all video coding standards use block-based processing and share similar video coding block diagrams to achieve video compression.

[0022] In block-based video coding, an input video signal is processed block-by-block. For each block (also called a coding unit (CU)), spatial and / or temporal prediction may be performed.

[0023] Spatial prediction (also called "intra prediction") predicts the current block using pixels from samples of already coded neighboring blocks (called reference samples) in the same video picture / slice. Spatial prediction reduces the spatial redundancy inherent in video signals.

[0024] Temporal prediction (also called "inter prediction" or "motion compensated prediction") predicts the current block using reconstructed pixels from already coded video pictures. Temporal prediction reduces the temporal redundancy inherent in video signals. The temporal prediction signal for a given CU is typically signaled by one or more motion vectors (MVs) that indicate the amount and direction of motion between the current CU and its temporal references. Also, if multiple reference pictures are supported, one reference picture index is additionally transmitted, which is used to identify which reference picture in the reference picture store the temporal prediction signal comes from.

[0025] After spatial and / or temporal prediction, a mode decision block in the encoder selects the best prediction mode, for example based on a rate-distortion optimization method. The prediction block is then subtracted from the current block, and the prediction residual is decorrelated and quantized using a transform. The quantized residual coefficients are inverse quantized and inverse transformed to form a reconstructed residual, which is then re-added to the prediction block to form a reconstructed signal for the block.

[0026] After spatial and / or temporal prediction, further in-loop filtering, such as a deblocking filter, sample adaptive offset (SAO), and adaptive in-loop filter (ALF), may be applied to the reconstructed CU before it is put into a reference picture store and used to encode future video blocks. To form an output video bitstream, the coding mode (inter or intra), prediction mode information, motion information, and quantized residual coefficients are all sent to an entropy coding unit, where they are further compressed and packed to form a bitstream.

[0027] During the decoding process, the video bitstream is first entropy decoded in the entropy decoding unit. The coding mode and prediction information are sent to either the spatial prediction unit (if intra-coded) or the temporal prediction unit (if inter-coded) to form a prediction block. The residual transform coefficients are sent to the inverse quantization unit and the inverse transform unit to reconstruct the residual block. The prediction block and the residual block are then added. The reconstructed block may further undergo in-loop filtering before being stored in the reference picture store. The reconstructed video in the reference picture store is then sent to drive a display device and is also used to predict future video blocks.

[0028] In many hybrid video coding schemes, each block may use either inter- or intra-prediction methods, but not both. However, the residual signals generated by inter-predicted and intra-predicted blocks present very different characteristics. Therefore, if the two kinds of predictions can be combined in an efficient way, a more accurate prediction can be expected to reduce the energy of the prediction residual and improve the coding efficiency. Furthermore, in certain video content, the motion of moving objects may be complicated. For example, there may be regions that contain both old content (e.g., objects included in a previously coded picture) and newly appearing content (e.g., objects excluded in a previously coded picture). In such a scenario, neither inter nor intra prediction can provide an accurate prediction of the current block.

[0029] To further improve prediction efficiency, the current version of VVC introduces a Combined Inter- and Intra-Prediction (CIIP) mode, which combines intra-prediction and inter-prediction of one CU coded by a combined mode. The application of the CIIP mode generally improves coding efficiency.

[0030] However, application of the CIIP mode involves more operations than those involved in the inter or intra modes, which tends to increase the computational complexity and reduce the encoding / decoding throughput.

[0031] Furthermore, the current version of VVC is not entirely consistent in its handling of intra-mode and CIIP mode blocks in the context of building Most Probable Mode (MPM) candidate lists for their neighboring blocks.

[0032] FIG. 1 shows a block diagram of an exemplary block-based hybrid video encoder 100 that may be used in conjunction with many video coding standards. In the encoder 100, a video frame is divided into multiple video blocks for processing. For each given video block, a prediction is formed based on either an inter-prediction approach or an intra-prediction approach. In inter-prediction, one or more predictors are formed based on pixels from a previously reconstructed frame by motion estimation and motion compensation. In intra-prediction, a predictor is formed based on reconstructed pixels in a current frame. A mode decision allows the selection of the best predictor to predict the current block.

[0033] A prediction residual, which represents the difference between the current video block and its predictor, is sent to a transform circuit 102. The transform coefficients are then sent from the transform circuit 102 to a quantization circuit 104 for entropy reduction. The quantized coefficients are then provided to an entropy coding circuit 106 to generate a compressed video bitstream. As shown in Figure 1, prediction related information 110 from an inter prediction circuit and / or an intra prediction circuit 112, such as video block partition information, motion vectors, reference picture indexes, and intra prediction modes, are also provided through the entropy coding circuit 106 for inclusion in the compressed video bitstream 114.

[0034] The encoder 100 also requires decoder-related circuitry to reconstruct pixels for prediction purposes. First, the prediction residual is reconstructed through an inverse quantization circuit 116 and an inverse transform circuit 118. This reconstructed prediction residual is combined with a block predictor 120 to generate unfiltered reconstructed pixels for the current video block.

[0035] In-loop filters are commonly used to improve coding efficiency and visual quality. For example, deblocking filters are available in current versions of AVC, HEVC, and VVC. In HEVC, SAO filters are used to further improve coding efficiency. An additional in-loop filter called the sample adaptive offset (ALF) is defined. In the current VVC standard, yet another in-loop filter called the Adaptive Loop Filter (ALF) is under active consideration and is likely to be included in the final standard.

[0036] These in-loop filter operations are optional. Performing these operations helps improve coding efficiency and visual quality. They can also be turned off as decisions made by the encoder 100 to reduce computational complexity.

[0037] Note that intra prediction is typically based on unfiltered reconstructed pixels, while inter prediction is based on filtered reconstructed pixels if these filter options are turned on by the encoder 100.

[0038] FIG. 2 is a block diagram illustrating an exemplary video decoder 200 that may be used with many video coding standards. The decoder 200 is similar to the reconstruction-related portion present in the encoder 100 of FIG. 1. In the decoder 200 (FIG. 2), an input video bitstream 201 is first decoded via entropy decoding 202 to derive quantized coefficient levels and prediction-related information. The quantized coefficient levels are then processed via inverse quantization 204 and inverse transform 206 to obtain a reconstructed prediction residual. A block prediction mechanism implemented in an intra / inter mode selector 208 is configured to perform either intra prediction 210 or motion compensation 212 based on the decoded prediction information. The reconstructed prediction residual from the inverse transform 206 and the prediction output generated by the block prediction mechanism are summed using an adder 214 to obtain a set of unfiltered reconstructed pixels. In the situation where an in-loop filter is turned on, a filtering operation is performed on these reconstructed pixels to derive the final reconstructed video. The reconstructed video in the reference picture store is then sent to drive a display device and is used to predict future video blocks.

[0039] In video coding standards such as HEVC, blocks may be divided based on a quadtree. In newer video coding standards such as current VVC, more division methods are used, and one coding tree unit (CTU) can be divided into CUs to adapt to different local characteristics based on quadtree, binary tree, or ternary tree. The separation of CU, prediction unit (PU) and transform unit (TU) does not exist in most coding modes in current VVC, and each CU is always used as a basic unit for both prediction and transformation without further division. However, in some specific coding modes such as intra sub-partition coding mode, each CU can still contain multiple TUs. In the multi-type tree structure, first, one CTU is divided by a quadtree structure. Then, each quadtree leaf node can be further divided by binary tree structure and ternary tree structure.

[0040] 3A-3E show five exemplary partition types: 4-partition (FIG. 3A), horizontal 2-partition (FIG. 3B), vertical 2-partition (FIG. 3C), horizontal 3-partition (FIG. 3D), and vertical 3-partition (FIG. 3E). In video coding standards such as HEVC and current VVC, each CU can be predicted with intra prediction or inter prediction.

[0041] The current VVC also introduces a Combined Intra- and Inter-Prediction (CIIP) coding mode, under which intra-prediction and inter-prediction are combined for one CU coded by the combined mode.

[0042] In CIIP mode, for each merged CU, indicates whether CIIP is enabled for the current CU. One additional flag is signaled. For the luma component of the current CU, CIIP supports four frequently used intra modes: planar mode, DC mode, horizontal mode, and vertical mode. For the chroma component of the current CU, the DM mode, which means the same intra mode of the luma component is reused, is always applied without extra signaling.

[0043] Furthermore, in CIIP mode, for each merged CU, weighted averaging is applied to combine the inter-predicted samples and intra-predicted samples of the current CU. In particular, when planar mode or DC mode is selected, equal weights (i.e., 0.5) are applied. Otherwise (i.e., when either horizontal mode or vertical mode is applied), the current CU is first divided horizontally (for horizontal mode) or vertically (for vertical mode) into four equally sized regions. (w_intra i , w_inter i Four weight sets, denoted as (w_intra), are applied to combine inter- and intra-predicted samples in different regions, where i=0 and i=3 represent the regions closest and furthest from the reconstructed neighbor samples used for intra prediction. In the current CIIP design, a predefined set of values ​​for the weight sets are 0 , w_inter 0 )=(0.75, 0.25), (w_intra 1 , w_inter 1 )=(0.625, 0.375), (w_intra 2 , w_inter 2 )=(0.375, 0.625) and (w_intra 3 , w_inter 3 )=(0.25, 0.75).

[0044] 4A-4C illustrate the combination of inter and intra prediction in CIIP mode. In FIG. 4A, the operation of combining weights for CU 401 in horizontal mode is illustrated. The current CU is divided horizontally into four equally sized regions 402, 403, 404, and 405, represented by four different grayscale shades. For each region, a weight set is applied to combine inter and intra predicted samples. For example, for the leftmost region 402, represented by the darkest grayscale shade, a weight set (w_intra 0 , w_inter 0)=(0.75, 0.25) is applied. This means that the CIIP prediction is obtained as the sum of (i) 0.75, i.e., 3 / 4 times, of the intra prediction, and (ii) 0.25, i.e., 1 / 4 times, of the inter prediction. This is shown by the formula 406P, which is indicated by an arrow in the same region 402. CIIP =P inter / 4+3P intra The weight sets used for the other three regions 403, 404, and 405 are also shown by equation 409P. The weight sets used for the other three regions 403, 404, and 405 are shown similarly. In FIG. 4B, the operation of the combining weights for CU 407 in vertical mode is shown in a manner similar to FIG. 4A. In FIG. 4C, the operation of the combining weights for CU 408 in planar or DC mode is shown, where only one set of weights, a set of equal weights, is applied to the entire CU 408, again indicated by an arrow pointing to the entire CU 408, equation 409P. CIIP =P inter / 2+P intra / 2.

[0045] The current VVC working specification also provides for the use of the intra mode of one CIIP CU as a predictor for predicting the intra mode of its neighboring CIIP CU via the most probable mode (MPM) mechanism, but does not provide for the use of the same intra mode of the same CIIP CU as a predictor for predicting the intra mode of its neighboring intra-coded CU via the MPM mechanism.

[0046] In detail, for each CIIP CU, if some of its neighboring blocks are also CIIP CUs, the intra modes of those neighboring blocks are first rotated to the closest mode among planar, DC, horizontal, and vertical modes, and then added to the MPM candidate list of the current CU. That is, the intra mode of one CIIP CU predicts the intra mode of its neighboring CIIP CU. However, for each intra CU, if some of its neighboring blocks are coded by CIIP modes, these neighboring blocks are not used. It is considered impossible, i.e., the intra mode of one CIIP CU does not predict the intra mode of its neighboring intra CUs.

[0047] 5A and 5B constitute a pair of flowcharts showing an exemplary generation process for an MPM candidate list in a current VVC. FIG. 5A illustrates the generation process when the current CU is an intra block. In this case, it is determined whether the neighboring CU is an intra block (501), and only if it is an intra block, the intra mode of the neighboring CU is added to the MPM candidate list (502). FIG. 5B illustrates the generation process when the current CU is an intra block. In this case, it is determined whether the neighboring CU is an intra block (503), and if it is an intra block, the intra mode of the neighboring CU is rotated (504) and added to the MPM candidate list (505), while if it is not an intra block, it is still determined whether the neighboring CU is a CIIP block (506), and if it is a CIIP block, the intra mode of the neighboring CU is added to the MPM candidate list (507).

[0048] 5A and 5B illustrate the inconsistency between the use of CIIP blocks and the use of intra blocks in the process of generating an MPF ​​candidate list in a current VVC, that is, the method of using adjacent CIIP blocks in the process of generating an MPM candidate list depends on whether the current block is an intra block or a CIIP block, and the two cases are different.

[0049] Newer video coding standards, such as current VVC, have introduced new inter-mode coding tools. Two examples of new inter-mode coding tools are Bi-Directional Optical Flow (BDOF) and Decoder-side Motion Vector Refinement (DMVR).

[0050] Traditional bi-prediction in video coding is a simple combination of two temporal prediction blocks obtained from already reconstructed reference pictures. However, due to the limitations of block-based motion compensation, there may be small remaining motion observable between the samples of the two prediction blocks, thus reducing the efficiency of motion-compensated prediction. To solve this problem, BDOF is applied in current VVC to reduce the effect of such motion for all samples in one block.

[0051] BDOF is a sample-wise motion refinement performed on block-based motion compensation prediction when bi-prediction is used. The motion refinement of each 4x4 subblock is calculated by minimizing the difference between the reference picture list 0 (L0) and reference picture list 1 (L1) predicted samples after applying BDOF inside one 6x6 window around the subblock. Based on the motion refinement so derived, the final bi-predicted sample of the CU is calculated by interpolating the L0 / L1 predicted samples along the motion trajectory based on the optical flow model.

[0052] DMVR is a bi-prediction technique for merging two initially signaled MVs and blocks, which can be further refined by using bilateral matching prediction. Bilateral matching is used to derive the motion information of the current CU by finding the closest match between two blocks along the motion trajectory of the current CU in two different reference pictures. The cost function used in the matching process is row subsample sum of absolute difference (SAD). After the matching process is done, the refined MV is used for motion compensation in the prediction stage, boundary strength calculation in the deblocking filter, temporal motion vector prediction for subsequent pictures, and cross-CTU spatial motion vector prediction for subsequent CUs. Under the assumption of continuous motion trajectory, the motion vectors MV0 and MV1 pointing to the two reference blocks shall be proportional to the temporal distances between the current picture and the two reference pictures, i.e., TD0 and TD1. As a special case, when the current picture is temporally between the two reference pictures and the temporal distances from the current picture to the two reference pictures are the same, the bilateral matching becomes mirror-based bidirectional MV.

[0053] In the current VVC, both BDOF and DMVR can be used in conjunction with the CIIP mode.

[0054] 6 is a flowchart showing an example workflow of CIIP design using Bi-Directional Optical Flow (BDOF) in current VVC. In this workflow, L0 and L1 motion compensation (601 and 602) are processed through BDOF (603), and the output from BDOF is processed together with intra prediction (604) to form a weighted average (605) in CIIP mode.

[0055] The current VVC also provides a similar application of DMVR in conjunction with the CIIP mode.

[0056] In the current VVC, the CIIP mode can improve the efficiency of motion compensation prediction. However, this disclosure has identified three problems that exist in the current CIIP design included in the current VVC.

[0057] First, since CIIP combines inter-prediction and intra-prediction samples, each CIIP CU needs to use its reconstructed neighboring samples to generate a prediction signal. This means that the decoding of one CIIP CU depends on the complete reconstruction of its neighboring blocks. Due to such interdependence, in a practical hardware implementation, CIIP needs to be performed in the reconstruction stage where neighboring reconstructed samples become available for intra prediction. Since the decoding of CUs in the reconstruction stage must be performed sequentially (i.e., one by one), it is undesirable to include too many computation operations (e.g., multiplications, additions, and bit shifts) in the CIIP process to ensure sufficient throughput for real-time decoding. Furthermore, in the current CIIP design in the current VVC, new inter-mode encoding tools such as BDOF and DMVR are also included to generate inter-prediction samples for CIIP modes. Considering the additional complexity introduced by the new inter-mode encoding tools, such a design may significantly reduce the encoding / decoding throughput of a hardware codec when CIIP is enabled.

[0058] Second, in the current CIIP design in the current VVC, when one CIIP CU references one merge candidate that is bi-predicted, both the motion compensation prediction signals in lists L0 and L1 need to be generated. If one or more MVs are not integer precision, an additional interpolation process must be invoked to interpolate samples at fractional sample positions. Such a process not only increases the amount of computation, but also increases the memory bandwidth when more reference samples need to be accessed from external memory, which may result in a significant reduction in the encoding / decoding throughput of the hardware codec when CIIP is enabled.

[0059] Third, in the current CIIP design in the current VVC, the intra mode of a CIIP CU and the intra mode of an intra CU are treated differently when constructing the MPM list of their neighboring blocks. In particular, when a current CU is coded by a CIIP mode, its neighboring CIIP CUs are considered as intra, i.e., However, if the current CU is coded by intra mode, the intra mode of its neighboring CIP CUs can be added to the MPM candidate list. The CU is considered as an inter mode, i.e., the intra modes of adjacent CIP CUs are excluded from the MPM candidate list. Such a non-uniform design may not be optimal for the final version of the VVC standard.

[0060] This disclosure proposes constraints and adjustments on the application of CIIP mode to address the above three issues.

[0061] According to the present disclosure, after identifying a CU that is a candidate for application of the CIIP mode, a determination is made as to whether the CU identified as a candidate for application of the CIIP mode is bi-predicted or uni-predicted, and then, based on that determination, application of the CIIP mode to the CU is restricted.

[0062] According to one embodiment of the present disclosure, constraining the application of a CIIP mode to a CU based on a determination includes disabling operation of one or more predefined inter-prediction techniques in generating inter-predicted samples during application of the CIIP mode to the CU when the CU is bi-predicted.

[0063] In one example, the one or more pre-defined inter-prediction techniques include BDOF and DMVR.

[0064] FIG. 7 is a flowchart showing an example workflow for selectively bypassing DMVR and BDOF calculations when calculating a bi-prediction of a current predicted block. During processing of a current predicted block, a first reference picture and a second reference picture associated with the current predicted block are obtained (702), where the first reference picture is before the current picture and the second reference picture is after the current picture in display order. Then, a first prediction L0 is obtained based on a first motion vector MV0 from the current predicted block to a reference block of the first reference picture (703), and a second prediction L1 is obtained based on a second motion vector MV1 from the current predicted block to a reference block of the second reference picture (704). Then, a decision is made whether to apply a DMVR calculation (705). If applied, the DMVR calculation adjusts the first motion vector MV0 and the second motion vector MV1 based on the first prediction L0 and the second prediction L1 to generate a first updated prediction L0' and a second updated prediction L1' (706). A second determination is then made as to whether to apply a BDOF computation (707). If so, the BDOF computation calculates first horizontal and vertical gradient values ​​for prediction samples associated with the first updated prediction L0' and second horizontal and vertical gradient values ​​associated with the second updated prediction L1' (708). Finally, a bi-prediction of the current prediction block is calculated based on the first prediction L0, the second prediction L1, optionally the first updated prediction L0', optionally the second updated prediction L1', optionally the first horizontal and vertical gradient values, and optionally the second horizontal and vertical gradient values ​​(709).

[0065] In another example, the one or more predefined inter prediction techniques include BDOF. Figure 8 is a flowchart showing an example workflow of CIIP design in this example of the present disclosure. In this workflow, L0 and L1 motion compensation (801 and 802) are averaged (803) instead of being processed through BDOF, and the resulting average is then processed together with intra prediction (804) to form a weighted average (805) in CIIP mode.

[0066] In a third example, the one or more pre-defined inter-prediction techniques include DMVR.

[0067] According to another embodiment of the present disclosure, the restriction of the application of the CIIP mode to the CU based on the decision is , including using a predefined criterion to select, from all available uni-predictive samples, a number of uni-predictive samples to be used in combination with intra-predicted samples for the CU during application of a CIIP mode to the CU when the CU is bi-predicted.

[0068] In one example, the predefined criteria includes selecting all uni-predictive samples based on pictures in reference picture list 0 (L0).

[0069] In another example, the predefined criteria includes selecting all uni-predictive samples based on pictures in reference picture list 1 (L1).

[0070] In a third example, the predefined criteria includes selecting all uni-prediction samples based on one reference picture with the smallest picture order count (POC) distance from the picture where the CU is located ("current picture"). Figure 9 is a flowchart showing an example workflow of the proposed CIIP design in this example of the present disclosure. In this workflow, a determination is made as to whether the L0 or L1 reference picture is closer to the current picture in POC distance (901), and the motion compensation from one reference picture with the smallest POC distance from the current picture is selected (902 and 903), which is then processed together with intra prediction (904) to form a weighted average in CIIP mode (905).

[0071] According to another embodiment of the present disclosure, restricting the application of the CIIP mode to the CU based on the decision includes disabling the application of the CIIP mode to the CU, and disabling the signaling of the CIIP flag for the CU when the CU is bi-predicted. In particular, to reduce overhead, the signaling of the CIIP enable / disable flag depends on the prediction direction of the current CIIP CU. If the current CU is uni-predictive, the CIIP flag is signaled in the bitstream to indicate whether the CIIP is enabled or disabled. Otherwise (i.e., if the current CU is bi-predicted), the signaling of the CIIP flag is skipped and inferred to be always false, i.e., the CIIP is always disabled.

[0072] Also, according to the present disclosure, when forming an MPM candidate list for a CU (the "current CU"), a determination is made as to whether each CU among the neighboring CUs of the current CU is CIIP coded or not, and then, for each neighboring CU among the CIIP coded neighboring CUs, a unified criterion that is independent of the determination of whether the current CU is intra coded or CIIP coded is used in the process of using the intra mode of the neighboring CU in forming the MPM candidate list for the current CU.

[0073] According to one embodiment of this aspect of the disclosure, the unified criteria includes treating the intra mode of a neighboring CU as unavailable for forming an MPM candidate list when the neighboring CU is CIIP coded.

[0074] According to another embodiment of this aspect of the disclosure, the unified criteria includes treating CIIP inventions of neighboring CUs as equivalent to intra inventions in forming the MPM candidate list if the neighboring CUs are CIIP coded.

[0075] 10A and 10B form a pair of flowcharts illustrating an example workflow of two embodiments of this aspect of the present disclosure. FIG. 10A illustrates the workflow of the first embodiment of this aspect of the present disclosure. In this workflow, first, a decision is made as to whether the neighboring CU is an intra block or a CIIP block (1001), and only if it is an intra block or a CIIP block, its intra mode is added to the MPM candidate list of the current CU, regardless of whether the current CU is an intra block or a CIIP block. 10B shows a workflow of a second embodiment of this aspect of the present disclosure, in which the neighboring CU is first determined to be an intra block (1003), and only if it is an intra block, the intra mode is added to the MPM candidate list of the current CU (1004), regardless of whether the current CU is an intra block or a CIIP block.

[0076] In one or more examples, the functions described may be implemented in hardware, software, firmware, or any combination thereof. If implemented in software, the functions may be stored on or transmitted over a computer-readable medium as one or more instructions or code and executed by a hardware-based processing unit. A computer-readable storage medium may include a computer-readable storage medium corresponding to a tangible medium, such as a data storage medium, or a communication medium including any medium that facilitates transfer of a computer program from one place to another, for example according to a communication protocol. In this manner, a computer-readable storage medium may generally correspond to (1) a tangible computer-readable storage medium that is non-transitory, or (2) a communication medium such as a signal or carrier wave. A data storage medium may be any available medium accessible by one or more computers or one or more processors to retrieve instructions, code and / or data structures for implementation as described herein. A computer program product may include a computer-readable storage medium.

[0077] Furthermore, the above method can be implemented in ASICs (application specific integrated circuits), DSPs (digital signal processors), DSPDs (digital signal processing The modules, sub-modules, units, or sub-units disclosed above may be implemented, at least in part, using one or more circuits, including programmable logic devices (PLDs), field programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors, or other electronic components. The apparatus may use the circuits in combination with other hardware or software components to perform the methods described above. Each module, sub-module, unit, or sub-unit disclosed above may be implemented, at least in part, using one or more circuits.

[0078] Other embodiments of the invention will be apparent to those skilled in the art from consideration of the specification and practice of the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of the invention in accordance with the general principles of the invention, including departures from the present disclosure that come within known or customary practice in the art. It is intended that the specification and embodiments are merely exemplary, with the true scope and spirit of the invention being indicated by the following claims.

[0079] It will be understood that the invention is not limited to the exact examples described above and shown in the accompanying drawings, but various modifications and changes can be made thereto without departing from the scope of the invention, which is intended to be limited only by the scope of the appended claims.

[0080] CROSS-REFERENCE TO RELATED APPLICATIONS

[0081] This application claims the benefit of U.S. Provisional Patent Application No. 62 / 817,503, filed March 12, 2019. The entire disclosure of the aforementioned application is incorporated herein by reference in its entirety.

Claims

1. Obtain a first reference picture and a second reference picture associated with the current block; the first reference picture is before a current picture and the second reference picture is after the current picture in display order; determining whether to apply a decoder-side motion vector refinement (DMVR) operation based on a combined inter and intra prediction (CIIP) flag of the current block; determining whether to apply a bidirectional optical flow (BDOF) operation based on the combined inter and intra prediction (CIIP) flag; calculating a prediction of the current block based on the first reference picture, the second reference picture, and a decision as to whether to apply the DMVR operation and the BDOF operation; Video encoding methods.

2. 2. The video encoding method of claim 1, wherein determining whether to apply the DMVR operation comprises determining not to apply the DMVR operation in response to the CIIP flag indicating that a CIIP is applied to the current block.

3. 2. The video encoding method of claim 1, wherein determining whether to apply the BDOF operation comprises determining not to apply the BDOF operation in response to the CIIP flag indicating that CIIP is applied to the current block.

4. Calculating a prediction of the current block comprises: responsive to determining not to apply the DMVR and BDOF operations, calculating a bi-prediction of the current block based on a first prediction L0 and a second prediction L1; the first prediction L0 is obtained based on a first motion vector MV0 from the current block to a reference block of the first reference picture preceding the current picture in display order; the second prediction L1 is obtained based on a second motion vector MV1 from the current block to a reference block of the second reference picture that is after the current picture in display order; 2. The video encoding method of claim 1.

5. Determining whether to apply the DMVR operation includes: The video encoding method of claim 1 , comprising: determining to apply the DMVR operation in response to the CIIP flag indicating that CIIP is not applied to the current block.

6. adjusting the first motion vector MV0 and the second motion vector MV1 to generate the first updated prediction L0' and the second updated prediction L1'; the first motion vector MV0 is a vector from the current block to a reference block of the first reference picture that precedes the current picture in display order; the second motion vector MV1 is a vector from the current block to a reference block of the second reference picture that is after the current picture in display order; 6. The video encoding method of claim 5.

7. Determining whether to apply the BDOF operation includes: The video encoding method of claim 6 , comprising: determining to apply the BDOF operation in response to the CIIP flag indicating that the CIIP is not applied to the current block.

8. Calculating first horizontal and vertical gradient values ​​associated with the first updated prediction L0'; calculating second horizontal and vertical gradient values ​​associated with the second updated prediction L1'; The video encoding method of claim 7, further comprising:

9. Calculating a prediction of the current block comprises: calculating a bi-prediction of the current block based on the first updated prediction L0′, the second updated prediction L1′, the first horizontal and vertical gradient values, and the second horizontal and vertical gradient values; 9. The video encoding method of claim 8, comprising:

10. 1. A program for execution by a computing device having one or more processors, comprising: When executed by one or more of said processors, the method causes the computer device to perform the video encoding method of any one of claims 1 to 9. program.

11. A method for transmitting a bitstream generated by an image encoding device, comprising the steps of: The image encoding device comprises: one or more processors; A computer readable medium having instructions stored thereon; Including, The instructions, when executed by one or more of the processors, cause the image coding device to perform the video coding method of any one of claims 1 to 9. Bitstream transmission method.

Images