Constrained and adjusted application of combined inter and intra prediction modes

Abstract

A method of performing, at a computing device, an application of constraining and adjusting a combined Inter and Intra Prediction (CIIP) mode employed in a video coding standard, such as now current Universal Video Coding (VVC). In one method, a computing device determines whether to apply decoder-side motion vector refinement (DMVR) and bidirectional optical flow (BDOF) to a bidirectional prediction block. In another method, a computing device determines whether each of neighboring CUs of a CU ("current CU") is CIIP-coded, respectively, and in a process of forming a most probable mode (MPM) candidate list for the current CU using intra modes of the neighboring CIIP-coded CUs, employs a unified criterion that is independent of determining whether the current CU is intra-coded or CIIP-coded.

Description

[0001] This application is a divisional application of the application with application number "202080020524.0", application date "March 12, 2020", and invention name "Constrained and Regulated Application of Combined Inter-Frame and Intra-Frame Prediction Modes".

[0002] CROSS-REFERENCE TO RELATED APPLICATIONS

[0003] This application claims the benefit of U.S. Provisional Patent Application No. 62 / 817,503, filed on March 12, 2019. The entire disclosure of the above application is incorporated herein by reference in its entirety. Technical Field

[0004] The present disclosure relates generally to video coding and compression. More specifically, the present disclosure relates to systems and methods for performing video coding using constraints and adjustments for application of combined inter and intra prediction (CIIP) modes. Background Art

[0005] This section provides background information related to the present disclosure. The information contained in this section should not necessarily be construed as prior art.

[0006] Any of a variety of video codec techniques may be used to compress video data. Video coding may be performed according to one or more video codec standards. Some illustrative video codec standards include Versatile Video Codec (VVC), Joint Exploration Test Model (JEM) codec, High Efficiency Video Codec (H.265 / HEVC), Advanced Video Codec (H.264 / AVC), and Moving Picture Experts Group (MPEG) codec.

[0007] Video codecs usually use prediction methods (such as inter-frame prediction, intra-frame prediction, etc.) that exploit the inherent redundancy in video images or sequences. One goal of video codec technology is to compress video data into a form that uses a lower bit rate while avoiding or minimizing the degradation of video quality.

[0008] Prediction methods used in video codecs typically include performing spatial (intra-frame) prediction and / or temporal (inter-frame) prediction to reduce or remove redundancy inherent in video data, and are typically associated with block-based video codecs.

[0009] Public Content

[0010] This section provides a general summary of the disclosure, and is not a comprehensive disclosure of its full scope or all of its features.

[0011] According to a first aspect of the present disclosure, a video encoding and decoding method is performed at a computing device having one or more processors and a memory, wherein the memory stores a plurality of programs to be executed by the one or more processors. The method includes: dividing each picture in a video stream into a plurality of blocks or CUs. The method includes: when a CU is bidirectionally predicted, during the application of a CIIP mode to the CU, bypassing the operation of one or more inter-frame prediction processes when generating inter-frame prediction samples. One or more bypassed inter-frame prediction processes include decoder-side motion vector refinement (DMVR) and bidirectional optical flow (BDOF).

[0012] According to a second aspect of the present disclosure, a video encoding and decoding method is performed at a computing device, the computing device having one or more processors and a memory, the memory storing multiple programs to be executed by the one or more processors. The method includes: dividing each picture in the video stream into multiple blocks or CUs. The method also includes: identifying a CU that is a candidate for application of the CIIP mode. The method also includes: determining whether the CU identified as a candidate for application of the CIIP mode is bidirectionally predicted or unidirectionally predicted. The method also includes: constraining the application of the CIIP mode to the CU based on the determination.

[0013] According to a third aspect of the present disclosure, a video coding method is performed at a computing device having one or more processors and a memory, wherein the memory stores a plurality of programs to be executed by the one or more processors. The method includes: dividing each picture in a video stream into a plurality of blocks or coding units (CUs). The method also includes: deriving an MPM candidate list for each CU. The method also includes: determining whether each CU in the adjacent CUs of the CU ("current CU") is a block encoded and decoded by CIIP, respectively. The method also includes: for each adjacent CU in the adjacent CUs encoded and decoded by CIIP, in the process of forming the MPM candidate list for the current CU using the intra-frame mode of the adjacent CU, a unified criterion that does not depend on determining whether the current CU is intra-frame encoded or CIIP encoded is adopted.

[0014] According to a fourth aspect of the present application, a computing device includes one or more processors, a memory, and a plurality of programs stored in the memory. When executed by the one or more processors, the programs enable the computing device to perform the operations described above.

[0015] According to a fifth aspect of the present application, a non-transitory computer-readable storage medium stores a plurality of programs for execution by a computing device having one or more processors. When executed by the one or more processors, the programs cause the computing device to perform the operations described above. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Hereinafter, a set of illustrative, non-limiting embodiments of the present disclosure will be described in conjunction with the accompanying drawings. A person of ordinary skill in the relevant art may implement changes in structure, method, or function based on the examples presented herein, and these changes are all included in the scope of the present disclosure. In the absence of conflicts, the teachings of different embodiments may, but need not, be combined with each other.

[0017] Figure 1 is a block diagram illustrating an illustrative block-based hybrid video encoder that may be used in conjunction with many video codec standards.

[0018] Figure 2 is a block diagram illustrating an illustrative video decoder that can be used in conjunction with many video codec standards.

[0019] Figure 3A-3E An exemplary split type, namely a quadruple split ( Figure 3A ), horizontal binary segmentation ( Figure 3B ), vertical binary segmentation ( Figure 3C ), horizontal ternary segmentation ( Figure 3D ) and vertical ternary split ( Figure 3E ).

[0020] Figure 4A-4C is a diagram of combining inter prediction and intra prediction in CIIP mode.

[0021] Figure 5A-Figure 5B is a pair of flow charts describing an illustrative generation process of the MPM candidate list in the current VVC.

[0022] Figure 6 is a flow chart describing an illustrative workflow for CIIP design using bidirectional optical flow (BDOF) in current VVC.

[0023] Figure 7 is a flow chart setting forth an illustrative workflow for selectively bypassing DMVR operations and BDOF operations when computing a bidirectional prediction for a current prediction block.

[0024] Figure 8 is a flow chart setting forth an illustrative workflow of the CIIP design proposed in this disclosure.

[0025] Fig. 9 is a flow chart illustrating an illustrative workflow of the second CIIP design proposed in this disclosure.

[0026] Figure 10A-10B is a pair of flow charts setting forth the illustrative workflows of two proposed methods for processing CIIP codec blocks in the generation of an MPM candidate list in the present disclosure. DETAILED DESCRIPTION

[0027] The terms used in this disclosure are intended to illustrate specific examples rather than to limit the disclosure. The singular forms "a", "an", and "the" used in this disclosure and the appended claims also refer to the plural forms unless other meanings are clearly implied by the context. It should be understood that the term "and / or" as used herein refers to any and all possible combinations of one or more of the associated listed items.

[0028] It should be understood that although the terms first, second, third, etc. may be used herein to describe various information, such information should not be limited to these terms. These terms are only used to distinguish one information from another. For example, without departing from the scope of the present disclosure, the first information may be referred to as the second information; and similarly, the second information may be referred to as the first information. As used herein, the term "if" may be understood to mean "when..." or "upon..." or "in response to...", depending on the context.

[0029] References throughout this specification to "one embodiment," "another embodiment," etc. in the singular or plural form mean that one or more specific features, structures, or characteristics described in connection with the embodiment are included in at least one embodiment of the present disclosure. Thus, the phrases "in one embodiment" or "in one embodiment," "in another embodiment," etc. appearing in singular or plural form at various places throughout this specification do not necessarily refer to the same embodiment. Furthermore, the specific features, structures, or characteristics of one or more embodiments may be combined in any suitable manner.

[0030] Conceptually, many video codec standards are similar, including those mentioned previously in the Background section. For example, almost all video codec standards use block-based processing and share similar video codec block diagrams to achieve video compression.

[0031] In block-based video coding, the input video signal is processed block by block. For each block (also called a codec unit (CU)), spatial prediction and / or temporal prediction can be performed.

[0032] Spatial prediction (also called "intra-prediction") uses pixels from samples of already coded neighboring blocks in the same video picture / slice (called reference samples) to predict the current block. Spatial prediction reduces the spatial redundancy inherent in video signals.

[0033] Temporal prediction (also called "inter prediction" or "motion compensated prediction") uses reconstructed pixels from already coded video pictures to predict the current block. Temporal prediction reduces the temporal redundancy inherent in video signals. The temporal prediction signal for a given CU is typically signaled via one or more motion vectors (MVs) that indicate the amount and direction of motion between the current CU and its temporal reference. In addition, when multiple reference pictures are supported, a reference picture index is additionally sent that identifies which reference picture in the reference picture library the temporal prediction signal comes from.

[0034] After spatial prediction and / or temporal prediction, a mode decision block in the encoder selects the best prediction mode, for example based on a rate-distortion optimization method. The prediction block is then subtracted from the current block, and the prediction residual is decorrelated using transformation and quantization. The quantized residual coefficients are inverse quantized and inverse transformed to form a reconstruction residual, which is then added back to the prediction block to form the reconstructed signal for that block.

[0035] After spatial prediction and / or temporal prediction, further loop filtering, such as deblocking filter, sample adaptive offset (SAO) and adaptive loop filter (ALF), may be applied to the reconstructed CU before it is placed in the reference picture library and used to encode and decode future video blocks. To form the output video bitstream, the coding mode (inter or intra), prediction mode information, motion information and quantized residual coefficients are all sent to the entropy coding unit for further compression and packaging to form the bitstream.

[0036] During the decoding process, the video bitstream is first entropy decoded in the entropy decoding unit. The codec mode and prediction information are sent to the spatial prediction unit (in the case of intra-frame coding) or the temporal prediction unit (in the case of inter-frame coding) to form a prediction block. The residual transform coefficients are sent to the inverse quantization unit and the inverse transform unit to reconstruct the residual block. The prediction block and the residual block are then added. The reconstructed block may be further loop filtered before it is stored in the reference picture library. The reconstructed video in the reference picture library is then sent out to drive the display device and is used to predict future video blocks.

[0037] In many hybrid video coding schemes, each block can use either an inter-frame or intra-frame prediction method, rather than both. However, the residual signals generated by the inter-frame prediction block and the intra-frame prediction block may exhibit very different characteristics from each other. Therefore, if the two predictions can be combined in an efficient manner, a more accurate prediction can be expected to reduce the energy of the prediction residual and thereby improve the coding efficiency. In addition, in some video content, the motion of moving objects may be complex. For example, there may be areas that contain both old content (e.g., objects included in previously coded pictures) and new content that is revealed (e.g., objects not included in previously coded pictures). In such a situation, neither inter-frame prediction nor intra-frame prediction can provide an accurate prediction of the current block.

[0038] In order to further improve the prediction efficiency, the current version of VVC introduces the combined inter-frame and intra-frame prediction (CIIP) mode, which combines the intra-frame prediction and inter-frame prediction of a CU encoded and decoded by the merge mode. The application of the CIIP mode improves the encoding and decoding efficiency overall.

[0039] However, application of the CIIP mode involves more operations than those involved in the inter-mode or intra-mode, which tends to increase computational complexity and reduce encoding / decoding throughput.

[0040] Furthermore, within the context of constructing Most Probable Mode (MPM) candidate lists for neighboring blocks of intra mode blocks and CIIP mode blocks, the now current version of VVC is not completely consistent in its treatment of intra mode blocks and CIIP mode blocks.

[0041] Figure 1 A block diagram of an illustrative block-based hybrid video encoder 100 is shown, which can be used in conjunction with many video codec standards. In the encoder 100, a video frame is divided into a plurality of video blocks for processing. For each given video block, a prediction is formed based on an inter-frame prediction method or an intra-frame prediction method. In inter-frame prediction, one or more prediction values ​​are formed through motion estimation and motion compensation based on pixels from a previously reconstructed frame. In intra-frame prediction, prediction values ​​are formed based on pixels reconstructed in the current frame. Through mode decision, the best prediction value can be selected to predict the current block.

[0042] The prediction residual representing the difference between the current video block and its predicted value is sent to the transform circuit 102. The transform coefficients are then sent from the transform circuit 102 to the quantization circuit 104 for entropy reduction. The quantized coefficients are then fed to the entropy codec circuit 106 to generate a compressed video bitstream. Figure 1As shown, prediction related information 110 from the inter prediction circuit and / or the intra prediction circuit 112 , such as video block segmentation information, motion vector, reference picture index and intra prediction mode, is also fed through the entropy encoding and decoding circuit 106 and saved into the compressed video bitstream 114 .

[0043] In encoder 100, decoder-related circuitry is also required to reconstruct pixels for prediction purposes. First, the prediction residual is reconstructed by inverse quantization circuit 116 and inverse transform circuit 118. The reconstructed prediction residual is combined with block prediction value 120 to generate unfiltered reconstructed pixels for the current video block.

[0044] In order to improve the codec efficiency and visual quality, loop filters are usually used. For example, deblocking filters are available in the current versions of AVC, HEVC and VVC. In HEVC, an additional loop filter called SAO (Sample Adaptive Offset) is defined to further improve the codec efficiency. In the current version of the VVC standard, another loop filter called ALF (Adaptive Loop Filter) is being actively studied and is likely to be included in the final standard.

[0045] These loop filter operations are optional. Performing these operations helps improve codec efficiency and visual quality. They can also be turned off as a decision made by the encoder 100 to save computational complexity.

[0046] It should be noted that intra prediction is usually based on unfiltered reconstructed pixels, while inter prediction is based on filtered reconstructed pixels if these filter options are turned on by the encoder 100.

[0047] Figure 2 is a block diagram illustrating an illustrative video decoder 200 that can be used in conjunction with many video codec standards. The decoder 200 is similar to the one residing in Figure 1 The reconstruction-related part of the encoder 100. In the decoder 200 ( Figure 2), the incoming video bitstream 201 is first decoded by entropy decoding 202 to derive quantization coefficient levels and information related to prediction. The quantized coefficient levels are then processed by inverse quantization 204 and inverse transform 206 to obtain reconstructed prediction residuals. The block prediction value mechanism implemented in the intra / inter mode selector 208 is configured to perform intra prediction 210 or motion compensation 212 based on the decoded prediction information. Using an adder 214, the reconstructed prediction residual from the inverse transform 206 is added to the prediction output generated by the block prediction value mechanism to obtain a set of unfiltered reconstructed pixels. With the loop filter turned on, a filtering operation is performed on these reconstructed pixels to derive the final reconstructed video. The reconstructed video in the reference picture library is then sent out to drive the display device and is used to predict future video blocks.

[0048] In video codec standards such as HEVC, blocks can be segmented based on quadtrees. In newer video codec standards (such as the current VVC), more segmentation methods are used, and a codec tree unit (CTU) can be split into multiple CUs to adapt to different local characteristics based on quadtrees, binary trees, or ternary trees. In the current VVC, there is no separation of CU, prediction unit (PU), and transform unit (TU) in most codec modes, and each CU is always used as a basic unit for both prediction and transformation without further segmentation. However, in some specific codec modes (such as intra-frame sub-segmentation codec mode), each CU can still contain multiple TUs. In a multi-type tree structure, a CTU is first segmented by a quadtree structure. Then, each quadtree tree child node can be further segmented by a binary tree and a ternary tree structure.

[0049] Figure 3A-3E Five example split types are shown, namely, quadruple splits ( Figure 3A ), horizontal binary segmentation ( Figure 3B ), vertical binary segmentation ( Figure 3C ), horizontal ternary segmentation ( Figure 3D ) and vertical ternary split ( Figure 3E ). In video codec standards such as HEVC and the current VVC, each CU can be intra-predicted or inter-predicted.

[0050] The current VVC now also introduces a combined intra and inter prediction (CIIP) codec mode, in which the intra prediction and inter prediction of a CU coded and decoded by the merge mode are combined.

[0051] In CIIP mode, for each merged CU, an additional flag is signaled to indicate whether CIIP is enabled for the current CU. For the luma component of the current CU, CIIP supports four frequently used intra modes, namely planar mode, DC mode, horizontal mode, and vertical mode. For the chroma components of the current CU, the DM mode is always applied without additional signaling, and the DM mode means reusing the same intra mode of the luma component.

[0052] In addition, in CIIP mode, for each merged CU, a weighted average is applied to combine the inter-frame prediction samples and intra-frame prediction samples of the current CU. Specifically, when the planar mode or DC mode is selected, an equal weight (i.e., 0.5) is applied. Otherwise (i.e., horizontal mode or vertical mode is applied), the current CU is first split horizontally (for horizontal mode) or vertically (for vertical mode) into four regions of equal size. It is represented as (w_intra i ,w_inter i ) will be applied to combine inter-prediction samples and intra-prediction samples in different regions, where i=0 and i=3 represent the regions closest and farthest from the reconstructed neighboring samples for intra-prediction. In the current CIIP design, the predefined value sets of the weight sets are: (w_intra0, w_inter0)=(0.75, 0.25), (w_intra1, w_inter1)=(0.625, 0.375), (w_intra2, w_inter2)=(0.375, 0.625) and (w_intra3, w_inter3)=(0.25, 0.75).

[0053] Figure 4A-4C is an illustration of combining inter-frame prediction and intra-frame prediction in CIIP mode. Figure 4A , the operation of combining weights for CU 401 in horizontal mode is shown. The current CU is split horizontally into four regions 402, 403, 404 and 405 of equal size, represented by four different grayscale shades. For each region, a set of weights is applied to combine inter-frame predicted samples and intra-frame predicted samples. For example, for the leftmost region 402 represented by the darkest grayscale shade, the weight set (w_intra0, w_inter0) = (0.75, 0.25) is applied, which means that a CIIP prediction is obtained as the sum of (i) 0.75 or 3 / 4 times the intra prediction and (ii) 0.25 or 1 / 4 times the inter prediction. This is also shown by formula 406 attached to the same region 402 with an arrow The weight sets for the other three regions 403, 404 and 405 are similarly shown. Figure 4B In Figure 4AThe operation of the combining weights for CU 407 in vertical mode is described in a similar manner. Figure 4C , the operation of combining weights for CU 408 in planar mode or DC mode is illustrated, where only one set of weights, namely a set of equal weights, is applied to the entire CU 408, which is also indicated by formula 409 attached to the entire CU 408 with an arrow. To explain.

[0054] The current VVC working specification also stipulates that the intra-frame mode of a CIIP CU is used as a prediction value to predict the intra-frame mode of its adjacent CIIP CU through the most probable mode (MPM) mechanism, but it does not stipulate that the same intra-frame mode of the same CIIP CU is used as a prediction value to predict the intra-frame mode of its adjacent intra-coded CU through the MPM mechanism.

[0055] Specifically, for each CIIP CU, if some of its neighboring blocks are also CIIP CUs, the intra modes of these neighbors are first rounded to the closest mode among the plane mode, DC mode, horizontal mode, and vertical mode, and then added to the MPM candidate list of the current CU, that is, the intra mode of a CIIP CU is allowed to predict the intra mode of its neighboring CIIP CU. However, for each intra CU, if some of its neighboring blocks are encoded and decoded by the CIIP mode, these neighboring blocks are considered unavailable, that is, the intra mode of a CIIP CU is not allowed to predict the intra mode of its neighboring intra CU.

[0056] Figure 5A-Figure 5B is a pair of flow charts describing an illustrative generation process of the MPM candidate list in the current VVC. Figure 5A The generation process of the current CU being an intra block is illustrated. In this case, a determination is made as to whether the neighboring CU is an intra block (501), and if and only if it is an intra block, the intra mode of the neighboring CU is added to the MPM candidate list (502). Figure 5B The generation process of the current CU being an intra block is illustrated. In this case, a determination is made as to whether the adjacent CU is an intra block (503), and when it is an intra block, the intra mode of the adjacent CU is rounded (504) and then added to the MPM candidate list (505), on the one hand, when it is not an intra block, a subsequent determination is made as to whether the adjacent CU is a CIIP block (506), and when it is a CIIP block, the intra mode of the adjacent CU is added to the MPM candidate list (507).

[0057] Figure 5A-Figure 5BThe diagram illustrates the difference between the use of CIIP blocks and intra blocks in the generation process of the MPF candidate list in the current VVC, that is, the method of using adjacent CIIP blocks in the generation process for the MPM candidate list depends on whether the current block is an intra block or a CIIP block, and the two cases are different.

[0058] In newer video codec standards, such as the now current VVC, new inter-mode codec tools have been introduced, and two examples of these new inter-mode codec tools are: Bidirectional Optical Flow (BDOF) and Decoder-side Motion Vector Refinement (DMVR).

[0059] Conventional bidirectional prediction in video codecs is a simple combination of two temporal prediction blocks obtained from an already reconstructed reference picture. However, due to the limitations of block-based motion compensation, there may be residual small motion that can be observed between the samples of the two prediction blocks, which reduces the efficiency of motion compensated prediction. To solve this problem, BDOF is applied in the current VVC to reduce the impact of such motion on each sample within a block.

[0060] BDOF is a sample-by-sample motion refinement performed on top of block-based motion compensated prediction when bidirectional prediction is used. The motion refinement for each 4×4 sub-block is calculated by minimizing the difference between reference picture list 0 (L0) prediction samples and reference picture list 1 (L1) prediction samples after applying BDOF within a 6×6 window around the sub-block. Based on the motion refinement so derived, the final bidirectional prediction samples of the CU are calculated by interpolating the L0 / L1 prediction samples along the motion trajectory based on the optical flow model.

[0061] DMVR is a bidirectional prediction technique for merging blocks with two initially signaled MVs that can be further refined using bilateral matching prediction. Bilateral matching is used to derive the motion information of the current CU by finding the closest match between two blocks along the motion trajectory of the current CU in two different reference pictures. The cost function used in the matching process is the sum of absolute differences (SAD) of row subsampling. After the matching process is completed, the refined MV is used for motion compensation in the prediction stage, boundary strength calculation in the deblocking filter, temporal motion vector prediction for subsequent pictures, and cross-CTU spatial motion vector prediction for subsequent CUs. Under the assumption of continuous motion trajectory, the motion vectors MV0 and MV1 pointing to the two reference blocks should be proportional to the temporal distance (i.e. TD0 and TD1) between the current picture and the two reference pictures. As a special case, when the current picture is temporally located between the two reference pictures and the temporal distance from the current picture to the two reference pictures is the same, bilateral matching becomes a mirror-based bidirectional MV.

[0062] In current VVC, both BDOF and DMVR can be used in conjunction with CIIP mode.

[0063] Figure 6 601 and 602, respectively. The flowchart illustrates an illustrative workflow of a CIIP design using bidirectional optical flow (BDOF) in current VVC. In this workflow, L0 motion compensation and L1 motion compensation (601 and 602) are processed by BDOF (603), and the output from BDOF is then processed together with intra prediction (604) to form a weighted average (605) in CIIP mode.

[0064] The current VVC now also provides similar applications that combine DMVR and CIIP modes.

[0065] In the current VVC, the CIIP mode can improve the efficiency of prediction after motion compensation. However, the present disclosure has found three problems in the current CIIP design included in the current VVC.

[0066] First, because CIIP combines samples of inter-frame prediction and intra-frame prediction, each CIIP CU needs to use its reconstructed neighboring samples to generate a prediction signal. This means that the decoding of a CIIP CU depends on the complete reconstruction of its neighboring blocks. Due to this interdependence, for practical hardware implementation, CIIP needs to be performed in the reconstruction phase where adjacent reconstructed samples become available for intra-frame prediction. Because the decoding of CUs in the reconstruction phase must be performed sequentially (i.e., one by one), the number of computational operations (such as multiplication, addition, and displacement) involved in the CIIP process cannot be too large to ensure sufficient real-time decoding throughput. In addition, in the current CIIP design in the current VVC, new inter-mode codec tools (such as BDOF and DMVR) are also involved to generate inter-frame prediction samples for the CIIP mode. In view of the additional complexity introduced by the new inter-mode codec tools, this design may seriously reduce the encoding / decoding throughput of the hardware codec when CIIP is enabled.

[0067] Second, in the current CIIP design in the current VVC, when a CIIP CU references a merge candidate that is bi-predicted, both the motion compensation prediction signals in lists L0 and L1 need to be generated. When one or more MVs are not integer precision, an additional interpolation process must be called to interpolate samples at fractional sample positions. This process not only increases the computational complexity, but also increases the memory bandwidth given that more reference samples need to be accessed from external memory, and therefore may also severely reduce the encoding / decoding throughput of the hardware codec when CIIP is enabled.

[0068] Third, in the current CIIP design in the current VVC, the intra mode of the CIIP CU and the intra mode of the intra CU are treated differently when constructing the MPM lists of their neighboring blocks. Specifically, when a current CU is encoded and decoded with the CIIP mode, its adjacent CIIP CU is regarded as intra, that is, the intra mode of the adjacent CIIP CU can be added to the MPM candidate list. However, when the current CU is encoded and decoded with the intra mode, its adjacent CIIP CU is regarded as inter, that is, the intra mode of the adjacent CIIP CU is excluded from the MPM candidate list. This non-unified design may not be optimal for the final version of the VVC standard.

[0069] The present disclosure proposes a constrained and regulated application of the CIIP model to address the above three issues.

[0070] According to the present disclosure, after identifying a CU as a candidate for applying the CIIP mode, a determination will be made as to whether the CU identified as a candidate for applying the CIIP mode is bidirectionally predicted or unidirectionally predicted, and then based on the determination, the application of the CIIP mode to the CU will be restricted.

[0071] According to an embodiment of the present disclosure, constraining the application of the CIIP mode to the CU based on the determination includes: when the CU is bidirectionally predicted, during the application of the CIIP mode, disabling the operation of one or more predefined inter-frame prediction techniques when generating inter-frame prediction samples.

[0072] In one example, the one or more predefined inter prediction techniques include BDOF and DMVR.

[0073] Figure 7is a flow chart illustrating an illustrative workflow for selectively bypassing a DMVR operation and a BDOF operation when calculating a bidirectional prediction of a current prediction block. When processing the current prediction block, a first reference picture and a second reference picture associated with the current prediction block are obtained (702), wherein the first reference picture precedes the current picture and the second reference picture follows the current picture in display order. Subsequently, a first prediction L0 is obtained based on a first motion vector MV0 from the current prediction block to a reference block in the first reference picture (703), and a second prediction L1 is obtained based on a second motion vector MV1 from the current prediction block to a reference block in the second reference picture (704). Then, a determination is made as to whether to apply a DMVR operation (705). If applied, the DMVR operation will adjust the first motion vector MV0 and the second motion vector MV1 based on the first prediction L0 and the second prediction L1, and generate an updated first prediction L0' and an updated second prediction L1' (706). Then, a second determination is made as to whether to apply a BDOF operation (707). If applied, the BDOF operation will calculate the first horizontal and vertical gradient values ​​for the prediction samples associated with the updated first prediction L0', and the second horizontal and vertical gradient values ​​associated with the updated second prediction L1' (708). Finally, based on the first prediction L0, the second prediction L1, the optional updated first prediction L0', the optional updated second prediction L1', the optional first horizontal and vertical gradient values, and the optional second horizontal and vertical gradient values, the bidirectional prediction of the current prediction block is calculated (709).

[0074] In another example, the one or more predefined inter prediction techniques include BDOF. Figure 8 804 is a flow chart illustrating an illustrative workflow of the CIIP design in this example of the present disclosure. In this workflow, L0 motion compensation and L1 motion compensation (801 and 802) are averaged (803) instead of being processed through BDOF, and the resulting average is then processed together with intra prediction (804) to form a weighted average (805) in CIIP mode.

[0075] In a third example, the one or more predefined inter prediction techniques include DMVR.

[0076] According to another embodiment of the present disclosure, constraining the application of the CIIP mode to a CU based on determination includes: using predefined criteria, selecting multiple unidirectional prediction samples from all available unidirectional prediction samples for use in combination with intra-frame prediction samples for the CU during application of the CIIP mode to the CU when the CU is bidirectionally predicted.

[0077] In one example, the predefined criteria includes selecting all uni-directional prediction samples based on pictures in reference picture list 0 (L0).

[0078] In another example, the predefined criteria includes selecting all uni-directional prediction samples based on pictures in reference picture list 1 (L1).

[0079] In a third example, the predefined criteria include: selecting all unidirectional prediction samples based on a reference picture that has a minimum picture order count (POC) distance to the picture where the CU is located ("current picture"). Fig. 9 is a flow chart illustrating an illustrative workflow of the CIIP design proposed in this example of the present disclosure. In the workflow, a determination is made as to whether the L0 reference picture or the L1 reference picture is closer to the current picture in terms of POC distance (901), and motion compensation from a reference picture having the smallest POC distance to the current picture is selected (902 and 903), and then processed together with intra prediction (904) to form a weighted average (905) in CIIP mode.

[0080] According to another embodiment of the present disclosure, constraining the application of CIIP mode to a CU based on determination includes: when the CU is bidirectionally predicted, disabling the application of CIIP mode to the CU, and disabling the signaling of the CIIP flag for the CU. Specifically, in order to reduce overhead, the signaling of the CIIP enable / disable flag depends on the prediction direction of the current CIIP CU. If the current CU is unidirectionally predicted, the CIIP flag will be signaled in the bitstream to indicate whether CIIP is enabled or disabled. Otherwise (i.e., the current CU is bidirectionally predicted), the signaling of the CIIP flag will be skipped and always inferred to be false, i.e., CIIP is always disabled.

[0081] In addition, according to the present disclosure, when forming an MPM candidate list for a CU ("current CU"), a determination will be made as to whether each CU in the neighboring CUs of the current CU is CIIP encoded respectively, and then for each neighboring CU in the neighboring CUs that is CIIP encoded, a unified criterion will be adopted in the process of forming the MPM candidate list for the current CU using the intra-frame mode of the neighboring CU, which unified criterion does not depend on the determination of whether the current CU is intra-frame encoded or CIIP encoded.

[0082] According to one embodiment of this aspect of the present disclosure, the unified criterion includes: when the neighboring CU is encoded and decoded by CIIP, the intra mode of the neighboring CU is considered unavailable for forming the MPM candidate list.

[0083] According to another embodiment of this aspect of the present disclosure, the unified criterion includes: when the neighboring CU is CIIP coded, considering the CIIP mode of the neighboring CU to be equivalent to the intra mode in the formation of the MPM candidate list.

[0084] Figure 10A-10Bare a pair of flow charts setting forth illustrative workflows for two embodiments of this aspect of the disclosure. Fig. 10A The workflow of the first embodiment of this aspect of the present disclosure is illustrated. In this workflow, a determination is first made as to whether the neighboring CU is an intra block or a CIIP block (1001), and if and only if it is an intra block or a CIIP block, its intra mode is added to the MPM candidate list of the current CU (1002), regardless of whether the current CU is an intra block or a CIIP block. Fig. 10B The workflow of the second embodiment of this aspect of the present disclosure is illustrated. In this workflow, a determination is first made as to whether the neighboring CU is an intra block (1003), and if and only if it is an intra block, its intra mode is added to the MPM candidate list of the current CU (1004), regardless of whether the current CU is an intra block or a CIIP block.

[0085] In one or more examples, the functions described may be implemented in hardware, software, firmware, or any combination thereof. If implemented in software, the functions may be stored on or transmitted via a computer-readable medium as one or more instructions or codes and executed by a hardware-based processing unit. A computer-readable medium may include a computer-readable storage medium or a communication medium, where the computer-readable storage medium corresponds to a tangible medium such as a data storage medium, and the communication medium includes any medium that facilitates the transfer of a computer program (e.g., according to a communication protocol) from one place to another. In this manner, a computer-readable medium may generally correspond to (1) a non-temporary tangible computer-readable storage medium or (2) a communication medium such as a signal or carrier wave. The data storage medium may be any available medium that can be accessed by one or more computers or one or more processors to retrieve instructions, codes, and / or data structures to implement the embodiments described in the present application. A computer program product may include a computer-readable medium.

[0086] In addition, the above method can be implemented using an apparatus including one or more circuits, including an application specific integrated circuit (ASIC), a digital signal processor (DSP), a digital signal processing device (DSPD), a programmable logic device (PLD), a field programmable gate array (FPGA), a controller, a microcontroller, a microprocessor or other electronic components. The apparatus can use circuits combined with other hardware or software components to perform the methods described above. Each module, submodule, unit or subunit disclosed above can be implemented at least in part using one or more circuits.

[0087] Other embodiments of the present invention will be apparent to those skilled in the art in view of the description and practice of the invention disclosed herein. This application is intended to cover any variation, use or adaptation of the invention in accordance with its general principles, and includes deviations from the present disclosure in known or customary practice in the art. It is intended that the description and examples be considered exemplary only, with the true scope and spirit of the invention being indicated by the following claims.

[0088] It will be understood that the present invention is not limited to the exact examples described above and illustrated in the accompanying drawings and that various modifications and changes may be made without departing from the scope thereof. It is intended that the scope of the present invention be limited only by the appended claims.

Claims

1. A video decoding method, comprising: Obtaining a first reference picture and a second reference picture associated with a current block, wherein the first reference picture precedes the current picture and the second reference picture follows the current picture in display order; In response to a combined inter and intra prediction (CIIP) flag for the current block indicating that CIIP is not applied to the current block, determining to apply a decoder-side motion vector refinement (DMVR) operation to the current block; In response to a combined inter and intra prediction (CIIP) flag for the current block indicating that CIIP is not applied to the current block, determining to apply a bidirectional optical flow (BDOF) operation to the current block; A bidirectional prediction of the current block is calculated based on the first reference picture, the second reference picture, a determination to apply the DMVR operation, and a determination to apply the BDOF operation.

2. The method according to claim 1, further comprising: adjusting the first motion vector MV0 and the second motion vector MV1 to generate a first prediction L0' and a second prediction L1', wherein the first motion vector MV0 is from the current block to a reference block in the first reference picture preceding the current picture in display order, and The second motion vector MV1 is from the current block to a reference block in the second reference picture that follows the current picture in display order.

3. The method according to claim 2, further comprising: Calculating a first horizontal gradient value and a first vertical gradient value associated with the first prediction L0'; as well as A second horizontal gradient value and a second vertical gradient value associated with the second prediction L1 ′ are calculated.

4. The method according to claim 3, wherein the calculating the bidirectional prediction of the current block based on the first reference picture, the second reference picture, and the determination to apply the DMVR operation and the determination to apply the BDOF operation comprises: A bidirectional prediction of the current block is calculated based on the first prediction L0', the second prediction L1', the first horizontal gradient value and the first vertical gradient value, and the second horizontal gradient value and the second vertical gradient value.

5. The method according to claim 1, further comprising: In response to the current block being bi-directionally predicted, it is determined that the CIIP is not applied to the current block.

6. A computing device comprising: Storage media; as well as One or more processors coupled to the storage medium, wherein the one or more processors are configured to execute the video decoding method according to any one of claims 1-5.

7. A non-transitory computer-readable storage medium storing a plurality of programs executed by a computing device having one or more processors, wherein when executed by the one or more processors, the plurality of programs cause the one or more processors to perform the method of any one of claims 1-5 and store a bit stream to be decoded by the video decoding method of any one of claims 1-5.

8. A method for processing a bit stream, comprising: Receive a bit stream; as well as The bit stream is decoded according to the video decoding method according to any one of claims 1-5.

Images