Video encoding and decoding method, device and non-transitory computer-readable storage medium

Abstract

The present disclosure relates to a method for video coding and decoding. The method includes acquiring a first reference picture and a second reference picture associated with a current prediction block, acquiring a first prediction L0 based on a first motion vector MV0 from the current prediction block to a reference block in the first reference picture, acquiring a second prediction L1 based on a second motion vector MV1 from the current prediction block to a reference block in the second reference picture, determining whether to apply a bidirectional optical flow (BDOF) operation, and calculating a bidirectional prediction of the current prediction block based on the first prediction L0 and the second prediction L1 and a first gradient value and a second gradient value.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application is based on and claims priority from Provisional Application No. 62 / 790421 filed on January 9, 2019, the entire contents of which are incorporated herein by reference. Technical Field

[0003] The application relates to video coding and compression. More specifically, the application relates to a method and apparatus for a joint inter-intra prediction (CIIP) method for video coding. Background Art

[0004] Various video codec techniques can be used to compress video data. Video coding is performed according to one or more video codec standards. For example, video codec standards include Versatile Video Codec (VVC), Joint Exploration Test Model (JEM), High Efficiency Video Codec (H.265 / HEVC), Advanced Video Codec (H.264 / AVC), Moving Picture Experts Group (MPEG) codec, etc. Video codecs typically use prediction methods (e.g., inter-frame prediction, intra-frame prediction, etc.) that exploit redundancy present in video pictures or sequences. An important goal of video coding techniques is to compress video data into a form that uses a lower bit rate while avoiding or minimizing the degradation of video quality. Summary of the invention

[0005] Examples of the present disclosure provide methods for improving the efficiency of semantic signaling for merging related modes.

[0006] According to a first aspect of the present disclosure, a video encoding and decoding method includes: obtaining a first reference picture and a second reference picture associated with a current prediction block, wherein, in display order, the first reference picture is before the current picture, and the second reference picture is after the current picture, obtaining a first prediction L0 based on a first motion vector MV0 from the current prediction block to a reference block in the first reference picture, obtaining a second prediction L1 based on a second motion vector MV1 from the current prediction block to a reference block in the second reference picture, determining whether to apply a bidirectional optical flow (BDOF) operation, wherein the BDOF calculates first horizontal and vertical gradient values ​​of prediction samples associated with the first prediction L0 and and second horizontal and vertical gradient values ​​associated with the second prediction L1 and And based on the first prediction L0 and the second prediction L1 and the first gradient value and With the second gradient value and Computes the bidirectional prediction for the current prediction block.

[0007] According to a second aspect of the present disclosure, a video encoding and decoding method includes: obtaining a reference picture in a reference picture list associated with a current prediction block, generating an inter-frame prediction based on a first motion vector from the current picture to a first reference picture, obtaining an intra-frame prediction mode associated with the current prediction block, generating an intra-frame prediction of the current prediction block based on the intra-frame prediction, generating a final prediction of the current prediction block by averaging the inter-frame prediction and the intra-frame prediction, and for an intra-frame mode prediction based on a most probable mode (MPM), determining whether the current prediction block is regarded as an inter-frame mode or an intra-frame mode.

[0008] According to a third aspect of the present disclosure, a non-transitory computer-readable storage medium having instructions stored therein is provided. When executed by one or more processors, the computing device is caused to perform operations including: obtaining a first reference picture and a second reference picture associated with a current prediction block, wherein, in display order, the first reference picture is before the current picture and the second reference picture is after the current picture, obtaining a first prediction L0 based on a first motion vector MV0 from the current prediction block to a reference block in the first reference picture, obtaining a second prediction L1 based on a second motion vector MV1 from the current prediction block to a reference block in the second reference picture, determining whether to apply a bidirectional optical flow (BDOF) operation, wherein the BDOF calculates first horizontal and vertical gradient values ​​of prediction samples associated with the first prediction L0 and and second horizontal and vertical gradient values ​​associated with the second prediction L1 and And calculate the bidirectional prediction of the current prediction block.

[0009] According to a fourth aspect of the present disclosure, a non-temporary computer-readable storage medium having instructions stored therein is provided. When executed by one or more processors, the computing device is caused to perform operations including: obtaining a reference picture in a reference picture list associated with a current prediction block, generating an inter-frame prediction based on a first motion vector from the current picture to a first reference picture, obtaining an intra-frame prediction mode associated with the current prediction block, generating an intra-frame prediction of the current prediction block based on the intra-frame prediction, generating a final prediction of the current prediction block by averaging the inter-frame prediction and the intra-frame prediction, and for an intra-frame mode prediction based on a most probable mode (MPM), determining whether the current prediction block is regarded as an inter-frame mode or an intra-frame mode.

[0010] It is to be understood that both the foregoing general description and the following detailed description are exemplary only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate examples consistent with the disclosure and, together with the description, serve to explain the principles of the disclosure.

[0012] Figure 1 is a block diagram of an encoder according to an example of the present disclosure.

[0013] Figure 2 is a block diagram of a decoder according to an example of the present disclosure.

[0014] Figure 3 is a flow chart illustrating a method for generating an inter-intra joint prediction (CIIP) according to an example of the present disclosure.

[0015] Figure 4 is a flow chart illustrating a method for generating a CIIP according to an example of the present disclosure.

[0016] Figure 5A is a diagram illustrating block partitioning in a multi-type tree structure according to an example of the present disclosure.

[0017] Figure 5B is a diagram illustrating block partitioning in a multi-type tree structure according to an example of the present disclosure.

[0018] Figure 5C is a diagram illustrating block partitioning in a multi-type tree structure according to an example of the present disclosure.

[0019] Figure 5D is a diagram illustrating block partitioning in a multi-type tree structure according to an example of the present disclosure.

[0020] Figure 5E is a diagram illustrating block partitioning in a multi-type tree structure according to an example of the present disclosure.

[0021] Fig. 6A is a diagram illustrating inter-intra joint prediction (CIIP) according to an example of the present disclosure.

[0022] Figure 6B is a diagram illustrating inter-intra joint prediction (CIIP) according to an example of the present disclosure.

[0023] Figure 6C is a diagram illustrating inter-intra joint prediction (CIIP) according to an example of the present disclosure.

[0024] Fig. 7A is a flowchart of an MPM candidate list generation process according to an example of the present disclosure.

[0025] Figure 7B is a flowchart of an MPM candidate list generation process according to an example of the present disclosure.

[0026] Figure 8 is a diagram illustrating a workflow of an existing CIIP design in VVC according to an example of the present disclosure.

[0027] Fig. 9 is a diagram showing a workflow of a CIIP method proposed by removing BDOF according to an example of the present disclosure.

[0028] Fig.10 is a diagram illustrating a workflow of a unidirectional prediction based CIIP according to an example of the present disclosure, where a prediction list is selected based on a POC distance.

[0029] Fig.11A is a flowchart of a method when a CIIP block is enabled for MPM candidate list generation according to an example of the present disclosure.

[0030] Fig. 11B is a flowchart of a method when disabling CIIP blocks for MPM candidate list generation according to an example of the present disclosure.

[0031] Fig.12 is a diagram showing a computing environment coupled with a user interface according to an example of the present disclosure. DETAILED DESCRIPTION

[0032] Reference will now be made in detail to examples of the present disclosure, some of which are illustrated in the accompanying drawings. The following description refers to the accompanying drawings, in which the same numbers in different drawings represent the same or similar elements, unless otherwise indicated. The embodiments set forth in the following description of examples of the present disclosure do not represent all embodiments consistent with the present disclosure. Instead, they are merely examples of devices and methods consistent with aspects related to the disclosure recorded in the appended claims.

[0033] The terms used in this disclosure are only used for the purpose of describing specific embodiments and are not intended to limit the disclosure. As used in this disclosure and the appended claims, the singular forms "a", "an", and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. It should also be understood that the term "and / or" used herein is intended to indicate and include any or all possible combinations of one or more of the associated listed items.

[0034] It should be understood that although the terms "first", "second", "third", etc. may be used herein to describe various information, the information should not be limited by these terms. These terms are only used to distinguish one type of information from another type of information. For example, without departing from the scope of the present disclosure, the first information may be referred to as the second information; and similarly, the second information may also be referred to as the first information. As used herein, the term "if" may be understood to mean "when" or "at" or "in response to a judgment" depending on the context.

[0035] The first version of the HEVC standard was finalized in October 2013, and it provides about 50% bitrate savings or equivalent perceptual quality compared to the previous generation video codec standard H.264 / MPEG AVC. Although the HEVC standard provides significant codec improvements over its predecessor, there is evidence that higher codec efficiency than HEVC can be achieved using additional codec tools. Based on this, both VCEG and MPEG have started exploration work on new codec technologies for future video codec standardization. In October 2015, ITU-T VECG and ISO / IEC MPEG established a Joint Video Exploration Team (JVET) to start important research on advanced technologies that can significantly improve codec efficiency. By integrating several additional codec tools on top of the HEVC Test Model (HM), JVET maintains a reference software called the Joint Exploration Model (JEM).

[0036] In October 2017, ITU-T and ISO / IEC issued a joint call for proposals (CfP) on video compression with capabilities beyond HEVC. In April 2018, at the 10th JVET meeting, 23 CfP responses were received and evaluated, showing a compression efficiency gain of about 40% over HEVC. Based on such evaluation results, JVET launched a new project to develop a new generation of video codec standards, which was named Versatile Video Codec (VVC). In the same month, a reference software code base, called the VVC Test Model (VTM), was established to demonstrate a reference implementation of the VVC standard.

[0037] Like HEVC, VVC is built on a block-based hybrid video codec framework. Figure 1 (described below) A block diagram of a general block-based hybrid video codec system is given. The input video signal is processed block by block, called coding unit (CU). In VTM-1.0, a CU can be up to 128×128 pixels. However, unlike HEVC, which partitions blocks based only on quadtrees, in VVC, a coding tree unit (CTU) is partitioned into multiple CUs based on quadtrees / binary / ternary trees to adapt to changing local characteristics. In addition, the concept of multiple partition unit types in HEVC is removed, that is, there is no distinction between CU, prediction unit (PU) and transform unit (TU) in VVC; instead, each CU is always used as a basic unit for both prediction and transform without further partitioning. In the multi-type tree structure, a CTU is first partitioned by a quadtree structure. Then, each quadtree leaf node can be further divided by binary and ternary tree structures. As Figure 5A , Figure 5B , Figure 5C, Figure 5D and Figure 5E As shown in (described below), there are five types of partitioning, namely, quadrifugal partitioning, horizontal binary partitioning, vertical binary partitioning, horizontal trifugal partitioning, and vertical trifugal partitioning.

[0038] exist Figure 1 In the video code (described below), spatial prediction and / or temporal prediction can be performed. Spatial prediction (or "intra-frame prediction") uses pixels from samples of neighboring blocks that have been encoded in the same video picture / slice (called reference samples) to predict the current video block. Spatial prediction reduces the spatial redundancy inherent in the video signal. Temporal prediction (also known as "inter-frame prediction" or "motion compensated prediction") uses reconstructed pixels from an already encoded video picture to predict the current video block. Temporal prediction reduces the temporal redundancy inherent in the video signal. The temporal prediction signal for a given CU is usually signaled by one or more motion vectors (MVs) that indicate the amount and direction of motion between the current CU and its temporal reference. In addition, when multiple reference pictures are supported, a reference picture index is additionally sent, which is used to identify which reference picture in the reference picture library the temporal prediction signal comes from. After spatial and / or temporal prediction, the mode decision block in the encoder selects the best prediction mode, for example based on a rate-distortion optimization method. The prediction block is then subtracted from the current video block; and the prediction residual is decorrelated using transformation and quantization.

[0039] The quantized residual coefficients are inverse quantized and inverse transformed to form a reconstructed residual, which is then added back to the prediction block to form the reconstructed signal of the CU. Further in-loop filtering, such as deblocking filtering, sample adaptive offset (SAO), and adaptive in-loop filtering (ALF), may be applied to the reconstructed CU before it is placed into a reference picture library and used to encode future video blocks. To form an output video bitstream, the coding mode (inter or intra), prediction mode information, motion information, and quantized residual coefficients are all sent to the entropy coding unit to be further compressed and packaged to form a bitstream.

[0040] Figure 2 (described below) gives a general block diagram of a block-based video decoder. The video bitstream is first entropy decoded in an entropy decoding unit. The coding mode and prediction information are sent to the spatial prediction unit (when intra-frame coding) or the temporal prediction unit (when inter-frame coding) to form a prediction block. The residual transform coefficients are sent to the inverse quantization unit and the inverse transform unit to reconstruct the residual block. The prediction block and the residual block are then added together. The reconstructed block can be further subjected to in-loop filtering before being stored in a reference picture library. The reconstructed video in the reference picture library is then sent out to drive the display device and to predict future video blocks.

[0041] Figure 1 A typical encoder 100 is shown. The encoder 100 has a video input 110, motion compensation 112, motion estimation 114, intra / inter mode decision 116, block predictor 140, adder 128, transform 130, quantization 132, prediction related information 142, intra prediction 118, picture buffer 120, inverse quantization 134, inverse transform 136, adder 126, memory 124, in-loop filter 122, entropy coding 138, and bitstream 144.

[0042] Figure 2 A block diagram of a typical decoder 200 is shown. The decoder 200 has a bitstream 210, entropy decoding 212, inverse quantization 214, inverse transform 216, adder 218, intra / inter mode selection 220, intra prediction 222, memory 230, in-loop filter 228, motion compensation 224, picture buffer 226, prediction related information 234 and video output 232.

[0043] Figure 3 An example method 300 for generating inter-intra joint prediction (CIIP) according to the present disclosure is shown.

[0044] In step 310 , a first reference picture and a second reference picture associated with the current prediction block are obtained, wherein, in display order, the first reference picture is before the current picture, and the second reference picture is after the current picture.

[0045] At step 312 , a first prediction L0 is obtained based on a first motion vector MV0 from the current prediction block to a reference block in a first reference picture.

[0046] At step 314, a second prediction L1 is obtained based on a second motion vector MV1 from the current prediction block to a reference block in a second reference picture.

[0047] At step 316, it is determined whether to apply a bidirectional optical flow (BDOF) operation, wherein the BDOF calculates first horizontal and vertical gradient values ​​of the prediction sample associated with the first prediction L0 and second horizontal and vertical gradient values ​​associated with the second prediction L1. For example, the BDOF calculates the first horizontal and vertical gradient values ​​of the prediction sample associated with the first prediction L0. and and second horizontal and vertical gradient values ​​associated with the second prediction L1 and

[0048] In step 318, a bidirectional prediction of the current prediction block is calculated based on the first prediction L0 and the second prediction L1 and the first gradient value and the second gradient value. and and the second gradient value and

[0049] Figure 4 An example method for generating CIIP according to the present disclosure is shown. For example, the method includes inter-frame prediction based on unidirectional prediction and intra-frame prediction based on MPM for generating CIIP.

[0050] In step 410, a reference picture in a reference picture list associated with the current prediction block is obtained.

[0051] At step 412, an inter-frame prediction is generated based on a first motion vector from the current picture to a first reference picture.

[0052] At step 414, the intra prediction mode associated with the current prediction block is obtained.

[0053] At step 416, an intra prediction of the current prediction block is generated based on the intra prediction.

[0054] At step 418, a final prediction of the current prediction block is generated by averaging the inter prediction and the intra prediction.

[0055] At step 420, for intra-mode prediction based on the most probable mode (MPM), it is determined whether the current prediction block is considered as an inter mode or an intra mode.

[0056] Figure 5A A diagram illustrating block quad partitioning in a multi-type tree structure according to an example of the present disclosure is shown.

[0057] Figure 5B A diagram illustrating vertical binary partitioning of blocks in a multi-type tree structure according to an example of the present disclosure is shown.

[0058] Figure 5C A diagram illustrating block-level binary partitioning in a multi-type tree structure according to an example of the present disclosure is shown.

[0059] Figure 5D A diagram illustrating vertical ternary partitioning of blocks in a multi-type tree structure according to an example of the present disclosure is shown.

[0060] Figure 5E A diagram illustrating block-level ternary partitioning in a multi-type tree structure according to an example of the present disclosure is shown.

[0061] Inter-frame and intra-frame joint prediction

[0062] like Figure 1 and Figure 2As shown, inter-frame and intra-frame prediction methods are used in a hybrid video coding scheme, where each PU is only allowed to select inter-frame prediction or intra-frame prediction to exploit correlations in the temporal or spatial domain, and never both. However, as pointed out in previous literature, the residual signals generated by the inter-frame prediction block and the intra-frame prediction block may exhibit very different characteristics from each other. Therefore, when the two predictions can be combined in an effective manner, a more accurate prediction can be expected to reduce the energy of the prediction residual and thus improve the coding efficiency. In addition, in natural video content, the motion of moving objects may be complex. For example, there may be areas that contain both old content (e.g., objects included in previously encoded pictures) and new content that appears (e.g., objects not included in previously encoded pictures). In this case, neither inter-frame prediction nor intra-frame prediction can provide an accurate prediction of the current block.

[0063] In order to further improve the prediction efficiency, inter-intra joint prediction (CIIP) is adopted in the VVC standard, which combines the inter prediction and intra prediction of a CU encoded by the merge mode. Specifically, for each merged CU, an additional flag is sent through signaling to indicate whether CIIP is enabled for the current CU. For the luminance component, CIIP supports four commonly used intra modes, including planar prediction (PLANAR), directional angle prediction (DC), horizontal prediction (HORIZONAL), and vertical prediction (VERTICAL) modes. For chrominance components, DM (i.e., chrominance reuses the same intra mode of the luminance component) is always applied without additional signaling. In addition, in the existing CIIP design, weighted averaging is applied to combine the inter prediction samples and intra prediction samples of a CIIP CU. Specifically, when PLANAR or DC mode is selected, equal weights (i.e., 0.5) are applied. Otherwise (ie, HORIZONAL or VERTICAL mode is applied), the current CU is first split into four equal-sized regions horizontally (for HORIZONAL mode) or vertically (for VERTICAL mode).

[0064] Denote the application as (w_intra i ,w_inter i) to combine inter- and intra-prediction samples in different regions, where i=0 and i=3 represent the regions closest and farthest from the reconstructed neighboring samples for intra-prediction. In the current CIIP design, the values ​​of the weight sets are set to (w_intra0, w_inter0)=(0.75, 0.25), (w_intra1, w_inter1)=(0.625, 0.37.5), (w_intra2, w_inter2)=(0.375, 0.625), and (w_intra3, w_inter3)=(0.25, 0.75). Fig. 6A , Figure 6B and Figure 6C An example (described below) is provided to illustrate the CIIP mode.

[0065] In addition, in the current VVC working specification, the intra mode of a CIIP CU can be used as a predictor to predict the intra mode of its neighboring CIIP CU through the most probable mode (MPM) mechanism. Specifically, for each CIIP CU, when its neighboring blocks are also CIIP CUs, the intra modes of those neighboring blocks are first rounded to the closest mode among the PLANAR, DC, HORIZONAL, and VERTICAL modes, and then added to the MPM candidate list of the current CU. However, when constructing the MPM list for each intra CU, when one of its neighboring blocks is encoded by the CIIP mode, the neighboring block is considered unavailable, that is, the intra mode of a CIIP CU is not allowed to be used to predict the intra mode of its neighboring intra CU. Fig. 7A and Figure 7B (described below) The MPM list generation process for intra CU and CIIP CU is compared.

[0066] Bidirectional optical flow

[0067] Traditional bidirectional prediction in video codecs is a simple combination of two temporal prediction blocks obtained from an already reconstructed reference picture. However, due to the limitations of block-based motion compensation, residual small motion can be observed between the samples of the two prediction blocks, thus reducing the efficiency of motion compensated prediction. To address this problem, bidirectional optical flow (BDOF) is applied in VVC to reduce the impact of this motion on each sample within a block.

[0068] Specifically, Fig. 6A , Figure 6B and Figure 6CAs shown in Figure 1 (described below), BDOF is a sample-by-sample motion refinement performed on top of block-based motion compensated prediction when using bidirectional prediction. After applying BDOF within a 6×6 window Ω around the sub-block, the motion refinement (v) of each 4×4 sub-block is calculated by minimizing the difference between the L0 and L1 prediction samples. x , v y ). Specifically, (v x , v y ) is derived as follows:

[0069]

[0070]

[0071] in is the floor function; clip3(min, max, x) is the function that clips the given value x within the range [min, max]; the symbol >> represents a bitwise right shift operation; the symbol << represents a bitwise left shift operation; th BDOF is the motion refinement threshold to prevent propagation errors caused by irregular local motion, which is equal to 2 13-BD , where BD is the bit depth of the input video. In (1),

[0072] The values ​​of S1, S2, S3, S5, and S6 are calculated as follows:

[0073] S1=∑ (i，j)∈Ω ψ x (i, j)·ψ x (i, j), S3 = ∑ (i，j)∈Ω θ(i, j)·ψ x (i, j),

[0074] S2=∑ (i，j)∈Ω ψ x (i, j)·ψ y (i, j), S5 = ∑ (i，j)∈Ω ψ y (i, j)·ψ y (i, j),

[0075] S6=∑ (i，j)∈Ω θ(i, j)·ψ y (i, j)

[0076] in,

[0077]

[0078]

[0079] θ(i, j)=(I(1) (i, j)>>6)-(I (0) (i, j)>>6)

[0080] Among them, I (k) (i, j) is the sample value at coordinate (i, j) of the prediction signal in list k (k=0, 1), which is generated with medium-high precision (i.e., 16 bits); and The horizontal and vertical gradient values ​​of a sample point are obtained by directly calculating the difference between its two adjacent sample points, that is,

[0081]

[0082]

[0083] Based on the motion refinement derived in (1), the final bidirectional prediction samples of the CU are calculated by interpolating the L0 / L1 prediction samples along the motion trajectory based on the optical flow model, as indicated by the following equation:

[0084] pred BDOF (x, y) = (I (0) (x,y)+I (1) (x, y)+b+o offset )>>shift

[0085]

[0086] Among them, shift and o offset are the right shift value and offset value, which are applied to combine the L0 and L1 prediction signals for bidirectional prediction, and are equal to 15-BD and 1<<(14-BD)+2·(1<<13), respectively.

[0087] Fig. 6A A diagram showing inter-intra joint prediction for horizontal mode is shown according to an example of the present disclosure.

[0088] Figure 6B A diagram showing inter-intra joint prediction for VERTICAL mode is shown according to an example of the present disclosure.

[0089] Figure 6C A diagram showing inter-intra joint prediction for PLANAR and DC modes is shown according to an example of the present disclosure.

[0090] Fig. 7A A flowchart of a process of generating an MPM candidate list for an intra CU according to an example of the present disclosure is shown.

[0091] Figure 7BA flowchart of a process of generating an MPM candidate list for a CIIP CU according to an example of the present disclosure is shown.

[0092] Improvements to CIIP

[0093] Although CIIP can improve the efficiency of conventional motion compensated prediction, its design can still be further improved. Specifically, the following problems in the existing CIIP design in VVC are identified in this disclosure.

[0094] First, as discussed in the "Inter-Intra Joint Prediction" section, because CIIP combines samples for inter- and intra-frame prediction, each CIIP CU needs to use its reconstructed neighboring samples to generate a prediction signal. This means that the decoding of a CIIP CU depends on the complete reconstruction of its neighboring blocks. Due to this interdependency, for a practical hardware implementation, CIIP needs to be performed in the reconstruction phase when neighboring reconstructed samples become available for intra-frame prediction. Because the decoding of CUs in the reconstruction phase must be performed sequentially (i.e., one after another), in order to ensure sufficient throughput for real-time decoding, the number of computational operations (e.g., multiplication, addition, and bit shifting) involved in the CIIP process cannot be too high.

[0095] As mentioned in the "Bidirectional Optical Flow" section, when predicting an inter-coded CU from two reference blocks in the forward and backward temporal directions, BDOF is enabled to improve the prediction quality. Figure 8 As shown (described below), in the current VVC, BDOF is also involved to generate inter-frame prediction samples for CIIP mode. Considering the additional complexity brought by BDOF, this design may seriously reduce the encoding / decoding throughput of the hardware codec when CIIP is enabled.

[0096] Second, in the current CIIP design, when a CIIP CU references a bi-predicted merge candidate, it is necessary to generate motion compensation prediction signals in lists L0 and L1. When one or more MVs are not integer precision, an additional interpolation process must be called to interpolate samples at fractional sample positions. This process not only increases computational complexity, but also increases storage bandwidth because more reference samples need to be accessed from external memory.

[0097] Third, as discussed in the "Inter-Intra Joint Prediction" section, in the current CIIP design, the intra mode of the CIIP CU and the intra mode of the intra CU are treated differently when constructing the MPM list of its neighboring blocks. Specifically, when a current CU is encoded by the CIIP mode, its neighboring CIIP CU is considered to be intra, that is, the intra mode of the neighboring CIIP CU can be added to the MPM candidate list. However, when the current CU is encoded by the intra mode, its neighboring CIIP CU is considered to be inter, that is, the intra mode of the neighboring CIIP CU is not included in the MPM candidate list. This non-uniform design may not be optimal for the final version of the VVC standard.

[0098] Figure 8 A diagram illustrating a workflow of an existing CIIP design in VVC according to an example of the present disclosure is shown.

[0099] Simplifying CIIP

[0100] In the present disclosure, methods are provided to simplify existing CIIP designs to facilitate hardware codec implementations. In general, the main aspects of the techniques presented in the present disclosure are summarized as follows.

[0101] First, in order to improve CIIP encoding / decoding throughput, it is proposed to exclude BDOF from the generation of inter-frame prediction samples in CIIP mode.

[0102] Secondly, in order to reduce computational complexity and storage bandwidth consumption, when a CIIP CU is bidirectionally predicted (ie, has L0 MV and L1 MV), a method is proposed to convert the block from bidirectional prediction to unidirectional prediction to generate inter-frame prediction samples.

[0103] Third, two methods are proposed to coordinate the intra modes of CIIP CUs and intra CUs when forming MPM candidates for neighboring blocks of a CU.

[0104] CIIP without BDOF

[0105] As pointed out in the “Problem Statement” section, BDOF is always enabled to generate inter-frame prediction samples for CIIP mode when the current CU is bidirectionally predicted. Due to the additional complexity of BDOF, existing CIIP designs may significantly reduce encoding / decoding throughput, especially making real-time decoding difficult for VVC decoders. On the other hand, for CIIP CUs, their final prediction samples are generated by averaging inter-frame prediction samples and intra-frame prediction samples. In other words, the prediction samples refined by BDOF will not be directly used as prediction signals for CIIP CUs. Therefore, compared with conventional bidirectionally predicted CUs (where BDOF is directly applied to generate prediction samples), the corresponding improvements obtained from BDOF are relatively inefficient for CIIP CUs. Therefore, based on the above considerations, it is recommended to disable BDOF when generating inter-frame prediction samples for CIIP mode. Fig. 9 (described below) shows the corresponding workflow of the proposed CIIP process after removing the BDOF.

[0106] Fig. 9 A diagram showing the workflow of the proposed CIIP method by removing BDOF according to an example of the present disclosure is shown.

[0107] CIIP based on one-way prediction

[0108] As discussed above, when a merge candidate referenced by a CIIP CU is bidirectionally predicted, both L0 and L1 prediction signals are generated to predict samples within the CU. In order to reduce storage bandwidth and interpolation complexity, in one embodiment of the present disclosure, it is recommended to use only inter-frame prediction samples generated using unidirectional prediction (even when the current CU is bidirectionally predicted) in CIIP mode to combine with intra-frame prediction samples. Specifically, when the current CIIP CU is unidirectionally predicted, the inter-frame prediction samples will be directly combined with the intra-frame prediction samples. Otherwise (i.e., the current CU is bidirectionally predicted), the inter-frame prediction samples used by CIIP are generated based on unidirectional prediction from one prediction list (L0 or L1). In order to select a prediction list, different methods can be applied. In the first method, it is recommended to always select the first prediction (i.e., list L0) for any CIIP block predicted by two reference pictures.

[0109] In the second method, it is proposed to always select the second prediction (i.e., list L1) for any CIIP block predicted by two reference pictures. In the third method, an adaptive method is applied in which the prediction list associated with a reference picture having a smaller picture order count (POC) distance to the current picture is selected. Fig.10 (described below) shows the workflow of CIIP based on unidirectional prediction, where the prediction list is selected based on POC distance.

[0110] Finally, in the last method, it is recommended to enable CIIP mode only when the current CU is unidirectionally predicted. In addition, in order to reduce overhead, the signaling of the CIIP enable / disable flag depends on the prediction direction of the current CIIP CU. When the current CU is unidirectionally predicted, the CIIP flag will be signaled in the bitstream to indicate whether CIIP is enabled or disabled. Otherwise (i.e., the current CU is bidirectionally predicted), the signaling of the CIIP flag will be skipped and will always be inferred to be false, i.e., CIIP is always disabled.

[0111] Fig.10 A diagram showing a workflow of unidirectional prediction based CIIP for selecting a prediction list based on POC distance according to one example of the present disclosure is shown.

[0112] Coordination of intra modes for CIIP CUs and intra CUs for MPM candidate list construction

[0113] As discussed above, the current CIIP design is not unified in how to use the intra modes of CIIP CUs and intra CUs to form the MPM candidate list of their neighboring blocks. Specifically, the intra modes of both CIIP CUs and intra CUs can predict the intra modes of their neighboring blocks encoded in CIIP mode. However, only the intra mode of an intra CU can predict the intra mode of an intra CU. To achieve a more unified design, this section proposes two methods to coordinate the use of intra modes of CIIP CUs and intra CUs in MPM list construction.

[0114] In the first method, for MPM list construction, it is recommended to regard the CIIP mode as an inter-frame mode. Specifically, when generating an MPM list for a CIIP CU or an intra-frame CU, when its neighboring blocks are encoded in CIIP mode, the intra-frame mode of the neighboring block is marked as unavailable. With this method, intra-frame modes without CIIP blocks can be used to construct the MPM list. In contrast, in the second method, for MPM list construction, it is recommended to regard the CIIP mode as an intra-frame mode. Specifically, in this method, the intra-frame mode of the CIIP CU can predict the intra-frame modes of its neighboring CIIP blocks and intra-frame blocks. Fig.11A and Fig. 11B (described below) shows the MPM candidate list generation process when the above two methods are applied.

[0115] Other embodiments of the present disclosure will be apparent to those skilled in the art in view of the specification and practice of the present disclosure disclosed herein. This application is intended to cover any changes, uses or modifications of the present disclosure that follow the general principles of the present disclosure and include deviations from the present disclosure that fall within known or customary practices in the art. This specification and examples are intended to be considered as examples only, and the true scope and spirit of the present disclosure are indicated by the claims that follow.

[0116] It should be understood that the present disclosure is not limited to the exact examples described above and shown in the accompanying drawings, and that various modifications and changes may be made without departing from the scope thereof. It is intended that the scope of the present disclosure be limited only by the appended claims.

[0117] Fig.11A A flowchart of a method when enabling a CIIP block for MPM candidate list generation according to an example of the present disclosure is shown.

[0118] Fig. 11B A flowchart of a method when disabling CIIP blocks for MPM candidate list generation according to an example of the present disclosure is shown.

[0119] Fig.12 A computing environment 1210 is shown coupled to a user interface 1260. The computing environment 1210 may be part of a data processing server. The computing environment 1210 includes a processor 1220, a memory 1240, and an I / O interface 1250.

[0120] The processor 1220 generally controls the overall operation of the computing environment 1210, such as operations associated with display, data acquisition, data communication, and image processing. The processor 1220 may include one or more processors to execute instructions to perform all or some of the steps in the above method. In addition, the processor 1220 may include one or more circuits that facilitate interaction between the processor 1220 and other components. The processor may be a central processing unit (CPU), a microprocessor, a single-chip microcomputer, a GPU, etc.

[0121] The memory 1240 is configured to store various types of data to support the operation of the computing environment 1210. Examples of such data include instructions for any application or method operating on the computing environment 1210, video data, picture data, etc. The memory 1240 can be implemented by using any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk, or optical disk.

[0122] The I / O interface 1250 provides an interface between the processor 1220 and peripheral interface modules, such as a keyboard, a click wheel, buttons, etc. These buttons may include, but are not limited to, a home button, a start scan button, and a stop scan button. The I / O interface 1250 may be coupled to an encoder and a decoder.

[0123] In one embodiment, a non-transitory computer-readable storage medium including a plurality of programs is also provided, such as included in the memory 1240, and the plurality of programs can be executed by the processor 1220 in the computing environment 1210 to perform the above method. For example, the non-transitory computer-readable storage medium can be a ROM, RAM, CD-ROM, magnetic tape, floppy disk, optical data storage device, etc.

[0124] The non-transitory computer-readable storage medium stores therein a plurality of programs to be executed by a computing device having one or more processors, wherein when executed by the one or more processors, the plurality of programs causes the computing device to perform the above-mentioned method for motion prediction.

[0125] In one embodiment, the computing environment 1210 may be implemented using one or more application specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), graphics processing units (GPUs), controllers, microcontrollers, microprocessors, or other electronic components to perform the above methods.

Claims

1. A video encoding method, the method comprising: Divide the current image into one or more blocks; Generate an inter-frame prediction value of a current block based on the inter-frame prediction mode; Generate an intra-frame prediction value of the current block based on an intra-frame prediction mode; Generating a final prediction value of the current block based on the inter-frame prediction value and the intra-frame prediction value; When constructing a most probable mode (MPM) list of neighboring blocks, determining that the prediction mode of the current block is considered as a unified mode, the unified mode is defined as an inter mode, or is defined as an intra mode, wherein, in response to a final prediction of the current block being generated based on both the inter-frame prediction value and the intra-frame prediction value, disabling a bidirectional optical flow (BDOF) operation on the current block; and Based on the final prediction of the current block, a video bitstream output is formed.

2. The method according to claim 1, wherein: The generating the inter prediction value of the current block based on the inter prediction mode includes: generating the inter prediction value of the current block based on at least one motion vector from the current picture to at least one reference picture in the reference picture list.

3. The method according to claim 2, wherein: When the current block is inter-predicted from one reference picture in the first reference picture list L0, the reference picture list is the first reference picture list L0.

4. The method according to claim 2, wherein: When the current block is inter-predicted from one reference picture in the second reference picture list L1, the reference picture list is the second reference picture list L1.

5. The method according to claim 2, wherein: When the current block is inter-predicted from one first reference picture in the first reference picture list L0 and one second reference picture in the second reference picture list L1, the reference picture list is the first reference picture list L0.

6. The method according to claim 2, wherein: When the current block is inter-predicted from one of the first reference pictures in the first reference picture list L0 and one of the second reference pictures in the second reference picture list L1, the reference picture list is the second reference picture list L1.

7. The method according to claim 2, wherein: When inter-frame prediction is performed on the current block from a first reference picture in a first reference picture list L0 and a second reference picture in a second reference picture list L1, the reference picture list is a reference picture list associated with a reference picture, and the one reference picture is one of the first reference picture and the second reference picture having a smaller picture order count (POC) distance from the current picture.

8. The method according to claim 1, wherein: The generating the intra prediction value of the current block based on the intra prediction mode includes: generating the intra prediction value of the current block based on an intra planar prediction mode.

9. A video encoding device, comprising one or more processors and one or more memories coupled to the one or more processors, the one or more processors being configured to execute the method according to any one of claims 1-8.

10. A non-transitory computer-readable storage medium storing a plurality of programs for execution by a video encoding device having one or more processors, wherein: When executed by the one or more processors, the plurality of programs cause the video encoding device to perform the method according to any one of claims 1-8, obtain a corresponding video bitstream and store it in the non-transitory computer-readable storage medium.

11. A computer program product comprising instructions, which, when executed by a processor, implement the method according to any one of claims 1 to 8.

12. A method for generating a video bitstream, comprising: Execute the method of any one of claims 1-8 to generate the video bitstream.

Images