Method and apparatus for decoder-side motion vector optimization and bi-directional optical flow for video coding and decoding

Abstract

Methods and apparatus to optimize MV using bi-directional prediction are disclosed. According to one method, a sample-based optimization and a sub-block-based optimization of a current block are determined. A final optimization for the current block is determined according to the sample-based optimization and the sub-block-based optimization. According to another method, one or more high-order syntax is signaled or parsed, where the high-order syntax indicates whether non-equidistant reference pictures are allowed for bidirectional motion optimization. In response to the high-order syntax indicating that the non-equidistant reference picture is allowed, an optimized MV for at least one block in the current picture is determined based on the reference picture in List 0 and the reference picture in List 1, where a picture distance between the first reference picture and the current picture is different from a picture distance between the second reference picture and the current picture.

Description

[0001] Cross-references

[0002] The present invention claims priority to U.S. Provisional Patent Application Serial No. 63 / 342,180 filed on May 16, 2022 and U.S. Provisional Patent Application Serial No. 63 / 367,821 filed on July 7, 2022. These U.S. Provisional Patent Applications are incorporated herein by reference in their entirety. Technical Field

[0003] The present invention relates to bidirectional prediction based on decoder-side motion vector refinement (DMVR) or bidirectional optical flow (BDOF) MV optimization. Background Art

[0004] Versatile video coding (VVC) is the latest international video coding standard developed by the Joint Video Experts Team (JVET) of the ITU-T Video Coding Experts Group (VCEG) and the ISO / IEC Moving Picture Experts Group (MPEG). The standard has been published as an ISO standard: ISO / IEC 23090-3:2021, Information technology - Coded representation of immersive media - Part 3: Versatile video coding, published in February 2021. VVC is developed based on its predecessor High Efficiency Video Coding (HEVC) by adding more coding tools to improve coding efficiency and also handle various types of video sources including 3-dimensional (3D) video signals.

[0005] Figure 1AAn exemplary self-adjusting inter / intra video coding system incorporating loop processing is illustrated. For intra prediction, prediction data is obtained based on previously encoded video data in the current picture. For inter prediction 112, motion estimation (ME) is performed on the encoder side, and motion compensation (MC) is performed based on the results of ME to provide prediction data obtained based on other pictures and motion data. Switch 114 selects intra prediction 110 or inter prediction 112, and the selected prediction data is provided to adder 116 to form prediction errors, also known as residues. The prediction errors are then processed by transform (Transform, T) 118, followed by quantization (Quantization, Q) 120. The transformed and quantized residues are then encoded by entropy encoder 122 to be included in the video bitstream corresponding to the compressed video data. The bitstream associated with the transform coefficients is then packaged together with side information such as motion and coding modes associated with intra- and inter-frame predictions, and other information such as parameters associated with the loop filters applied to the base image region. Figure 1A As shown, auxiliary information associated with intra prediction 110, inter prediction 112 and in-loop filter (ILPF) 130 is provided to entropy encoder 122. When inter prediction mode is used, one or more reference pictures must also be reconstructed at the encoder end. Therefore, the transformed and quantized residual is processed by inverse quantization (IQ) 124 and inverse transformation (IT) 126 to recover the residual. The residual is then added back to the prediction data 136 at reconstruction (REC) 128 to reconstruct the video data. The reconstructed video data can be stored in a reference picture buffer 134 and used for prediction of other frames.

[0006] like Figure 1AAs shown, the incoming video data undergoes a series of processes in the encoding system. The reconstructed video data from REC 128 may be subjected to various impairments due to the series of processes. Therefore, before the reconstructed video data is stored in the reference picture buffer 134, an in-loop filter 130 is usually applied to the reconstructed video data in order to improve the video quality. For example, a deblocking filter (DF), a sample adaptive offset (SAO), and an adaptive loop filter (ALF) may be used. The loop filter information may need to be incorporated into the bitstream so that the decoder can correctly recover the required information. Therefore, the loop filter information is also provided to the entropy encoder 122 for incorporation into the bitstream. In Figure 1A In , a loop filter 130 is applied to the reconstructed video before the reconstructed samples are stored in a reference picture buffer 134 . Figure 1A The system in is intended to illustrate an exemplary structure of a typical video encoder. It may correspond to a High Efficiency Video Coding (HEVC) system, VP8, VP9, ​​H.264, or VVC.

[0007] like Figure 1B As shown, the decoder can use similar or identical functional blocks as the encoder except for transform 118 and quantization 120, because the decoder only needs inverse quantization 124 and inverse transform 126. Instead of the entropy encoder 122, the decoder uses an entropy decoder (Entropy Decoder) 140 to decode the video bitstream into quantized transform coefficients and required encoding information (e.g., ILPF information, intra-frame prediction information, and inter-frame prediction information). The intra-frame prediction 150 on the decoder side does not need to perform a mode search. Instead, the decoder only needs to generate an intra-frame prediction based on the intra-frame prediction information received from the entropy decoder 140. In addition, for inter-frame prediction, the decoder only needs to perform motion compensation (MC 152) based on the inter-frame prediction information received from the entropy decoder 140, without motion estimation.

[0008] According to VVC, similar to HEVC, the input picture is partitioned into non-overlapping square block areas (called Coding Tree Units (CTUs)). Each CTU can be partitioned into one or more smaller-sized coding units (CUs). The resulting CU partitions can be square or rectangular in shape. In addition, VVC divides the CTU into prediction units (PUs) as units for applying prediction processes (such as inter-frame prediction, intra-frame prediction, etc.).

[0009] The VVC standard incorporates various new codec tools to further improve the codec efficiency of the HEVC standard. In addition, various new codec tools (e.g., decoder-side motion vector optimization (DMVR), bidirectional optical flow (BDOF), and some other codec tools) have been proposed for consideration when developing new coding standards other than VVC.

[0010] In the present invention, a method for improving the coding efficiency related to DMVR and BDOF is disclosed. Summary of the invention

[0011] A method and apparatus for motion vector optimization using bidirectional prediction are disclosed. According to a method of the present invention, input data associated with a current block in a current picture is received, wherein the current block is coded using bidirectional prediction. A sample-based optimization of the current block is determined, wherein each sample of the sample-based optimization is obtained based on a first sample in a first reference picture in list 0 and a second sample in a second reference picture in list 1. A sub-block-based optimization of the current block is determined, wherein each sub-block of the sub-block-based optimization is obtained based on a first sub-block in a first reference picture in list 0 and a second sub-block in a second reference picture in list 1. A final optimization of the current block is determined based on the sample-based optimization and the sub-block-based optimization. The current block is encoded or decoded by using prediction information including the final optimization.

[0012] In one embodiment, the final optimization corresponds to a weighted sum of a sample-based optimization and a sub-block-based optimization. In one embodiment, one or more weights for the weighted sum of the sample-based optimization and the sub-block-based optimization are predetermined. In another embodiment, one or more weights for the weighted sum of the sample-based optimization and the sub-block-based optimization are signaled at the encoder side or parsed at the decoder side. For example, one or more syntaxes associated with the one or more weights may be signaled or parsed at the coding unit (CU) level or the coding tree unit (CTU) level.

[0013] In one embodiment, a final optimization is selected between a sample-based optimization and a sub-block-based optimization. In one embodiment, if a sub-block-based optimization is not available, the sample-based optimization is selected as the final optimization. In another embodiment, if a sample-based optimization is not available, the sub-block-based optimization is selected as the final optimization. In one embodiment, if the magnitude of the motion optimization associated with the sample-based optimization is less than the magnitude of the motion optimization associated with the sub-block-based optimization, the sub-block-based optimization is selected as the final optimization. In another embodiment, if the magnitude of the motion optimization associated with the sub-block-based optimization is less than the magnitude of the motion optimization associated with the sample-based optimization, the sample-based optimization is selected as the final optimization.

[0014] In one embodiment, the high-order syntax is signaled at the encoder side or parsed at the decoder side, wherein the high-order syntax indicates whether a final optimization is allowed to be obtained according to the sample-based optimization and the sub-block-based optimization. In one embodiment, the high-order syntax is signaled at the encoder side or parsed at the decoder side at a slice level, a picture level, or a sequence level. In one embodiment, if the high-order syntax is equal to a first value, the sub-block-based optimization or the sample-based optimization is selected as the final optimization. In another embodiment, if the high-order syntax is equal to a second value, both the sub-block-based optimization or the sample-based optimization are used to obtain the final optimization. In one embodiment, a first motion optimization associated with the sub-block-based optimization is obtained from a second motion optimization associated with a plurality of samples of the sample-based optimization.

[0015] According to another method, one or more high-level syntaxes are signaled or parsed, wherein the one or more high-level syntaxes indicate whether non-equidistant reference pictures are allowed for bidirectional motion optimization. In response to the one or more high-level syntaxes indicating that non-equidistant reference pictures are allowed, an optimized motion vector (MV) of at least one block in a current picture is obtained based on a first reference picture in list 0 and a second reference picture in list 1, and wherein a first picture distance between the first reference picture and the current picture and a second picture distance between the second reference picture and the current picture are different. The at least one block is encoded or decoded by using prediction information including the optimized MV.

[0016] In one embodiment, the one or more high-order syntaxes include a first high-order syntax and a second high-order syntax, wherein the first high-order syntax indicates whether the non-equidistant reference picture is allowed to be used for bidirectional motion optimization based on decoder-side motion vector optimization (DMVR), and the second high-order syntax indicates whether the non-equidistant reference picture is allowed to be used for bidirectional motion optimization based on bidirectional optical flow (BDOF). In one embodiment, the one or more high-order syntaxes are signaled in a sequence parameter set (Sequence Parameter Set, SPS) or parsed from the SPS. In another embodiment, one or more low-order syntaxes are signaled or parsed to indicate whether non-equidistant reference pictures are allowed to be used for lower-order bidirectional motion optimization. For example, the one or more low-order syntaxes may be signaled or parsed at a picture level or a slice level. In another embodiment, the one or more low-order syntaxes are signaled or parsed only in response to the one or more high-order syntaxes indicating that non-equidistant reference pictures are allowed. DETAILED DESCRIPTION

[0017] It will be readily understood that the components of the present invention as generally described and illustrated in the drawings herein may be arranged and designed in a variety of different configurations. Therefore, the following more detailed description of the embodiments of the systems and methods of the present invention as represented in the drawings is not intended to limit the scope of the claimed invention, but rather represents only selected embodiments of the present invention. References throughout this specification to "one embodiment," "an embodiment," or similar language mean that a particular feature, structure, or characteristic described in conjunction with the embodiment may be included in at least one embodiment of the present invention. Therefore, the phrases "in one embodiment" or "in an embodiment" appearing throughout this specification do not necessarily all refer to the same embodiment.

[0018] In addition, in one or more embodiments, the described features, structures or characteristics may be combined in any suitable manner. However, those skilled in the relevant art will recognize that the present invention may be practiced without one or more specific details or utilizing other methods, components, etc. In other cases, known structures or operations are not shown or described in detail to avoid blurring the various aspects of the present invention. The illustrated embodiments of the present invention will be best understood by reference to the accompanying drawings, in which like parts are always represented by like numbers. The following description is by way of example only, and simply illustrates some selected embodiments of the apparatus and method consistent with the present invention as claimed herein.

[0019] Decoder-Side Motion Vector Refinement (DM VR)

[0020] In JVET-D0029 (Xu Chen et al., "Decoder-Side Motion Vector Refinement Based on Bidirectional Template Matching," in Joint Video Exploration Team (JVET) of ITU-T SG 16WP 3 and ISO / IEC JTC 1 / SC29 / WG 11, Fourth Meeting: Chengdu, China, October 15-21, 2016, document: JVET-D0029), decoder-side motion vector refinement (DMVR) based on bidirectional template matching is disclosed. Figure 2 As shown, a template is generated by using bi-prediction of reference blocks (blocks 222 and 232) according to MV0 224 and MV1 234, respectively, where block 212 corresponds to the current block, reference block 222 is located in reference picture 0 (220), and reference block 232 is located in reference picture 1 (230). The template is used as a new current block, and motion estimation is performed to find better matching blocks in reference picture 0 and reference picture 1, as shown in FIG. Figure 3 As shown. The optimized MVs are MV0' (324) and MV1' (334). Then, the optimized MVs (MV0' and MV1') are used to generate the final bidirectional prediction prediction block of the current block. Figure 3 , reference blocks 322 and 332 are positioned according to the optimized MV0′ (324) and MV1′ (334), respectively.

[0021] Multi-Pass Decoder-Side Motion Vector Refinement (MP-DMVR)

[0022] In order to improve the accuracy of the MV in the merge mode, decoder-side motion vector optimization based on bilateral-matching (BM) is applied in VVC. In the bidirectional prediction operation, the optimized MV is searched around the initial MVs (432 and 434) in the reference picture list L0 412 and the reference picture list L1 414 for the current block 420 of the current picture 410. The collocated blocks 422 and 424 in L0 and L1 are determined based on the initial MVs 430 and 432 and the position of the current block 420 in the current picture, such as Figure 4As shown. The BM method calculates the distortion between the two candidate blocks (442 and 444) in the reference picture list L0 and the list L1. The positions of the two candidate blocks (442 and 444) are determined by adding two opposite offsets (462 and 464) to the two initial MVs (432 and 434) to obtain two candidate MVs (452 ​​and 454). Figure 4 As shown, the SAD between the candidate blocks (442 and 444) based on each MV candidate around the initial MV (432 or 434) is calculated. The MV candidate (452 ​​or 454) with the lowest SAD becomes the optimized MV and is used to generate a bidirectional prediction signal.

[0023] In ECM-2.0, if the selected merge candidate meets the DMVR condition, a multi-pass (MP) DMVR method is applied in regular merge mode. In the first pass, bilateral matching (BM) is applied to the coding block. In the second pass, BM is applied to each 16×16 subblock within the coding block. In the third pass, the MV in each 8×8 subblock is optimized by applying bi-directional optical flow (BDOF).

[0024] Similar to DMVR in VVC, BM optimizes a pair of motion vectors MV0 and MV1 under the constraint that MVD0 (MV0'-MV0) is just the opposite sign of MVD1 (MV1'-MV1), as Figure 4 shown.

[0025] Bi-directional Optical Flow (BIO)

[0026] Bidirectional optical flow (BIO or BDOF) is a motion estimation / compensation technique disclosed in JCTVC-C204 (E. Alshina et al., Bidirectional optical flow, Joint Collaboration Team on Video Coding (JCT-VC) of ITU-T SG 16 WP3 and ISO / IEC JTC 1 / SC 29 / WG 11, 3rd meeting: Guangzhou, China, October 7-15, 2010, document: JCT VC-C204) and VCEG-AZ05 (E. Alshina et al., Performance survey of known tools for next generation video codecs, ITU-T SG 16 Issue 6, Video Coding Experts Group (VCEG), 52nd meeting: June 19-26, 2015, Warsaw, Poland, document: VCEG-AZ05). BIO is based on the following: Figure 5 The assumptions of optical flow and steady motion shown in derive sample level motion optimization, where the current pixel 522 in a B slice (bi-predicted slice) 520 is predicted by a pixel (532) in reference picture 0 (530) and a pixel (512) in reference picture 1 (510). Figure 5 As shown, the current pixel 522 is predicted by pixel B 512 in reference picture 1 (510) and pixel A 532 in reference picture 0 (530). Figure 5 In , vx and vy are the pixel displacement vectors in the x-direction and y-direction obtained using the bidirectional optical flow (BIO) model. It is only applied to true bidirectional prediction blocks, which are predicted based on two reference pictures corresponding to the previous picture and the next picture. In VCEG-AZ05, BIO uses a 5×5 window to obtain the motion optimization of each sample. Therefore, for an N×N block, the motion compensation results and corresponding gradient information of (N+4)×(N+4) blocks are required to obtain the sample-based motion optimization of the N×N block. According to VCEG-AZ05, a 6-Tap gradient filter and a 6-Tap interpolation filter are used to generate the gradient information of BIO. Therefore, the computational complexity of BIO is much higher than that of traditional bidirectional prediction. In order to further improve the performance of BIO, the method described below is proposed.

[0027] In conventional bidirectional prediction in HEVC, the predictor is generated using equation (1), where P(0) and P(1) are list 0 and list 1 predictors, respectively.

[0028] P Conventional [i,j]=(P (0) [i,j]+P(1) [i,j]+1)>>1 (1)

[0029] In JCTVC-C204 and VECG-AZ05, BIO predictors were generated using formula (2).

[0030] P OpticalFlow =(P (0) [i,j]+P (1) [i,j]+v x [i,j](I x (0) -I x (1) [i,j])+

[0031] v y [i,j](I y (0) -I y (1) [i,j])+1)>>1 (2)

[0032] In formula (2), Ix (0) and Ix (1) Iy represents the x-direction gradient in the predictors of list 0 and list 1 respectively; (0) and Iy (1) Denote the y-direction gradient in the predictors of list 0 and list 1 respectively; vx and vy denote the offset or displacement in the x-direction and y-direction respectively. The derivation process of vx and vy is shown below. First, the cost function is defined as diffCost(x,y) to find the optimal values ​​of vx and vy. In order to find the optimal values ​​of vx and vy to minimize the cost function diffCost(x,y), a 5×5 window is used. The solution of vx and vy can be represented by using S1, S2, S3, S5 and S6.

[0033]

[0034] The minimum cost function mindiffCost(x,y) can be obtained according to the following formula:

[0035]

[0036] By solving equations (3) and (4), vx and vy can be solved according to equation (5):

[0037]

[0038] in,

[0039]

[0040] In the above formula, The x-direction gradient of the pixel at (x, y) in the list 0 image, Corresponding to the x-direction gradient of the pixel at (x, y) in the picture in List 1, corresponds to the y-gradient of the pixel at (x, y) in the list 0 image, and Corresponds to the y-gradient of the pixel at (x, y) in the image in Listing 1.

[0041] In some related techniques, S2 can be ignored, and vx and vy can be solved according to the following equations:

[0042]

[0043] in,

[0044]

[0045] In JVET-M1001 (Benjamin Bross et al., "Generic Video Codec (Draft 4)", in the Joint Video Exploration Group (JVET) of ITU-T SG16WP 3 and ISO / IEC JTC 1 / SC 29 / WG 11, 13th Meeting: Marrakech, Morocco, January 9-18, 2019, Document: J VET-M1001), a block-based BIO process is performed, and the block size is 4×4. In addition, in order to reduce the computational complexity of BIO, two early termination mechanisms are utilized. For a CU, the sum of absolute difference (SAD) between two predictors (one from L0 and one from L1) is first calculated. If the SAD is less than a threshold, the BIO process is skipped. Otherwise, the BIO process is applied to the current CU. If BIO is applied to the current CU, the sum of absolute differences (SAD) between two predictors (one from L0 and one from L1) in a 4x4 block is calculated, and whether to apply BIO to the current 4x4 block depends on the SAD of a 4x4 block.

[0046] In JVET-Z2025 (Muhammed Coban et al., "Algorithmic Description of Enhanced Compression Model 5 (ECM5)"), the Joint Video Exploration Team (JVET) of ITU-T SG 16WP 3 and ISO / IEC JTC 1 / SC 29 / WG 11, 26th Meeting by Telephone, April 20-29, 2022, Document: JVET-Z2025), sample-based BDOF is utilized. In sample-based BDOF, it is performed per sample instead of obtaining motion optimization (Vx, Vy) on a block basis. The codec block is divided into 8×8 sub-blocks. For each sub-block, it is determined whether to apply BDOF by checking the SAD between two reference sub-blocks relative to a threshold. If it is decided to apply BDOF to the sub-block, for each sample in the sub-block, a sliding 5×5 window is used, and the existing BDOF process is applied for each sliding window to obtain Vx and Vy. The resulting motion optimization (Vx, Vy) is applied to adjust the bi-directionally predicted sample values ​​of the center sample of the window.

[0047] In order to increase the flexibility and / or coding efficiency of DMVR and BDOF, it is proposed to signal a high-level flag to indicate whether DMVR and BDOF can be utilized to optimize bidirectional prediction motion with non-equidistant reference pictures.

[0048] In one embodiment, a flag may be signaled in a sequence parameter set (SPS) and is used to indicate whether optimization of non-equidistant bidirectional prediction by DMVR and BDOF is allowed. In another embodiment, two flags are signaled. One flag is used to indicate whether optimization of non-equidistant bidirectional prediction by DMVR is allowed. Another flag is used to indicate whether optimization of non-equidistant bidirectional prediction by BDOF is allowed. In another embodiment, the flag may be signaled in a picture header or a slice header. In another embodiment, two flags may be signaled. One flag is used for DMVR, such as ph_dmvr_non_equal_refine or sh_dmvr_non_equal_refine in a picture header or a slice header. Another flag is used for BDOF, such as ph_bdof_non_equal_refine or sh_bdof_non_equal_refine in a picture header or a slice header.

[0049] In another embodiment, a higher order syntax is signaled, such as sps_non_equal_refine_enable. Only when the higher order syntax is true, lower order related syntaxes, such as ph_non_equal_refine_enable, ph_dmvr_non_equal_refine, sh_dmvr_non_equal_refine, ph_bdof_non_equal_refine, or sh_bdof_non_equal_refine, may be signaled. In another embodiment, a higher order syntax is signaled, such as sps_non_equal_refine_enable. Only when the higher order syntax is true, lower order related syntaxes, such as ph_non_equal_refine_enable, ph_dmvr_non_equal_refine, sh_dmvr_non_equal_refine, ph_bdof_n on_equal_refine, or sh_bdof_non_equal_refine, may be signaled. In addition, if a related syntax, such as ph_dmvr_non_equal_refine or ph_bdof_non_equal_refine, is signaled in a picture header, the related syntax (e.g., sh_dmvr_non_equal_refine or sh_bdof_non_equal_refine) signaled in a slice header will not be signaled. In other words, the related syntax in the slice header (e.g., sh_dmvr_non_equal_refine or sh_bdof_non_equal_refine) can be signaled only when the related syntax (e.g., ph_dmvr_non_equal_refine or ph_bdof_non_equal_refine) does not exist in the picture header.

[0050] In the related art, one of sample-based BDOF or block-based BDOF is used. In order to further improve the performance of BDOF, we propose to use sample-based BDOF and block-based BDOF together below. The final output is obtained according to the output of sample-based BDOF and the output of block-based BDOF. In addition, since the gradient of a sample to be processed is the same in sample-based BDOF and block-based BDOF, we can directly combine sample-based motion optimization and block-based motion optimization instead of combining the output of sample-based BDOF and the output of block-based BDOF. The derivation process of the final output can be a linear operation or a nonlinear operation.

[0051] In one embodiment, the final output is a weighted sum of the output of the sample-based BDOF and the output of the block-based BD OF. The weights may be predetermined or signaled at a particular level, such as a coding unit (CU) level, a coding tree unit (CTU) level, a slice level, a picture level, or a sequence level. The weights may be predetermined based on QP, video resolution, CU size, temporal ID, reference picture selection, temporal distance between a reference picture and a current picture, etc. Since the gradient of a sample to be processed is the same in sample-based BDOF and block-based BDOF, we can directly combine sample-based motion optimization and block-based motion optimization instead of combining the output of sample-based BDOF and the output of block-based BDOF.

[0052] In one embodiment, the final output is selected from one of the output of the sample-based BDOF and the output of the block-based BDOF. For example, in one embodiment, if motion optimization (e.g., zero division) cannot be determined in the sample-based BDOF, the output of the block-based BDOF is selected. Otherwise, the output of the sample-based BDOF is selected. In another embodiment, if motion optimization (e.g., zero division) cannot be determined in the block-based BDOF, the output of the sample-based BDOF is selected. Otherwise, the output of the block-based BDOF is selected.

[0053] In one embodiment, the final output is selected from one of the output of the sample-based BDOF and the output of the block-based BDOF according to the sample-based motion optimization and the block-based motion optimization. For example, if the magnitude of the sample-based motion optimization is less than the magnitude of the block-based motion optimization, the output of the block-based BDOF is selected. Otherwise, the output of the sample-based BDOF is selected. In another example, if the magnitude of the sample-based motion optimization is greater than the magnitude of the block-based motion optimization, the output of the block-based BDOF is selected. Otherwise, the output of the sample-based BDOF is selected. The magnitude of the motion optimization may be the sum of the absolute values ​​of the motion optimization in the x-dimension and the y-dimension, the absolute value of the product of the motion optimization in the x-dimension and the y-dimension, or the sum of the squares of the motion optimization in the x-dimension and the y-dimension. In another embodiment, the final output is selected from one of the output of the sample-based BDOF and the output of the block-based BDOF according to the motion direction of the sample-based motion optimization and the motion direction of the block-based motion optimization. If the motion direction of the sample-based motion optimization is different from the motion direction of the block-based motion optimization, the output of the block-based BDOF is selected. Otherwise, the output of the sample-based BDOF is selected. In another example, if the motion direction of the sample-based motion optimization is the same as the motion direction of the block-based motion optimization, the output of the block-based BDOF is selected. Otherwise, the output of the sample-based BDOF is selected. The judgment of whether the motion directions are the same can be determined by the product of the two motion optimizations being less than zero. If both products of the sample-based motion optimization and the block-based sample-based motion optimization in the x-dimension and the y-dimension are greater than or equal to zero, they are considered to be the same motion direction. Otherwise, they are considered to be different motion directions. In another embodiment, the decision of whether the motion directions are the same is made independently in the x-dimension and the y-dimension.

[0054] In one embodiment, the final output is selected from one of the output of the sample-based BDOF and the output of the block-based BDOF based on the collected statistics used in the sample-based BDOF and the block-based BDOF. For example, if the denominator of the sample-based BDOF is less than the denominator of the block-based BDOF, the output of the block-based BDOF is selected. Otherwise, the output of the sample-based BDOF is selected. In another example, if the denominator of the sample-based BDOF is greater than the denominator of the block-based motion optimization, the output of the block-based BDOF is selected. Otherwise, the output of the sample-based BDOF is selected. Since the window sizes used in the sample-based BDOF and the block-based BDOF are different, it may be necessary to have a scaling factor in the comparison. The denominator can be changed to one of the statistics used in the BDOF process, such as pixel difference, gradient, etc.

[0055] In another embodiment, the above methods can be combined together. For example, the final output is conditionally selected from one of the output of the sample-based BDOF and the output of the block-based BDOF. If one of the motion optimization of the sample-based BDOF or the motion optimization in the block-based BDOF cannot be determined (e.g., division by zero), we take the other as the final output. Otherwise (i.e., both the motion optimization in the sample-based BDOF and the block-based BDOF can be determined), the final output is a weighted sum of the output of the sample-based BDOF and the output of the block-based BDOF. In another example, the final output is conditionally selected from one of the output of the sample-based BDOF and the output of the block-based BDOF according to the motion optimization. If one of the product of the motion optimization of the sample-based BDOF or the product of the motion optimization in the block-based BDOF is zero, we take the other as the final output. Otherwise (i.e., both products of motion optimization in sample-based BDOF and block-based BDOF are non-zero), the final output is a weighted sum of the output of sample-based BDOF and the output of block-based BDOF. In another example, the final output is conditionally selected from one of the output of sample-based BDOF and the output of block-based BDOF according to the motion direction of motion optimization. If the motion directions of motion optimization in sample-based BDOF and block-based BDOF are the same, the output of sample-based BDOF is used as the final output. Otherwise (i.e., the directions of motion optimization in sample-based BDOF and block-based BDOF are different), the final output is a weighted sum of the output of sample-based BDOF and the output of block-based BDOF.

[0056] Since the gradient of a processed sample is the same in sample-based BDOF and block-based BDOF, when the weighted sum of the two outputs of sample-based BDOF and block-based BDOF is applied in the method proposed above, we can directly combine the sample-based motion optimization and the block-based motion optimization instead of combining the output of sample-based BDOF and the output of block-based BDOF.

[0057] In another embodiment, a high-level syntax is added at the slice level, picture level or sequence level to indicate whether the proposed method is used. If the flag is equal to the first value, sample-based BDOF or block-based BDOF is used. If the flag is equal to the second value, both are used together.

[0058] In another embodiment, the block-based BDOF motion optimization in the above method can be replaced by averaging multiple motion optimizations of sample-based BDOF. For example, the block-based BDOF motion optimization can be obtained by averaging 4 motion optimizations of sample-based BDOF in a 2×2 block. In another example, the block-based BDOF motion optimization can be obtained by averaging 16 motion optimizations of sample-based BDOF in a 4×4 block.

[0059] Any of the aforementioned proposed MV optimization methods using bidirectional prediction can be implemented in the inter / intra / prediction module of the encoder and / or the inter / intra / prediction module of the decoder. For example, at the encoder side, the required MV optimization process using bidirectional prediction based on decoder-side motion vector optimization (DMVR) or bidirectional optical flow (BDOF) can be implemented as Figure 1A The encoder may also use additional processing units to implement the required processing. For the decoder side, the required MV optimization processing using bidirectional prediction based on decoder-side motion vector optimization (DMVR) or bidirectional optical flow (BDO F) can be implemented as follows Figure 1B Part of the MC unit 152 shown. However, the decoder can also use additional processing units to implement the required processing. Alternatively, any of the proposed methods can be implemented as a circuit connected to the inter-frame / intra-frame / prediction module of the encoder and / or the inter-frame / intra-frame / prediction module of the decoder to provide the information required by the inter-frame / intra-frame / prediction module. Although the inter-frame prediction (Inter-Pred.) unit 112 on the encoder side and the MC 152 on the decoder side are shown as separate processing units, they can correspond to executable software or firmware code for a central processing unit (CPU) or a programmable device (e.g., a digital signal processor (DSP) or a field programmable gate array (FPGA)) stored on a medium such as a hard disk or flash memory.

[0060] Figure 6An exemplary flowchart of a video codec (e.g., encoding and / or decoding) system using bidirectional optical flow (BIO) to optimize the motion of a bidirectionally predicted block according to an embodiment of the present invention is illustrated. The steps shown in the flowchart and other flowcharts in the present disclosure can be implemented as program codes that can be executed on one or more processors (e.g., one or more CPUs) on the encoder side and / or the decoder side. The steps shown in the flowchart can also be implemented based on hardware (such as one or more electronic devices or processors arranged to perform the steps in the flowchart). According to the method, in step 610, input data associated with a current block in a current picture is received, wherein the current block is encoded (e.g., encoded (encoded) and / or decoded (decoded)) using bidirectional prediction. In step 620, a sample-based optimization of the current block is determined, wherein each sample of the sample-based optimization is obtained based on the first sample in the first reference picture in list 0 and the second sample in the second reference picture in list 1. In step 630, a sub-block based optimization of the current block is determined, wherein each sub-block of the sub-block based optimization is obtained based on a first sub-block in a first reference picture in list 0 and a second sub-block in a second reference picture in list 1. In step 640, a final optimization of the current block is determined based on the sample based optimization and the sub-block based optimization. In step 650, the current block is encoded or decoded by using prediction information including the final optimization.

[0061] Figure 7 An exemplary flow chart of a video coding (e.g., encoding and / or decoding) system for optimizing a predictor of a bidirectional prediction block using bidirectional prediction according to an embodiment of the present invention is illustrated. According to the method, input data associated with a current block in a current picture is received in step 710, wherein the current block is coded (e.g., encoded and / or decoded) using bidirectional prediction. One or more high-level syntaxes are signaled or parsed in step 720, wherein the one or more high-level syntaxes indicate whether non-equidistant reference pictures are allowed for bidirectional motion optimization. In response to the one or more high-level syntaxes indicating that non-equidistant reference pictures are allowed, an optimized MV of at least one block in the current picture is determined in step 730 based on a first reference picture in list 0 and a second reference picture in list 1, and wherein a first picture distance between the first reference picture and the current picture and a second picture distance between the second reference picture and the current picture are different. In step 740, the at least one block is encoded or decoded by using prediction information including the optimized MV.

[0062] The flowchart shown above is intended to illustrate an example of video coding according to the present invention. Without departing from the spirit of the present invention, those skilled in the art may modify each step, rearrange the steps, split the steps, or combine the steps to practice the present invention. In the present disclosure, specific syntax and semantics have been used to illustrate examples of implementations of the present invention. A technician can practice the present invention by replacing syntax and semantics with equivalent syntax and semantics without departing from the spirit of the present invention.

[0063] The above description is presented to enable one of ordinary skill in the art to practice the present invention as provided in the context of a specific application and its requirements. Various modifications to the described embodiments will be apparent to those skilled in the art, and the general principles defined herein may be applied to other embodiments. Therefore, the present invention is not intended to be limited to the specific embodiments shown and described, but is given the widest scope consistent with the principles and novel features disclosed herein. In the above detailed description, various specific details are shown in order to provide a thorough understanding of the present invention. However, it will be appreciated by those skilled in the art that the present invention may be put into practice.

[0064] The embodiments of the present invention as described above can be implemented with various hardware, software codes or a combination of the two. For example, the embodiments of the present invention can be a program code integrated into one or more circuits of a video compression chip or integrated into video compression software to perform the processing described herein. The embodiments of the present invention can also be a program code to be executed on a digital signal processor (Digital Signal Processor, DSP) to perform the processing described herein. The present invention can also be related to many functions performed by a computer processor, a digital signal processor, a microprocessor or a field programmable gate array (field programmable gate array, FPGA). These processors can be configured to perform specific tasks according to the present invention by executing machine-readable software code or firmware code, and the machine-readable software code or firmware code defines a specific method embodied by the present invention. The software code or firmware code can be developed in different programming languages ​​and different formats or styles. The software code can also be encoded for different target platforms. However, the different code formats, styles and languages ​​of the software code and other means of configuring the code to perform the tasks according to the present invention will not depart from the spirit and scope of the present invention.

[0065] The present invention may be implemented in other specific forms without departing from the spirit or essential characteristics of the present invention. The described examples are considered in all respects to be illustrative only and not restrictive. Therefore, the scope of the present invention is indicated by the appended claims rather than by the foregoing description. All changes within the meaning and scope of the equivalents of the claims will be included within their scope.

Claims

1. A video encoding and decoding method, comprising: receiving input data associated with a current block, wherein the input data comprises pixel data of the current block to be encoded at an encoder side or coded data associated with the current block to be decoded at a decoder side, and wherein the current block is coded and decoded using bidirectional prediction; Determining a sample-based optimization of the current block, wherein each sample of the sample-based optimization is obtained based on a first sample in a first reference picture in list 0 and a second sample in a second reference picture in list 1; Determining a sub-block-based optimization of the current block, wherein each sub-block of the sub-block-based optimization is obtained based on a first sub-block in the first reference picture in list 0 and a second sub-block in the second reference picture in list 1; determining a final optimization of the current block according to the sample-based optimization and the sub-block-based optimization; and The current block is encoded or decoded by using the prediction information including the final optimization.

2. The method of claim 1, wherein: The final optimization corresponds to a weighted sum of the sample-based optimization and the sub-block-based optimization.

3. The method of claim 2, wherein one or more weights for the weighted sum of the sample-based optimization and the sub-block-based optimization are predetermined.

4. The method of claim 2, wherein one or more weights for the weighted sum of the sample-based optimization and the sub-block-based optimization are signaled at the encoder side or resolved at the decoder side.

5. The method of claim 4, wherein the one or more syntaxes associated with the one or more weights are signaled or parsed at a coding unit (CU) level or a coding tree unit (CTU) level.

6. The method of claim 1, wherein: The final optimization is selected between the sample-based optimization and the sub-block-based optimization.

7. The method of claim 6, wherein: If the sub-block based optimization cannot be obtained, the sample based optimization is selected as the final optimization.

8. The method of claim 6, wherein: If the sample-based optimization cannot be obtained, the sub-block-based optimization is selected as the final optimization.

9. The method of claim 6, wherein: If the magnitude of the optimization associated with the sample-based optimization is less than the magnitude of the optimization associated with the sub-block-based optimization, the sub-block-based optimization is selected as the final optimization.

10. The method of claim 6, wherein: If the magnitude of the optimization associated with the sub-block-based optimization is less than the magnitude of the optimization associated with the sample-based optimization, the sample-based optimization is selected as the final optimization.

11. The method of claim 1, wherein: A high-level syntax is signaled at the encoder side or parsed at the decoder side, wherein the high-level syntax indicates whether the final optimization is allowed to be obtained according to the sample-based optimization and the sub-block-based optimization.

12. The method of claim 11, wherein: The high-level syntax is signaled at the encoder side or parsed at the decoder side at slice level, picture level or sequence level.

13. The method of claim 11, wherein: If the high-order syntax is equal to a first value, the sub-block based optimization or the sample based optimization is selected as the final optimization.

14. The method of claim 11, wherein: If the high-order syntax is equal to a second value, both the sub-block based optimization or the sample based optimization are used to obtain the final optimization.

15. The method of claim 11, wherein: A first motion optimization associated with the sub-block based optimization is derived from a second motion optimization associated with a plurality of samples of the sample based optimization.

16. An apparatus for video encoding and decoding, the apparatus comprising one or more electronic devices or processors, the one or more electronic devices or processors being arranged to: Receive input data associated with the current block, where The input data comprises pixel data of the current block to be encoded at an encoder side or coded data associated with the current block to be decoded at a decoder side, and wherein the current block is coded and decoded using bidirectional prediction; Determining a sample-based optimization of the current block, wherein each sample of the sample-based optimization is obtained based on a first sample in a first reference picture in list 0 and a second sample in a second reference picture in list 1; Determining a sub-block-based optimization of the current block, wherein each sub-block of the sub-block-based optimization is obtained based on a first sub-block in a first reference picture in list 0 and a second sub-block in a second reference picture in list 1; determining a final optimization of the current block according to the sample-based optimization and the sub-block-based optimization; and The current block is encoded or decoded by using the prediction information including the final optimization.

17. A video encoding and decoding method, comprising: Receiving input data associated with a current picture, wherein the input data comprises pixel data of the current block to be encoded at an encoder side or coded-decoded data associated with the current block to be decoded at a decoder side; signaling or parsing one or more high-level syntaxes, wherein the one or more high-level syntaxes indicate whether non-equidistant reference pictures are allowed for bidirectional motion optimization; In response to the one or more high-level syntax indicating that the non-equidistant reference pictures are allowed, deriving an optimized motion vector (MV) of at least one block in the current picture based on a first reference picture in list 0 and a second reference picture in list 1, wherein a first picture distance between the first reference picture and the current picture and a second picture distance between the second reference picture and the current picture are different; and The at least one block is encoded or decoded using prediction information including the optimized MV.

18. The method of claim 17, wherein: The one or more high-order syntaxes include a first high-order syntax and a second high-order syntax, wherein the first high-order syntax indicates whether the non-equidistant reference picture is allowed to be used for bidirectional motion optimization based on decoder-side motion vector refinement (DMVR), and the second high-order syntax indicates whether the non-equidistant reference picture is allowed to be used for bidirectional motion optimization based on bi-directional optical flow (BDOF).

19. The method of claim 17, wherein: The one or more high-level syntaxes are signaled in a Sequence Parameter Set (SPS) or parsed from the SPS.

20. The method of claim 19, wherein: One or more low-level syntaxes are signaled or parsed to indicate whether the non-equidistant reference pictures are allowed to be used for lower-order bi-directional motion optimization.

21. The method of claim 20, wherein: The one or more low-level syntaxes are signaled or parsed at a picture level or a slice level.

22. The method of claim 20, wherein: The one or more low-level syntaxes are signaled or parsed only in response to the one or more high-level syntaxes indicating that the non-equidistant reference pictures are allowed.

23. An apparatus for video encoding and decoding, the apparatus comprising one or more electronic devices or processors, the one or more electronic devices or processors being arranged to: Receive input data associated with the current image, where: The input data comprises pixel data of the current block to be encoded at an encoder side or coded-decoded data associated with the current block to be decoded at a decoder side; signaling or parsing one or more high-level syntaxes, wherein the one or more high-level syntaxes indicate whether non-equidistant reference pictures are allowed for bidirectional motion optimization; In response to the one or more high-level syntax indicating that the non-equidistant reference pictures are allowed, deriving an optimized motion vector MV of at least one block in the current picture based on a first reference picture in list 0 and a second reference picture in list 1, and wherein a first picture distance between the first reference picture and the current picture and a second picture distance between the second reference picture and the current picture are different; and The at least one block is encoded or decoded using prediction information including the optimized MV.

Images